Strategy & Metrics (SM)

Software Domain - HAIAMM v3.5

Generated from the canonical cell records in cells/. Do not edit by hand; edit cells/SM-Software-L*.cell.json and re-run tools/render.py.

---

Practice Overview

Business Function: Governance

Objective: Stand up the AI/HAI Software Assurance program, inventory everything the organization ships with AI in it, and baseline the metrics that prove shadow AI in engineering is decreasing.

Description: A program operating at this level knows what AI its engineers have shipped. There is a published charter with an accountable executive, a single inventory covering all seven software archetypes, and a monthly discovery sweep that reconciles that inventory against source-code imports, dependency manifests, CI/CD telemetry, runtime egress and the cloud bill. Shadow AI in engineering is visible and attributed rather than suspected. A consumption ledger sits beside the inventory, joined on the artifact key, so every AI feature has an owner and a cost per unit of work. The program is positioned as a fast path rather than a gate: low-risk archetypes clear intake in days, which is why engineering routes through it instead of around it.

---

Maturity Level 1, Foundational

Objective: Stand up the AI/HAI Software Assurance program, inventory everything the organization ships with AI in it, and baseline the metrics that prove shadow AI in engineering is decreasing.

A program operating at this level knows what AI its engineers have shipped. There is a published charter with an accountable executive, a single inventory covering all seven software archetypes, and a monthly discovery sweep that reconciles that inventory against source-code imports, dependency manifests, CI/CD telemetry, runtime egress and the cloud bill. Shadow AI in engineering is visible and attributed rather than suspected. A consumption ledger sits beside the inventory, joined on the artifact key, so every AI feature has an owner and a cost per unit of work. The program is positioned as a fast path rather than a gate: low-risk archetypes clear intake in days, which is why engineering routes through it instead of around it.

Dependencies

• Required: None.
• Alignment (not blocking): Enterprise security strategy and the existing AppSec / SAMM program, so the AI program plugs into existing risk committees rather than forming a parallel stack; FinOps cost allocation for the AI scope, which the consumption ledger depends on
• Supports / unblocks: PC-Software-L1 (policy needs the inventory); TA-Software-L1 (threat modeling needs the asset list); SR-Software-L1 (requirements key on archetype); SA-Software-L1 (reference patterns need the archetype list); ML-Software-L1 (logging baseline needs the inventory); IM-Software-L1 (incident routing needs the owner structure); Required by: DR-Software-L1, EG-Software-L1, EH-Software-L1, IR-Software-L1, SM-Software-L2, ST-Software-L1

Desired Outcomes

• Shadow AI in engineering is visible, attributed to a named owning team, and trending down quarter over quarter.
• A single AI/HAI software inventory is the authoritative source of truth across Security, Engineering, Data/ML, Product and Privacy.
• An accountable executive owns AI/HAI software risk, and decision rights for approval, block, exception and go-live are unambiguous.
• Every AI/HAI software artifact in production has a named owning team and a cost per unit of work that someone can read.
• Engineering experiences the program as a faster path to shipping sanctioned AI than building around it.

Activities

A) Charter the AI/HAI Software Assurance program

Publish a program charter of no more than four pages, signed by the CISO jointly with the CTO or Head of Engineering, within the first 30 days. The charter states the problem in first-party terms: the organization ships AI features and therefore carries deployer duties, regardless of whose model is behind them; it enumerates the seven in-scope archetypes, being LLM-integrated application or feature, autonomous agent, RAG pipeline, fine-tuning or training workload, evaluation harness, model-serving service, and classical ML integrated into a product surface; it names a cross-functional working group with one engineering representative per product line that ships AI, plus Data/ML platform, Privacy/Legal, Product and Platform/SRE; it assigns four decision rights explicitly, being who approves an artifact for production, who can block one, who grants exceptions and who owns the go-live gate; it commits to a published intake service-level agreement, because a program without one becomes a queue teams learn to bypass; and it sets a numeric year-one target for each L1 outcome metric rather than a statement of intent. The working group meets monthly and publishes minutes.

B) Build the inventory and run monthly shadow-AI discovery

Stand up one AI/HAI software inventory as the program's source of truth, with a record per artifact carrying name, owning team, archetype, production status, exposure (customer-facing, internal, developer-only), AI capabilities (tool use, autonomy, multi-turn memory, retrieval sources, training-data sources), data classes processed, model provider and version at both inference and fine-tuning, approval status, and links to the TA snapshot, SR evidence map, SA pattern, latest DR decision and ML logging baseline. Seed it from engineering records, then reconcile monthly against six discovery signals that platform and security teams already have: source-code scanning for LLM SDK, vector-store and ML-framework imports; dependency manifests across every package ecosystem in use; CI/CD job telemetry flagging training, fine-tuning and evaluation runs; runtime egress to model-provider domains from internal services; the internal model registry; and cloud and provider spend broken out by cost-allocation tag. Run a self-attestation campaign with a published amnesty window, because engineers who fear consequences hide unsanctioned AI rather than surface it. Any artifact appearing in a discovery signal but not the inventory is shadow AI until attributed.

C) Baseline the scoreboard and open the consumption ledger

Instrument five outcome metrics in the first quarter and report them to the executive sponsor quarterly, broken out by archetype: inventory coverage against discovery reconciliation, the shadow-AI-in-engineering ratio, acceptable-use attestation across engineering headcount, the share of production artifacts with a named owning team, and known data-exposure events originating from AI features. Alongside the inventory, open a consumption ledger joined on the artifact key, recording tokens and calls per artifact with the owning team and, critically, the denominator, being the unit of business work that artifact delivers. Baseline the Assured Spend Ratio, which is the share of AI consumption flowing through inventoried, owner-attributed artifacts, and report it beside the artifact-count shadow-AI ratio rather than instead of it, because the two diverging is the finding. Where AI spend is not allocated to an owning team, record the ledger as blocked on allocation and raise a tagging request with the FinOps practice; do not estimate the number, and do not score the program down for a gap it cannot close from the security side.

Outcome Metrics (L1)

Metric	Baseline	L1 Target	Source
AI/HAI software inventory coverage (discovered artifacts present in the inventory)	first reconciliation sweep, recorded with a timestamp and an uncertainty range	≥90% within 12 months	Inventory reconciled against the six discovery signals
Shadow-AI-in-engineering ratio (unsanctioned artifacts in production ÷ total artifacts in production)	first sweep result	≤15% and falling for two consecutive quarters	Inventory approval-status field
Engineering headcount with an acknowledged AI Acceptable Use and Engineering Standards Policy	first attestation run	≥95% of engineering headcount	HR / LMS attestation records
Production AI/HAI artifacts carrying a named owning team	first sweep result	100%	Inventory owner field
Assured Spend Ratio (AI consumption through inventoried, owner-attributed artifacts)	first ledger close, or 'blocked on allocation' where AI spend is untagged	≥60% baselined and reported quarterly	Consumption ledger joined to inventory on artifact key

Process Metrics (leading)

• Discovery cadence: the six-signal reconciliation sweep runs at least monthly, with the last-run timestamp published.
• Intake service-level agreement: new artifacts triaged within 5 business days, provisional approval for low-risk archetypes within 10.
• Inventory freshness: ≥80% of records reviewed or auto-updated within the last 90 days, keyed to last-deploy timestamp.
• Working-group cadence honoured monthly, with minutes published to engineering.

Effectiveness Metrics (business value)

• Intake-to-provisional-approval time falls quarter over quarter; a program that slows engineering down gets routed around.
• Archetype reuse rate rises: the share of new AI artifacts adopting a sanctioned archetype rather than inventing one.
• Documented avoided-incident cases where discovery caught an AI-specific risk before it reached production.

Consumption Metrics (denominated, never absolute spend)

Metric	Denominator	Economic failure it detects	Source
Token and call cost per completed task, by archetype	per completed task	loop burn and retry storms (RA), and silent scope creep where an artifact quietly acquires authority its use case never justified (EA)	AI gateway or provider billing API, joined to task-completion telemetry
Assured Spend Ratio: AI consumption through inventoried, owner-attributed artifacts	per dollar of AI consumption across the estate	artifact-count coverage reads healthy while the load-bearing production systems sit outside the program, because counting prototypes is easy and money follows load	Consumption ledger joined to inventory on artifact key
Unattributed AI consumption, as a share of the ledger	per dollar of AI consumption across the estate	shadow AI that discovery sweeps missed but the invoice did not, and allocation gaps that make every other consumption metric uncomputable	Cloud and provider cost-allocation tags reconciled against inventory

Level of Effort (incremental over the level below)

• Stand-up: 160 hours · Ongoing: 80 hours/quarter
• Program Manager / Practice Lead — primary, 30 days/year
• Executive Sponsor — low, 3 days/year
• AppSec / SecEng — regular, 12 days/year
• AI/ML Engineer — regular, 8 days/year
• DevOps / Platform Engineer — regular, 10 days/year
• Material new tooling: Inventory system of record (an existing CMDB or asset register is sufficient), Source-code and dependency scanning across all package ecosystems in use

Success Criteria

• A charter is published and jointly sponsored by the CISO and the CTO or Head of Engineering, with four decision rights named and a year-one numeric target per outcome metric.
• One inventory exists as the source of truth, covering all seven archetypes, at ≥90% coverage of discovered artifacts within 12 months.
• The shadow-AI-in-engineering ratio is baselined and has fallen for two consecutive quarters.
• ≥95% of engineering headcount has acknowledged the AI Acceptable Use and Engineering Standards Policy.
• A consumption ledger is open, joined to the inventory, with a denominator declared per archetype and the Assured Spend Ratio baselined or explicitly recorded as blocked on allocation.

---

Maturity Level 2, Comprehensive

Objective: Risk-tier the AI/HAI software inventory and calibrate program intensity per tier, establishing the rubric and treatment matrix every other Software-domain L2 practice inherits.

At this level the program stops treating every artifact the same. A published rubric derives a tier from auditable dimensions rather than reviewer instinct, and a treatment matrix states what each tier receives from every downstream practice, so a customer-facing agent acting on user accounts and an internal eval harness are no longer processed by the same queue. Shadow AI is measured per tier, because one uncontrolled Critical-tier agent is not equivalent to five Low-tier prototypes, and spend caps are proportionate to tier with a declared behaviour at the cap. Tier movements are normal, logged, sponsor-visible events. The program can now say where it is mature and where it is still building, per tier, and defend the answer.

Dependencies

• Required: SM-Software-L1, PC-Software-L1, TA-Software-L1
• Alignment (not blocking): Enterprise risk-tiering conventions, so AI tiers map onto existing severity language rather than introducing a second scale
• Supports / unblocks: PC-Software-L2; TA-Software-L2; SR-Software-L2; SA-Software-L2; DR-Software-L2; IR-Software-L2; ST-Software-L2; EH-Software-L2; IM-Software-L2; ML-Software-L2; Required by: EG-Software-L2, SM-Software-L3

Desired Outcomes

• Every artifact in the inventory carries a tier derived from explicit, auditable dimensions rather than reviewer judgment.
• Program intensity is visibly differentiated: Critical receives the full program, Low receives the fast track, and nobody confuses the two.
• The quarterly scoreboard splits by tier, so the sponsor can see which tiers are healthy and which are drifting.
• Tier movements are logged with rationale and reviewed, rather than being quiet reclassifications that follow the workload.
• Every Critical- and High-tier artifact operates inside a spend cap with a declared behaviour at the cap.
• Maturity is defensible per tier, so 'mature at Critical, still building at Medium' is an evidenced statement.

Activities

A) Publish the risk-tier rubric

Define four tiers, Critical, High, Medium and Low, derived deterministically from seven auditable dimensions, each of which a reviewer can evidence rather than assert. The dimensions are: data sensitivity processed at inference or training, where regulated data elevates to Critical or High; decision-affecting use, where output materially drives a decision with legal or significant effect on a person under GDPR Art. 22, or falls in an EU AI Act Annex III high-risk category, elevating to Critical; agentic capability, where a tool surface acting on organizational or customer systems elevates, with tool count and scope as inputs; user exposure, where customer-facing elevates, partner-facing elevates less, and developer-only lowers; training-data posture, where fine-tuning on customer or regulated data elevates; production load-bearing status, where sitting on a revenue path, a customer-onboarding flow or an authentication decision elevates; and concentration, where being the sole AI capability behind a product elevates. Publish the rubric as a short table with the derivation rule written out. Human override is permitted but must be recorded with rationale and reviewed by the working group, because an override nobody sees is a rubric nobody trusts.

B) Calibrate program intensity and spend caps per tier

Publish a tier-treatment matrix stating what each tier receives across intake depth, threat-modeling depth, pattern adherence, design review lane, implementation-review cadence, security-test battery, environment hardening, monitoring detections, issue-management service levels and re-review triggers on material change. Every downstream Software-domain L2 practice inherits this matrix and defines its own tier-specific activities against it; the matrix is authored here and changes flow through the working group, so the tiers cannot drift apart practice by practice. Extend the matrix with a consumption row: every Critical- and High-tier artifact carries a spend cap proportionate to its tier and a declared behaviour when the cap is reached, being degrade, queue or refuse, and never silently continue. Treat a cost-driven change to a Critical- or High-tier artifact as a material change that triggers the tier's re-review obligation, because switching to a cheaper model, shortening log retention, sampling telemetry or extending a cache time-to-live each alters the security profile that testing baselined; thrift is not an exemption from change governance.

C) Report the scoreboard per tier and govern tier movement

Convert the L1 scoreboard to a tier-aware one reported quarterly to the sponsor and monthly to the working group. Report inventory state by tier and archetype, with every Critical-tier artifact as its own row rather than aggregated away; report the shadow-AI ratio per tier, where a Critical-tier unsanctioned artifact in production is a headline finding and a Low-tier one is a line item; report service-level adherence per tier across intake, design review, implementation review, testing, monitoring and issue management; and report the Assured Spend Ratio per tier, targeting full attribution of Critical-tier consumption. Maintain a tier-movement log recording every artifact that moved up a tier, which tightens its treatment, and every artifact that moved down, with rationale, and review it quarterly with the sponsor. The quarterly review asks one question explicitly: is the program's effort distribution matching the program's risk distribution, or is reviewer time going where the work is easiest rather than where the risk is.

Outcome Metrics (L2)

Metric	Baseline	L2 Target	Source
Inventory carrying a current tier assignment	L1 achieved inventory coverage	100% of the inventory, with tier age under 12 months	Inventory tier field
Critical-tier artifacts receiving full-scope treatment in the last 12 months	first tier-treatment audit	≥95%	Cross-practice artifact records joined to inventory tier
Tier-weighted shadow AI: unsanctioned Critical-tier artifacts in production	L1 shadow-AI ratio, re-cut per tier	0 at Critical, and falling at every other tier	Inventory status and tier fields
Per-tier service-level adherence across intake, DR, IR, ST, ML and IM	first quarter of per-tier telemetry	≥90% per tier	Program telemetry from each practice's decision log
Critical- and High-tier artifacts operating under a spend cap with declared at-cap behaviour	first cap audit	100% of Critical, ≥90% of High	AI gateway budget configuration reconciled against inventory; where no AI gateway is deployed, provider-side usage and limit configuration read per provider account, recorded at E1
Assured Spend Ratio at Critical tier	L1 estate-wide ASR	100% of Critical-tier consumption attributed, ≥90% estate-wide	Consumption ledger joined to inventory tier field

Process Metrics (leading)

• Rubric review cadence: the tier rubric is reviewed every two quarters and changes are change-logged.
• Tier calibration exercise: at least quarterly, 20 artifacts are re-tiered by a second reviewer and drift is recorded.
• Per-tier queue depth is monitored, and no tier's backlog exceeds its published threshold.
• Cost-driven change events are captured as material changes and routed to re-review within the tier's window.

Effectiveness Metrics (business value)

• Effort allocation match: the share of reviewer hours spent on Critical and High tiers rises relative to L1.
• Throughput at Low and Medium tiers improves as sanctioned-archetype reuse replaces bespoke design review.
• Assurance premium: the unit cost of AI work delivered through the sanctioned path against the ungoverned path, targeting 1.2 or below.
• Tier-level dashboards are cited in quarterly engineering planning and budget conversations.

Consumption Metrics (denominated, never absolute spend)

Metric	Denominator	Economic failure it detects	Source
Cost per completed task at Critical tier against the archetype median	per completed task	excessive agency showing up economically before it shows up behaviourally, where an artifact does more work than its declared scope implies	AI gateway telemetry joined to inventory tier and archetype
Spend-cap breach rate and at-cap behaviour conformance, per tier	per artifact-quarter	denial-of-wallet, and caps that are configured but silently continue past the limit rather than degrading, queueing or refusing	AI gateway budget enforcement log
Assurance premium: assured unit cost over ungoverned unit cost	per completed task	the governed path costing materially more than the ungoverned one, which is the mechanism by which engineering teams route around an assurance program regardless of policy	Consumption ledger segmented by inventory approval status
Assured Spend Ratio per tier	per dollar of AI consumption within the tier	aggregate coverage masking that Critical-tier consumption is flowing through artifacts the program does not govern	Consumption ledger joined to inventory tier field

Level of Effort (incremental over the level below)

• Stand-up: 120 hours · Ongoing: 100 hours/quarter
• Program Manager / Practice Lead — primary, 25 days/year
• Security Architect — regular, 15 days/year
• AppSec / SecEng — regular, 12 days/year
• Legal / Privacy Counsel — low, 4 days/year
• Executive Sponsor — low, 3 days/year
• Material new tooling: Tier derivation implemented as a rule set against inventory fields rather than a spreadsheet formula, AI gateway with per-artifact budget enforcement

Success Criteria

• A tier rubric is published and applied, with 100% of the inventory carrying a current tier derived from the seven auditable dimensions.
• A tier-treatment matrix is published and the ten downstream Software-domain L2 practices are calibrated against it.
• The quarterly scoreboard reports per tier and per archetype, with unsanctioned Critical-tier artifacts in production standing at zero.
• Per-tier service-level adherence is at or above 90% across program activities.
• Every Critical-tier artifact operates under a spend cap with a declared and tested at-cap behaviour, and cost-driven changes route to re-review.
• A tier-movement log is maintained with rationale and reviewed quarterly by the sponsor.

---

Maturity Level 3, Industry-Leading

Objective: Drive inventory, tiering and the consumption ledger from live build and runtime signals, benchmark externally, and compute the program's return rather than narrating it.

At this level the program is signal-driven rather than ticket-driven. Inventory and tier assignments update from CI/CD events, model-registry changes, dependency scanning, runtime egress and provider billing feeds, with human curation reserved for exceptions and genuinely new archetypes. A published data-quality service-level objective states how fresh the picture is, in hours rather than months. The program can say where it stands against peers on named metrics, and it contributes back rather than only consuming. The return on the program is arithmetic over the consumption ledger, being contained denial-of-wallet events, reclaimed loop burn and avoided overage, rather than a modelled loss estimate nobody believes.

Dependencies

• Required: SM-Software-L2, ML-Software-L2, EG-Software-L2
• Alignment (not blocking): FinOps at Run maturity for the AI scope, which the automated ledger attribution depends on
• Supports / unblocks: Every other Software-domain practice at L3, which inherits automated inventory and tier data from this cell

Desired Outcomes

• Inventory accuracy is measured in hours of latency rather than months of staleness.
• Tier assignments adjust automatically when their dimensional inputs change, and humans intervene only on exceptions.
• External benchmarking is routine, and the sponsor can answer how the program compares with specific deltas rather than impressions.
• The organization is a net contributor to the AI/HAI software assurance ecosystem rather than only a consumer of it.
• Program return is computed from meters, with contained loss and reclaimed spend traceable to ledger entries.

Activities

A) Automate inventory, tiering and ledger attribution from live signals

Drive the inventory from authoritative engineering feeds rather than from tickets: CI/CD events tagged by job type for training, fine-tuning, evaluation and deploy; model-registry events for registration, version promotion and deprecation; dependency-manifest scanning on every commit; runtime egress showing a new outbound flow to a model-provider domain; gateway telemetry where a previously unseen artifact begins emitting prompts; and provider billing feeds where a new cost-allocation tag appears. Implement tier assignment as a versioned, replayable rule set over the L2 rubric inputs, so a tier change is reproducible and auditable, and wire tier changes to trigger downstream obligations automatically, where a Medium-to-Critical promotion opens the design-review, testing and monitoring work items without anyone remembering to. Automate ledger attribution from billing and gateway feeds against cost-allocation tags. Reserve human curation for three cases: genuinely new archetypes that do not fit the existing seven, ambiguous discoveries such as a shared library that may or may not be in production, and conflicting dimensional inputs. Publish a data-quality service-level objective covering tier correctness within 48 hours of a material change and inventory completeness against reconciliation.

B) Benchmark externally and publish the comparison

Compare program metrics against peer data twice a year through named channels rather than impressions: OWASP SAMM AI extensions and OpenSSF AI working groups, CSA AI Safety Initiative working groups, MITRE ATLAS practitioner exchanges, sector information-sharing bodies with AI working groups, the FinOps Foundation AI scope for unit-economics comparison, and formal peer roundtables. Publish a brief covering at least five peer-comparable metrics, being inventory coverage, shadow-AI ratio, per-tier service-level adherence, automation level, and denominated unit cost per completed task by archetype. Choose peers who stretch the program rather than flatter it; an organization shipping customer-facing agents does not benchmark against internal-tool builders, and the brief should state who was chosen and why. Feed benchmark deltas into the next year's investment priorities and into the board narrative, and record where the program is behind as explicitly as where it leads, because a benchmarking brief that only finds good news stops being read.

C) Compute the program's return and contribute back

Replace the asserted return narrative with arithmetic over the consumption ledger, refreshed at least annually and reconciled with Finance. Compute four components from meters rather than models: contained denial-of-wallet and loop-burn events, valued as the mean burn rate multiplied by the mean time to detect, multiplied by the events contained; reclaimed loop burn, being spend on non-completing tasks before and after cap enforcement; reclaimed licence and capacity waste; and avoided provider overage under committed-spend governance. Publish the Assurance Cost Ratio, being program spend over AI consumption spend, and defend the band rather than the number, because a program consuming a large share of the AI budget is over-built and one consuming almost nothing is theatre. Separately, contribute at least four substantive artifacts per year to the ecosystem, targeting MITRE ATLAS for techniques observed in first-party software, OWASP LLM and Agentic Top 10 reviews backed by real telemetry, NIST AI RMF Playbook contributions, AI Vulnerability Database submissions for responsibly disclosed findings, and OpenSSF AI supply-chain work; conference talks and press releases do not count, because the quality gate is whether a technical artifact lands in the external repository.

Outcome Metrics (L3)

Metric	Baseline	L3 Target	Source
Inventory and tier auto-update latency after a material change	L2 manual review cycle time	≤48 hours for 99% of material changes	Inventory event telemetry with change timestamps
Inventory entries auto-curated rather than human-curated	first quarter of automation telemetry	≥80% auto-curated	Curation telemetry from the inventory pipeline
Inventory completeness against discovery-source reconciliation	L2 achieved coverage	≥99%	Automated reconciliation report
Tier changes auto-triggering downstream practice obligations	first quarter of workflow telemetry	100% within 24 hours of the tier change	Workflow automation telemetry
Peer-comparable metrics tracked in the semi-annual benchmarking brief	0	≥5 metrics, with named peer cohort and stated selection rationale	Published benchmarking brief
Substantive ecosystem contributions accepted per year	0	≥4 accepted technical artifacts, excluding talks and press	Contribution log with external acceptance references

Process Metrics (leading)

• Automation health: signal-feed freshness and error rate are monitored, and on-call is paged when a feed exceeds its staleness threshold.
• Benchmarking cadence is honoured, with the semi-annual brief published on schedule.
• The contribution pipeline always carries at least two items in draft or under external review.
• Tier-rule changes are versioned, replayable and reviewed quarterly by the working group.

Effectiveness Metrics (business value)

• Sponsor decisions on budget, headcount and scope cite benchmark data and per-tier metrics rather than narrative.
• Assurance Cost Ratio is published and defended within a stated band, giving the program a proportionality argument.
• Computed return is reconciled with Finance and survives that reconciliation without restatement.
• Industry recognition: working-group invitations, citations of contributions, and peer adoption of published patterns.

Consumption Metrics (denominated, never absolute spend)

Metric	Denominator	Economic failure it detects	Source
Denominated unit cost per completed task, benchmarked against the external peer cohort	per completed task	the program optimising against its own history while drifting away from what comparable organizations achieve, which internal trend lines cannot reveal	Consumption ledger compared against the semi-annual benchmarking brief
Assurance Cost Ratio: assurance program spend over AI consumption spend	per dollar of AI consumption across the estate	a program that is over-built relative to the estate it governs, or one small enough to be theatre, neither of which is visible without the ratio	Program cost centre reconciled against the consumption ledger
Reclaimed loop burn: spend on non-completing tasks, before and after cap enforcement	per completed task	rogue-agent reflective loops and retry storms burning budget with no delivered output, which absolute spend reporting cannot distinguish from growth	AI gateway telemetry segmented by task-completion status
Ledger attribution latency after a new artifact begins consuming	per newly observed artifact	shadow AI visible on the invoice days or weeks before it is visible to the program, which is the gap denial-of-wallet exploits	Provider billing feed compared against inventory first-seen timestamp

Level of Effort (incremental over the level below)

• Stand-up: 200 hours · Ongoing: 60 hours/quarter
• Program Manager / Practice Lead — primary, 20 days/year
• DevOps / Platform Engineer — regular, 20 days/year
• Security Architect — regular, 10 days/year
• Compliance / Audit — low, 4 days/year
• Executive Sponsor — low, 2 days/year
• Material new tooling: Event pipeline joining CI/CD, model-registry, dependency-scan, egress and billing feeds into the inventory, Versioned rule engine for tier derivation with replay

Success Criteria

• A published data-quality service-level objective is met, with tier correctness within 48 hours of material change and inventory completeness at or above 99%.
• Tier assignment runs on a published, versioned, replayable rule set with exception-based human review, and tier changes trigger downstream obligations within 24 hours.
• A semi-annual benchmarking brief is published to the sponsor citing at least five peer-comparable metrics, with the peer cohort named and its selection defended.
• At least four substantive technical artifacts per year are accepted by external bodies, with talks and press explicitly excluded from the count.
• The return narrative is computed from ledger arithmetic, reconciled with Finance, and published alongside the Assurance Cost Ratio.

---

Key Success Indicators

Level 1:

• A charter is published and jointly sponsored by the CISO and the CTO or Head of Engineering, with four decision rights named and a year-one numeric target per outcome metric.
• One inventory exists as the source of truth, covering all seven archetypes, at ≥90% coverage of discovered artifacts within 12 months.
• The shadow-AI-in-engineering ratio is baselined and has fallen for two consecutive quarters.
• ≥95% of engineering headcount has acknowledged the AI Acceptable Use and Engineering Standards Policy.
• A consumption ledger is open, joined to the inventory, with a denominator declared per archetype and the Assured Spend Ratio baselined or explicitly recorded as blocked on allocation.

Level 2:

• A tier rubric is published and applied, with 100% of the inventory carrying a current tier derived from the seven auditable dimensions.
• A tier-treatment matrix is published and the ten downstream Software-domain L2 practices are calibrated against it.
• The quarterly scoreboard reports per tier and per archetype, with unsanctioned Critical-tier artifacts in production standing at zero.
• Per-tier service-level adherence is at or above 90% across program activities.
• Every Critical-tier artifact operates under a spend cap with a declared and tested at-cap behaviour, and cost-driven changes route to re-review.
• A tier-movement log is maintained with rationale and reviewed quarterly by the sponsor.

Level 3:

• A published data-quality service-level objective is met, with tier correctness within 48 hours of material change and inventory completeness at or above 99%.
• Tier assignment runs on a published, versioned, replayable rule set with exception-based human review, and tier changes trigger downstream obligations within 24 hours.
• A semi-annual benchmarking brief is published to the sponsor citing at least five peer-comparable metrics, with the peer cohort named and its selection defended.
• At least four substantive technical artifacts per year are accepted by external bodies, with talks and press explicitly excluded from the count.
• The return narrative is computed from ledger arithmetic, reconciled with Finance, and published alongside the Assurance Cost Ratio.

---

Common Pitfalls

Level 1:

• ❌ Seeding the inventory only from AI features the product team announced, which misses LLM calls added inside pre-existing features, internal-tool integrations, and fine-tunes run from a researcher's notebook.
• ❌ Treating AI features that call vendor APIs as a Vendors-domain concern. The organization ships the feature and owns the deployer duty; the vendor question is a sub-concern of it.
• ❌ Running the program as a gate with no published intake service-level agreement, so product teams ship behind feature flags and the inventory silently decays.
• ❌ Counting activity rather than outcome: sweeps run and tickets closed, instead of shadow-AI ratio down and attested coverage up.
• ❌ Reporting absolute AI spend to the sponsor. It rises with adoption, which is what the organization wants, so it says nothing about whether the AI is governed.

Level 2:

• ❌ Rubric dimensions that are subjective, such as important or sensitive, so reviewers tier differently, auditors do not trust the result and tier movements feel political.
• ❌ A treatment matrix that is published but not enforced, so Critical artifacts land in the same queue as Low and calibration exists only on paper.
• ❌ Reporting the scoreboard in aggregate, which hides Critical-tier shadow AI behind a healthy-looking estate-wide average.
• ❌ Over-engineering the rubric with so many dimensions and edge cases that tier derivation becomes a ritual rather than a computation.
• ❌ Configuring spend caps without declaring the behaviour at the cap, so the limit is reached and the system quietly keeps spending.

Level 3:

• ❌ Running automation without a data-quality service-level objective, so the signal-driven inventory drifts silently and practitioners quietly stop trusting it.
• ❌ Choosing benchmark peers who flatter the program, such as comparing an enterprise customer-facing AI estate against early-stage internal-tool builders.
• ❌ Counting conference talks and press releases as ecosystem contributions instead of technical artifacts that land in an external repository.
• ❌ Automated tier rules that encode historical bias by under-weighting archetypes the program has historically under-reviewed, with no audit of the rule inputs.
• ❌ Tier-change automation firing so noisily that every prompt-template tweak triggers re-review, so engineering teams disable the signal source rather than tune the rule.

---

Practice Maturity Questions

Level 1:

1. Is there a published AI/HAI Software Assurance charter, jointly sponsored by the CISO and the CTO or Head of Engineering, naming a cross-functional working group, four explicit decision rights, a published intake service-level agreement and a numeric year-one target for each L1 outcome metric?
2. Does one AI/HAI software inventory exist covering all seven archetypes, reconciled at least monthly against source-code imports, dependency manifests, CI/CD telemetry, runtime egress, the model registry and tagged cloud spend, reaching ≥90% coverage of discovered artifacts within 12 months?
3. Are the five L1 outcome metrics baselined and reported quarterly to the sponsor by archetype, and is a consumption ledger open and joined to the inventory with a declared denominator, or explicitly recorded as blocked on FinOps allocation?

Level 2:

1. Is every AI/HAI software artifact assigned a risk tier from a published rubric covering data sensitivity, decision-affecting use under GDPR Art. 22 or EU AI Act Annex III, agentic capability, user exposure, training-data posture, production load-bearing status and concentration, with overrides recorded and reviewed?
2. Is there a published tier-treatment matrix driving differential intensity across PC, TA, SR, SA, DR, IR, ST, EH, ML and IM, with at least 95% of Critical-tier artifacts having received full-scope treatment in the last 12 months, and are cost-driven changes treated as material changes?
3. Does the quarterly scoreboard report per tier and per archetype with unsanctioned Critical-tier artifacts explicitly tracked at zero, is the Assured Spend Ratio reported per tier, and is tier movement logged with rationale and reviewed by the sponsor?

Level 3:

1. Do inventory and tier assignment auto-update from CI/CD, model-registry, dependency-manifest, runtime-egress, gateway and billing signals against a published data-quality service-level objective, with at least 80% of curation automated and human review reserved for exceptions?
2. Do you publish a semi-annual external benchmarking brief comparing the program against at least five peer-comparable metrics through named channels, with the peer cohort selected to stretch rather than flatter the program, and does it drive investment decisions?
3. Is the program's return computed from consumption-ledger arithmetic covering contained denial-of-wallet events, reclaimed loop burn, reclaimed waste and avoided overage, reconciled with Finance, and published alongside the Assurance Cost Ratio?

---

Lightweight Questions (E0 rung)

Answerable yes/no by one person in under 30 seconds, with nothing open.

Level 1:

• Can you name one person accountable for AI risk in the software your engineers ship?
• Is there a single list of every AI feature currently in production?
• Do you know what your AI features cost per unit of work they deliver?

Level 2:

• Does every AI feature in your inventory carry a risk tier?
• Would a Critical-tier AI feature get more security review than a Low-tier one?
• Does any AI feature have a spend cap that actually stops it?

Level 3:

• Does your AI inventory update itself, or does someone update it?
• Can you say how your AI assurance metrics compare with peer organizations?
• Is your AI assurance return computed from real spend data?

---

Tracking

Level 1 — Jira Epic, labels haiamm-sm-software-l1, haiamm-governance

• Done when: Inventory coverage reported at or above 90% from reconciliation.json for two consecutive quarters, with zero production artifacts lacking a named owning team and the shadow-AI ratio at or below 0.15.
• Git artifacts: governance/ai-software-charter.md, inventory/artifacts.json, inventory/reconciliation.json, finops/consumption-ledger.json
• Git signal: reconciliation.json is regenerated by the monthly discovery job rather than hand-edited, and its commit timestamp is within 35 days at all times.
• CI check: ai-inventory-reconcile

Level 2 — Jira Epic, labels haiamm-sm-software-l2, haiamm-governance

• Done when: Tier coverage at 100% and zero unsanctioned Critical-tier artifacts in production, held across two consecutive quarterly scoreboards, with per-tier service-level adherence at or above 90%.
• Git artifacts: governance/tier-rubric.md, governance/tier-treatment-matrix.md, governance/tier-movement-log.json, inventory/artifacts.json, gateway/budget-config.yaml
• Git signal: tier-movement-log.json gains an entry for every tier change with a rationale field, and tier-rubric.md carries a commit within every two-quarter review interval.
• CI check: tier-assignment-audit

Level 3 — Jira Epic, labels haiamm-sm-software-l3, haiamm-governance

• Done when: Tier update latency at or below 48 hours at the 99th percentile and auto-curation at or above 80%, sustained for two quarters, with a benchmarking brief and contribution log current within their published intervals.
• Git artifacts: inventory/pipeline/, governance/tier-rules.versioned.json, governance/benchmarking-brief.md, governance/contribution-log.json, finops/return-computation.json
• Git signal: tier-rules.versioned.json carries a monotonic version and a replay test in CI, and return-computation.json is regenerated from ledger data rather than hand-edited.
• CI check: inventory-pipeline-slo

---

Mappings

Level	HAI TTP	ATLAS	OWASP LLM 2025	Compliance
L1	—	AML.T0029	LLM10	EU AI Act Art. 26; ISO/IEC 27001 A.5.9; ISO/IEC 42001 4.1; NIST AI RMF GOVERN 1.1; NIST AI RMF MAP 1.1
L2	EA	AML.T0029	LLM06, LLM10	EU AI Act Art. 9; EU AI Act Annex III; GDPR Art. 22; NIST AI RMF MAP 1.5; NIST AI RMF MANAGE 1.3; ISO/IEC 42001 6.1
L3	EA, RA	AML.T0029	LLM10	EU AI Act Art. 26; EU AI Act Art. 72; ISO/IEC 42001 9.1; NIST AI RMF GOVERN 1.6; NIST AI RMF GOVERN 5.1; NIST AI RMF MANAGE 4.1

---

Document Version: HAIAMM v3.5 (generated) / Practice: Strategy & Metrics (SM) / Domain: Software