HAIAMM v3.5, Vendors Domain Handbook

AI/HAI Vendors Assurance, the complete practice set for this domain

Generated document. Compiled from the canonical cell records in cells/. Do not edit by hand, edit the cell and re-run python3 tools/render.py domain.

Canonical model: HAIAMM-v3.5-Framing.md · Full model: HAIAMM-v3.5-Handbook.md

Part I, The domain

Subject of this domain: the AI/HAI capability the organization consumes from third parties

Boundary rule

If a third party is responsible for the security of the inside of the thing, it is Vendors — even when the organization operates the surface around it.

Archetypes

Cells in this domain key their content to these archetypes. A generated cell must be usable against every archetype listed, or must say explicitly which archetypes it applies to and why.

1. Consumer GenAI
2. AI-embedded SaaS feature
3. AI coding assistant
4. AI API / foundation-model vendor
5. AI agent / automation platform

A single asset is often several archetypes at once. Generated content must accommodate that rather than assuming a single clean classification.

Shadow AI in this domain

Expense and SSO signals plus SaaS-admin AI-feature dashboards — the primary L1 outcome in this domain.

Every domain's SM L1 carries a domain-appropriate shadow-AI discovery mechanism. Downstream practices at L1 assume the inventory it produces exists.

Consumption unit and denominator

Unit: cost per seat, cost per token, committed spend, overage exposure and exit cost

Denominator: per active seat, per governed workload

What it detects here: concentration risk priced in dollars rather than asserted as a worry, and lock-in measured as switching cost. This is the only domain where consumption is a contractual obligation: commit terms, overage rates, price-change notice and exit cost are negotiated, not observed.

Meter: procurement records, vendor invoices, SaaS management platform.

Consumption is a governed resource in v3.5, not an incidental discovery signal. Cells for the consumption-sensitive practices (SM, SR, EH, ML, IM) divide this domain's unit by this domain's denominator; absolute spend is a ledger input and never a metric.

Primary stakeholders

Procurement / Sourcing · TPRM Analyst · Legal / Privacy Counsel · Security Architect

HAI-TTP emphasis

TM and EA on agent/automation platforms whose permission model exceeds any single user's; AGH via tool-returned content; data egress as the dominant loss mode across all five archetypes.

Compliance overlay

EU AI Act Art. 25 value-chain and Art. 26 deployer duties; GDPR Art. 28 processor terms and subprocessor chains; DPA and AI-addendum coverage; ISO/IEC 42001 supplier controls.

Part II, Effort and ownership

What this domain costs to run, per cell, incremental over the level below. These are the BSAMM-style costs and personnel the v3.0 handbooks promised and did not ship.

Reaching L3 across all twelve practices in the Vendors domain is roughly 6,330 hours of stand-up and 4,490 hours per quarter of ongoing effort for a mid-size organization, incremental and cumulative across the three levels. L1 alone is a far smaller commitment, and is where the dependency graph says to start.

Tooling called for in this domain

• AI telemetry ingestion into the existing detection platform
• Adaptive limit management with governed ceilings
• Adversarial testing capability, internal or contracted
• Adversary emulation tooling and an isolated exercise environment
• Automated design criteria evaluation in the delivery pipeline
• Automated remediation platform with scoped action classes
• Automated triage and bounded containment orchestration
• Behavioural baselining across AI telemetry
• Configuration conformance scanning against the baseline definitions
• Continuous adversarial testing capability
• Continuous configuration drift detection against declared control state
• Continuous conformance assertion evaluation against existing telemetry
• Continuous conformance verification against configuration and runtime state
• Continuous control assertion framework spanning pipeline and runtime
• Continuous discovery feeds from identity provider, expense system and egress telemetry
• Continuous drift detection across configuration, policy mode and runtime state
• Contract register with structured AI-specific clause tracking
• Detection quality measurement with disposition labelling
• Detection-as-code pipeline with continuous test execution
• Egress monitoring against a maintained AI-provider domain list
• Evidence collection automation against existing systems of record
• Evidence map collection wired to repository, CI and configuration systems
• Evolving adversarial harness with case generation
• Executable pattern artifacts in the platform scaffolding
• Existing learning management system, extended with applied-scenario assessment
• Guidance search and question telemetry instrumentation
• Hands-on laboratory environment for Critical-tier curricula
• Policy-as-code engine integrated into the delivery pipeline
• Regression corpus management with freshness tracking
• SaaS admin API polling for newly enabled AI features
• SaaS management platform or equivalent covering expense, IdP and endpoint signals
• Seat utilisation reporting joined to identity provider activity
• Test harness capable of exercising AI-specific controls in CI
• Threat model register linked to the asset inventory and change events

Part III, Build order

L1 dependencies within this domain, from the cell records. Start where nothing is required.

Part IV, The twelve practices

Business Function: Governance

Strategy & Metrics (SM)

Objective: Stand up vendor AI assurance, inventory every AI tool and service the organization consumes, and baseline the metrics that prove shadow AI adoption is decreasing.

A program at this level knows what AI its people are using. One inventory covers the five vendor archetypes, being consumer generative AI, AI-embedded SaaS, AI coding assistants, AI API and foundation-model vendors, and AI agent or automation platforms. This is the domain where shadow AI is the primary L1 outcome rather than a secondary concern, because adoption happens on a corporate card in minutes and because AI features arrive silently inside SaaS products the organization already approved. Discovery therefore runs across expense records, identity provider app catalogues, network egress, endpoint management and SaaS admin consoles, and an amnesty window does more for coverage than any control.

Level 1, Foundational

Stand up vendor AI assurance, inventory every AI tool and service the organization consumes, and baseline the metrics that prove shadow AI adoption is decreasing.

Activities

A) Charter vendor AI assurance and open an amnesty window — Publish a charter within the first 30 days sponsored by the CISO jointly with the Chief Procurement Officer or CIO, co-signed by Privacy counsel. Name the five in-scope archetypes explicitly, because the most commonly missed category is the AI feature that appeared inside an already-approved SaaS product without a new contract, a new invoice or any procurement event at all. Establish the working group across Security, Procurement, Legal and Privacy, IT, Data Governance and one business-unit representative, and define decision rights over who sanctions an AI vendor, who blocks one, who grants exceptions and who owns offboarding. Open an amnesty window of at least 60 days, publicized through all-hands and team channels, under which anyone can disclose an AI tool they are using with no disciplinary consequence. Amnesty is the single highest-yield discovery mechanism in this domain and it only works once, so pair it with a sanctioned path that is genuinely faster than signing up with a credit card, or disclosure will not repeat.

B) Discover across five signals and inventory by archetype — Build one AI vendor inventory recording per vendor the archetype, business owner, sanction status, data classes shared, whether inputs are used for provider training and how that is contractually established, sub-processor chain, data residency and transfer basis, model provider behind the service, seat count and utilisation, contract and renewal dates, and links to the TA snapshot and PC compliance mapping. Reconcile monthly against five discovery signals: expense and procurement records including corporate card transactions, which catch the self-service adoptions that never reached procurement; identity provider application catalogues and single-sign-on grants, which catch tools users authenticated into; network egress to known AI provider domains, which catches everything with a browser or API footprint; endpoint management and browser extension audits, which catch locally installed assistants; and the admin consoles of major approved SaaS products, which is the only way to see AI features that were switched on inside a product the organization already owns. Treat any vendor appearing in a signal but not the inventory as shadow AI until attributed.

C) Baseline the scoreboard and open the vendor spend ledger — Instrument and report quarterly: AI vendor inventory coverage against discovery reconciliation, the shadow-AI ratio as the domain's primary L1 outcome, acceptable-use attestation across the workforce, the share of sanctioned vendors with a documented training-data position, and known data-exposure events to unsanctioned AI tools. Open the vendor consumption ledger recording cost per active seat and cost per token or unit of service, attributed to the business owner, together with committed-spend terms, overage rates, price-change notice periods and exit cost. This domain is the only one where consumption is contractually governed rather than merely observed, which makes the ledger a negotiating instrument as well as a control: concentration risk becomes a number in dollars, and lock-in becomes a measured switching cost rather than an asserted worry. Track seat utilisation alongside seat cost, since provisioned-and-never-used seats are simultaneously wasted spend and unnecessary data exposure.

Outcome metrics

Consumption metrics — Cost per active seat, by vendor and archetype per per active seat · Cost per unit of service against committed-spend terms and overage rates per per governed workload · Assured Spend Ratio for vendor AI: consumption through sanctioned vendors per per dollar of vendor AI consumption

Success criteria

• A charter is published and jointly sponsored, naming all five archetypes and four decision rights, with a publicized amnesty window of at least 60 days completed.
• One AI vendor inventory reaches 90% coverage or better against five-signal discovery within 12 months.
• The shadow AI ratio is baselined at the close of amnesty and has fallen for two consecutive quarters.
• Every vendor receiving confidential or regulated data has a contractually established training-data position on file.
• A vendor consumption ledger is open recording cost per active seat and per unit of service, with seat utilisation tracked beside seat cost.

Assessment questions

1. Is there a published vendor AI assurance charter jointly sponsored by the CISO and the procurement or IT leader, naming all five vendor archetypes, four decision rights and a completed amnesty window of at least 60 days paired with a sanctioned path faster than self-service adoption?
2. Does one AI vendor inventory reconcile monthly against expense and procurement records, identity provider catalogues, network egress, endpoint and browser-extension audits, and the admin consoles of major approved SaaS products, reaching 90% coverage or better within 12 months?
3. Do all vendors receiving confidential or regulated data have a contractually established training-data position, and is a consumption ledger open recording cost per active seat with utilisation, committed-spend terms, overage rates and exit cost?

Common pitfalls

• ❌ Inventorying only vendors that went through procurement, which misses every self-service adoption and every AI feature enabled inside an existing SaaS contract.
• ❌ Running an amnesty window without first building a faster sanctioned path, so disclosure happens once and never repeats.
• ❌ Accepting a vendor's marketing statement that it does not train on customer data instead of establishing the position contractually.
• ❌ Tracking seat cost without utilisation, so provisioned-and-unused seats persist as both waste and standing data exposure.
• ❌ Reviewing vendors at renewal only, so an AI capability added mid-contract is never assessed.

Level 2, Comprehensive

Tier AI vendors by data exposure and dependency, calibrate assessment depth per tier, and govern concentration, committed spend and exit cost as measured positions.

Activities

A) Publish the vendor risk-tier rubric — Derive four tiers from five auditable dimensions. Data exposure is the floor: a vendor receiving regulated or customer-confidential content sits at Critical or High regardless of spend, because contract value and risk exposure are only weakly correlated in this domain and a cheap tool can carry the whole customer base through it. Dependency criticality elevates where the vendor sits on a revenue path or a customer-facing flow. Service autonomy elevates where the vendor's AI acts rather than advises, which is what distinguishes an agent platform from a drafting assistant. Sub-processor depth elevates where the vendor's own model provider and its sub-processors extend the chain beyond the point where the organization has visibility or contractual reach. Switching cost elevates where exit would take longer than a quarter or require re-engineering, because a dependency that cannot be exited is a dependency that cannot be governed by the threat of exit. Publish the rubric and re-derive when data classes or service scope change.

B) Calibrate assessment depth and contract terms per tier — Publish a tier-treatment matrix covering per tier: assessment depth from questionnaire to evidence-backed review with technical validation; required contractual terms including training-data prohibition, sub-processor notification and approval rights, audit and evidence rights, incident-notification windows, model-change notification, data-residency commitments and deletion-on-termination obligations; monitoring obligations including provider status and advisory tracking; reassessment cadence and the events that force reassessment ahead of schedule; and exit requirements including data-export format and a tested exit runbook for Critical tier. Add the consumption row: Critical- and High-tier vendors carry governed committed-spend terms with a defined overage rate, a price-change notice period and a documented exit cost. Treat a vendor's model change, a sub-processor addition or a material pricing change as a material change triggering reassessment inside the tier window, since each alters the risk profile that the original assessment described.

C) Price concentration and exit, and report per tier — Produce a standing concentration and dependency statement, refreshed quarterly and signed by the sponsor, that prices exposure rather than describing it: the share of AI consumption carried by each vendor and each underlying model provider, the single points of failure that follow, the measured exit time and switching cost for every Critical-tier vendor, and an explicit accept, mitigate or migrate decision for each. Measuring exit cost requires actually specifying the exit, which is why most organizations never do it, and why the number is worth having. Report the scoreboard per tier quarterly covering inventory and tier state, shadow AI ratio per tier with unsanctioned Critical-tier vendors targeted at zero, contractual-term conformance per tier, reassessment currency, seat utilisation by tier, and the Assured Spend Ratio for vendor consumption. Maintain a tier-movement log recording the triggering dimension and review it with the sponsor.

Outcome metrics

Consumption metrics — Cost per active seat by tier, with utilisation per per active seat · Overage as a share of committed spend, by vendor per per governed workload · Concentration: share of vendor AI consumption carried by the largest vendor and the largest underlying model provider per per dollar of vendor AI consumption · Measured exit cost as a multiple of the vendor's annualised contract value per per Critical-tier vendor relationship

Success criteria

• A vendor tier rubric is published and applied to 100% of vendors in use, re-derived on data-class or service-scope change.
• A tier-treatment matrix is published covering assessment depth, contractual terms, monitoring, reassessment cadence and exit requirements.
• Unsanctioned Critical-tier AI vendors stand at zero, and Critical-tier contractual-term conformance is at 95% or better.
• Every Critical-tier vendor has a measured exit time and documented switching cost, refreshed annually.
• A sponsor-signed concentration and dependency statement prices vendor and model-provider exposure with an explicit decision per single point of failure.

Assessment questions

1. Is every AI vendor assigned a tier from a published rubric covering data exposure, dependency criticality, service autonomy, sub-processor depth and switching cost, re-derived when data classes or service scope change?
2. Is there a published tier-treatment matrix covering assessment depth, required contractual terms including training-data prohibition and model-change notification, monitoring, reassessment cadence and tested exit runbooks, with at least 95% Critical-tier contractual conformance?
3. Does a sponsor-signed concentration and dependency statement price the share of consumption carried by each vendor and each underlying model provider, with measured exit time and switching cost for every Critical-tier vendor and an explicit accept, mitigate or migrate decision?

Common pitfalls

• ❌ Tiering by contract value, which in this domain correlates weakly with exposure: a low-cost tool can carry the entire customer base through it.
• ❌ Assessing the vendor while ignoring the model provider behind it, so sub-processor concentration is far higher than the vendor list suggests.
• ❌ Writing exit requirements into contracts without ever testing an export, so the exit runbook is discovered to be fiction during the incident that requires it.
• ❌ Treating a vendor's model change as routine maintenance rather than a material change to the system that was assessed.
• ❌ Describing concentration and lock-in qualitatively, which gives negotiation nothing to work with and migration no budget line.

Level 3, Industry-Leading

Drive vendor discovery and tiering from live identity, expense and egress feeds, benchmark vendor unit economics, and contribute AI vendor assurance intelligence back.

Activities

A) Make discovery continuous across identity, expense and egress — Replace the monthly sweep with continuous feeds. Stream identity-provider application grants and first-authentication events, so a new single-sign-on grant to an AI service raises a discovery event the same day; stream expense and corporate-card feeds filtered against a maintained AI-vendor merchant list; stream network egress against a maintained AI-provider domain list that is itself updated from threat-intelligence and community sources rather than by hand; and poll the admin APIs of major approved SaaS products for newly enabled AI features, which remains the only reliable way to see AI that arrives without a transaction. Wire tier derivation to re-run automatically when a vendor's recorded data classes, service scope or sub-processor chain changes, and to open downstream reassessment obligations on tier increase. Publish a data-quality objective covering time from first adoption signal to inventory record, targeting 48 hours, and reserve human curation for ambiguous merchant records, shared-account adoptions and genuinely novel vendor archetypes.

B) Benchmark vendor unit economics and assessment depth — Publish a semi-annual brief comparing the program against named peers on at least five metrics: vendor inventory coverage and discovery latency, shadow AI ratio, contractual-term conformance at Critical tier, cost per active seat by archetype, and seat utilisation. Vendor unit economics benchmark unusually well because the products are the same across organizations, so a materially higher cost per active seat for the same archetype is a negotiating position rather than an observation, and a materially lower one usually indicates either better terms worth sharing or unused capacity worth reclaiming. Draw comparison data from peer chief information security officer communities, sector information-sharing bodies, the FinOps Foundation SaaS and AI scopes, and Shared Assessments or equivalent third-party risk communities. Name the cohort, defend its selection, and record where the program trails. Feed the deltas directly into the renewal calendar so that comparisons arrive before negotiations rather than after them.

C) Contribute assessment intelligence and compute the vendor return — Contribute at least four substantive artifacts per year: AI vendor assessment criteria and questionnaire modules published through Shared Assessments, CSA AI Safety Initiative or sector bodies; observed vendor-layer techniques such as silently enabled AI features, training-data term changes on renewal, and sub-processor chain extensions submitted to MITRE ATLAS or community advisories; anonymized contractual-term benchmarks that help peers negotiate; and OWASP LLM working-group input on third-party and supply-chain risks. Count only externally accepted artifacts. Compute the vendor-domain return from invoice and ledger arithmetic: reclaimed licence waste from deprovisioned unused seats, which is usually the largest and most defensible component; avoided overage where committed-spend governance caught consumption growth before invoicing; consolidation savings where duplicate archetype coverage was retired; and avoided price increases where notice periods were enforced. Reconcile with Finance and Procurement annually and publish alongside the domain Assurance Cost Ratio.

Outcome metrics

Consumption metrics — Cost per active seat by archetype, benchmarked against the peer cohort per per active seat · Reclaimed licence waste from deprovisioned unused seats per per active seat recovered · Avoided overage under committed-spend governance per per governed workload

Success criteria

• Discovery is continuous across identity, expense, egress and SaaS admin feeds, with adoption-to-inventory latency at or below 48 hours at the 90th percentile.
• Tier re-derivation triggers automatically on vendor scope or sub-processor change and opens downstream reassessment within 5 business days.
• A semi-annual benchmarking brief covers at least five peer-comparable metrics, delivered to the renewal calendar ahead of Critical-tier negotiations.
• At least four externally accepted vendor assurance contributions per year.
• Vendor-domain return is computed from invoice and ledger arithmetic and reconciled with Finance and Procurement annually.

Assessment questions

1. Is discovery continuous across identity-provider grants, expense feeds, egress telemetry and SaaS admin APIs, with adoption-to-inventory latency at or below 48 hours at the 90th percentile and at least 75% of entries auto-curated?
2. Does the semi-annual benchmarking brief compare at least five peer-comparable metrics including cost per active seat and seat utilisation by archetype, with the cohort named and the deltas delivered to the renewal calendar before Critical-tier negotiations?
3. Does the program contribute at least four externally accepted vendor assurance artifacts per year, and is the vendor return computed from invoice arithmetic covering reclaimed licence waste, avoided overage and consolidation savings, reconciled with Finance and Procurement?

Common pitfalls

• ❌ Automating expense and identity discovery while leaving SaaS admin consoles on a manual cycle, which misses the category that generates no transaction at all.
• ❌ Maintaining the AI-provider domain list by hand, so it decays and egress discovery quietly stops finding anything new.
• ❌ Producing benchmark data after renewals rather than before them, which makes the comparison interesting instead of useful.
• ❌ Counting reclaimed seats as savings without confirming deprovisioning actually removed the access as well as the charge.
• ❌ Publishing contractual benchmarks without legal review, which can breach confidentiality terms in the very contracts being compared.

Policy & Compliance (PC)

Objective: Publish the priority compliance map and the minimum policy stack governing AI/HAI tools and services consumed from third parties, so every obligation reaches a named policy and a named owner.

A program at this level has written down which rules apply to AI/HAI tools and services consumed from third parties and who answers for each. A one-page priority compliance map traces every in-scope obligation to a policy clause and an evidence artifact, so a regulator asking how a given article is addressed reaches one row, one policy and one piece of evidence rather than a research project. The AI Vendor Policy is published, acknowledged by the in-scope population, and specific enough to be violated, which is the test that separates a policy from a statement of values.

Level 1, Foundational

Publish the priority compliance map and the minimum policy stack governing AI/HAI tools and services consumed from third parties, so every obligation reaches a named policy and a named owner.

Activities

A) Publish the priority compliance map — Produce a one-page priority compliance map covering the obligations that actually apply to AI/HAI tools and services consumed from third parties: GDPR Art. 28 processor obligations; GDPR Art. 44-49 transfers; EU AI Act Art. 25 along the value chain; ISO/IEC 27001 A.5; SOC 2 CC9.2, plus any sector-specific rules in scope. Each row names the obligation, the specific clause of the internal policy that implements it, the accountable owner, the evidence artifact that demonstrates conformance, and the practice that produces that evidence. The map is one page by design. A compliance register of two hundred rows is a document nobody reads and nobody maintains, whereas a one-page map that traces the obligations a regulator is actually likely to ask about is a document that survives contact with an audit. Review it quarterly and whenever a new obligation enters scope, and treat an obligation with no named evidence artifact as an open finding rather than as a row to be filled in later.

B) Publish the AI Vendor Policy and the supporting policy stack — Publish an AI Vendor Policy naming sanctioned and prohibited categories, the mandatory contract clause set including a training-data prohibition and model-change notification, the data classes that may be shared with each archetype, and the intake gate every new AI vendor passes. Write each requirement so that conformance is observable: a rule stating that AI must be used responsibly cannot be assessed, whereas a rule naming the five vendor archetypes, being consumer generative AI, AI-embedded SaaS, AI coding assistant, AI API and foundation-model vendor, and AI agent or automation platform and stating what is permitted for each can be. Route the policy through the same approval and review cycle as the enterprise security policy stack rather than creating a parallel governance path, and set an explicit review interval of at most 12 months. Publish a short summary aimed at the population that must follow it, because a policy that requires reading twenty pages to answer a five-second question will be guessed at instead. Establish an exception process with a named approver, a required rationale, a mandatory expiry date and a register, since undocumented exceptions are how policy stacks quietly become fiction.

C) Attest the population and baseline conformance — Run an attestation campaign across the in-scope population, delivered through the existing human resources or learning system so that completion is recorded where compliance evidence already lives. Track coverage as a percentage of the in-scope population rather than as a count of completions, because the denominator is what an auditor asks for. Baseline three conformance measures in the first quarter: the share of obligations on the map carrying a named evidence artifact, the share of the population with a current acknowledgement, and the count of active exceptions with their age distribution. Reconcile the map against the contract register, procurement records and data processing agreements to confirm that what the policy requires is what the environment actually does, and record every divergence as a finding with an owner rather than as a note. Report all three to the executive sponsor quarterly.

Outcome metrics

Success criteria

• A one-page priority compliance map is published, tracing every in-scope obligation to a policy clause, an owner and an evidence artifact.
• The AI Vendor Policy is published with an explicit review interval of at most 12 months and written so that violations are identifiable.
• At least 95% of the in-scope population holds a current acknowledgement.
• An exception process operates with named approvers, mandatory expiry dates and a reviewed register carrying no expired entries.
• Policy requirements are reconciled against the contract register, procurement records and data processing agreements, with every divergence raised as a finding with an owner.

Assessment questions

1. Is there a published one-page priority compliance map tracing every in-scope obligation governing AI/HAI tools and services consumed from third parties to a specific policy clause, an accountable owner, an evidence artifact and the practice that produces it?
2. Is the AI Vendor Policy published with requirements written so conformance is observable, routed through the enterprise policy review cycle, carrying a review interval of at most 12 months, and acknowledged by at least 95% of the in-scope population?
3. Does an exception process operate with named approvers, mandatory rationale and expiry dates and a register carrying no expired entries, and are policy requirements reconciled against the contract register, procurement records and data processing agreements with divergences raised as findings?

Common pitfalls

• ❌ Building a compliance register of hundreds of rows instead of a one-page map, which nobody reads, nobody maintains and no auditor is helped by.
• ❌ Writing policy requirements that cannot be violated, such as requiring AI to be used responsibly, which makes conformance unassessable.
• ❌ Creating a parallel AI policy governance path separate from the enterprise stack, which doubles the review burden and halves the authority.
• ❌ Granting exceptions without expiry dates, after which the exception register becomes the real policy.
• ❌ Publishing the policy without reconciling it against what the environment actually does, so policy and practice diverge from day one.

Level 2, Comprehensive

Calibrate policy depth and evidence requirements to the risk tier of each vendors asset, and assemble tier-specific evidence packs that can be produced on demand.

Activities

A) Calibrate policy requirements per tier — Extend the L1 policy stack with tier-differentiated requirements keyed to the SM L2 rubric. For each requirement, state what Critical, High, Medium and Low each owe: which controls are mandatory versus recommended, what review cadence applies, what approval authority an exception requires, and what evidence must exist. A Critical-tier AI/HAI tools and services consumed from third partie might owe a documented conformance assessment, an annual independent review and a named accountable executive, where a Low-tier one owes an intake record and an annual self-attestation. Publishing the difference is what prevents the two failure modes that follow from uniform policy: either every asset carries Critical-tier burden, which makes the policy unenforceable and drives teams around it, or every asset carries Low-tier burden, which leaves the Critical ones under-governed. Route tier-calibrated requirements through the same approval cycle as the base policy. In this domain tier calibration turns mainly on processor role and transfer basis: a vendor processing regulated data outside an adequacy regime carries obligations that an internal-only drafting tool does not, and the obligation attaches whether or not the organisation negotiated the contract.

B) Build bidirectional control mappings and tier-specific evidence packs — Map each obligation on the priority compliance map to the specific controls that satisfy it, and map each control back to the obligations that depend on it. The reverse direction is the one usually missing and the one that matters operationally: when a control fails or is deferred, the programme needs to state immediately which obligations are now unevidenced, rather than discovering it during the next audit. Assemble an evidence pack per tier specifying exactly which artifacts constitute conformance, drawn from the contract register, procurement records and data processing agreements and from the outputs of TA, SR, DR, IR, ST and ML. Automate collection where the artifact already exists in a system of record, and set a service level for producing a complete pack on request, targeting 10 business days at Critical tier. An evidence pack assembled only when asked for is a pack that will be incomplete the one time it matters. The evidence pack for vendors draws on the contract register, data processing agreements, sub-processor lists and transfer impact assessments, which is the one domain where the evidence is largely written by someone else.

C) Operate regulatory change management and tiered exceptions — Establish a defined path from a regulatory change to the assets it affects. Assign a named owner to monitor the in-scope obligation sources, assess each change for applicability within 30 days of publication, identify the affected assets by tier, and open remediation work items with owners and dates. Without this, regulatory change arrives through whoever happens to read an update, which is neither reliable nor auditable. Calibrate the exception process by tier: a Critical-tier exception requires executive approval, a shorter maximum duration and quarterly review, where a Low-tier exception can be approved by the practice lead with an annual review. Report exception volume and age by tier to the sponsor, and treat a rising Critical-tier exception count as a signal that the policy is mis-calibrated rather than that teams are non-compliant. Regulatory change in this domain most often arrives as transfer-mechanism rulings and value-chain obligations, which can invalidate a contract the organisation signed in good faith and cannot unilaterally fix.

Outcome metrics

Success criteria

• Tier-calibrated policy requirements are published, stating what each tier owes in controls, cadence, approval authority and evidence.
• Evidence packs exist per tier and are refreshed on cadence, with a Critical-tier production service level of 10 business days met.
• Bidirectional control mappings are complete, so a failing control names the obligations it jeopardises.
• A named owner assesses every regulatory change for applicability within 30 days and opens remediation work items by tier.
• Exceptions are tier-calibrated in approval authority, maximum duration and review cadence, with Critical-tier volume and age reported to the sponsor.

Assessment questions

1. Are policy requirements calibrated by the SM L2 risk tier, stating for each tier what controls are mandatory, what review cadence applies, what approval authority an exception requires and what evidence must exist? Evidence this in the Vendors domain against the AI Vendor Policy and the mandatory contract clause set.
2. Do bidirectional control mappings exist so that a failing or deferred control immediately names the obligations it leaves unevidenced, and do tier-specific evidence packs get refreshed on cadence rather than assembled on request? For Vendors, the assessor should be shown the AI Vendor Policy and the mandatory contract clause set.
3. Does a named owner assess every regulatory change for applicability within 30 days of publication and open tier-prioritised remediation work items, and are exceptions tier-calibrated in approval authority, duration and review cadence? In Vendors, ask to see the AI Vendor Policy and the mandatory contract clause set.

Common pitfalls

• ❌ Applying one policy depth to every asset, which either over-burdens Low-tier work until teams route around the policy, or under-governs the Critical-tier assets that justify the programme.
• ❌ Mapping obligations to controls in one direction only, so a deferred control's regulatory consequence is discovered at audit rather than at the deferral decision.
• ❌ Assembling evidence packs only when requested, which guarantees the pack is incomplete on the one occasion it is needed urgently.
• ❌ Leaving regulatory monitoring to whoever happens to read an update, which is neither reliable nor demonstrable to an auditor.
• ❌ Treating a rising Critical-tier exception count as non-compliance by teams rather than as evidence that the policy is mis-calibrated.

Level 3, Industry-Leading

Generate compliance evidence continuously from the operating environment, prove conformance by assertion rather than assembly, and engage in the regulatory process for AI/HAI tools and services consumed from third parties.

Activities

A) Generate conformance evidence continuously — Instrument the environment so that each policy requirement has a continuously evaluated conformance assertion rather than a periodically collected artifact. Draw the signals from the contract register, procurement records and data processing agreements and from the telemetry the other practices already emit, and express each requirement as a check that returns conformant, non-conformant or not-evaluable. The third state matters and is usually omitted: a requirement that cannot be evaluated is a gap in the evidence chain, not a pass, and collapsing it into conformant is how continuous compliance dashboards become reassuring and wrong. Publish a service level covering detection of a conformance drift within 5 business days. Reserve manual attestation for requirements that are genuinely not machine-evaluable, list them explicitly, and review that list quarterly with the aim of shrinking it, since the list is the honest measure of how far the automation actually reaches.

B) Prove conformance on demand and shrink audit preparation — Convert the evidence pack from an assembled document into a generated view over the continuous assertions, so that producing it is a query rather than a project. Target production within one business day at every tier, and run unannounced internal exercises at least twice a year to verify the claim, because an evidence pipeline that has never been exercised under time pressure is an assumption. Track audit preparation effort explicitly in hours and drive it toward zero: the measure of this level is not that audits pass, which they did before, but that they cost almost nothing to prepare for. Where an external audit or regulatory enquiry does raise a finding, trace it back to the assertion that should have caught it and either fix the assertion or record why the requirement is not machine-evaluable.

C) Engage in the regulatory and standards process — Participate in the process that produces the obligations rather than only consuming its output. Submit implementation experience to consultations on GDPR implementing acts and successor guidance, contribute to the standards bodies developing conformance methods including ISO/IEC 42001 and the relevant national standards organisations, and share anonymised implementation patterns through sector bodies and practitioner communities. Implementation experience is the input regulators most consistently lack and most consistently ask for, which makes it the contribution with the highest leverage and the lowest cost to produce, since the programme generates it as a by-product of operating. Target at least four substantive submissions per year, count only those accepted or published, and have legal review every submission before it leaves the organisation.

Outcome metrics

Success criteria

• At least 85% of policy requirements carry a continuously evaluated conformance assertion, with the non-evaluable remainder listed explicitly rather than counted as conformant.
• Conformance drift is detected within 5 business days at the 90th percentile.
• A complete evidence pack is produced within one business day at any tier, verified by unannounced exercise twice a year.
• Audit preparation effort is tracked in hours and falling year over year.
• At least four accepted regulatory or standards contributions per year, each legally reviewed before submission.

Assessment questions

1. Do at least 85% of policy requirements carry a continuously evaluated conformance assertion returning conformant, non-conformant or not-evaluable, with the non-evaluable set listed explicitly and reviewed quarterly for reduction rather than collapsed into conformant? Evidence this in the Vendors domain against the AI Vendor Policy and the mandatory contract clause set.
2. Is a complete evidence pack produced within one business day at any tier, verified by unannounced exercises at least twice a year, and is audit preparation effort tracked in hours and falling year over year? For Vendors, the assessor should be shown the AI Vendor Policy and the mandatory contract clause set.
3. Does the organisation make at least four accepted regulatory or standards contributions per year drawn from its own implementation experience, each reviewed by legal counsel before submission? In Vendors, ask to see the AI Vendor Policy and the mandatory contract clause set.

Common pitfalls

• ❌ Collapsing the not-evaluable state into conformant, which produces a compliance dashboard that is reassuring and wrong.
• ❌ Claiming one-day evidence production without ever exercising it unannounced, which makes the claim an assumption rather than a capability.
• ❌ Automating the assertions that were easy and leaving the non-evaluable list unpublished, so the automation's real reach is unknown.
• ❌ Measuring audit success rather than audit preparation cost, when the latter is what this level actually changes.
• ❌ Submitting to consultations without legal review, which can commit the organisation to positions it has not agreed internally.

Education & Guidance (EG)

Objective: Give the people who select, contract for and administer AI vendors the role-based AI security literacy and the practical guidance they need to make safe decisions without asking, in the Vendors domain.

A program at this level has stopped relying on general security awareness to cover AI-specific risk. Role-based training reaches procurement and sourcing professionals, third-party risk analysts, legal and privacy counsel, and the business owners requesting the tools, and it teaches the failure modes those roles actually encounter rather than a survey of AI risk in general. A guidance library sits where the work happens, so the answer to a routine question is reachable in under a minute rather than requiring a security consultation. Competence is assessed rather than assumed, because completion rates measure attendance and say nothing about whether anyone can now do the thing.

Level 1, Foundational

Give the people who select, contract for and administer AI vendors the role-based AI security literacy and the practical guidance they need to make safe decisions without asking, in the Vendors domain.

Activities

A) Define roles and build role-based curricula — Identify the roles in scope, being procurement and sourcing professionals, third-party risk analysts, legal and privacy counsel, and the business owners requesting the tools, and build a short curriculum per role rather than one course for everyone. A single general course is the most common failure in this practice: it is long enough to be resented, general enough to be forgettable, and calibrated to no one. Each curriculum covers the AI failure modes that role actually meets, which in this domain means AI features silently enabled inside approved SaaS, training-data terms that change on renewal, sub-processor chains extending beyond contractual reach, and data shared with consumer tools on personal accounts, together with the specific controls that role is expected to apply and the escalation path when something looks wrong. Keep each curriculum under 90 minutes, because attention is the binding constraint and a curriculum nobody finishes teaches nothing. Set a 90-day competence deadline for new joiners in scope, and deliver through the existing learning system so that completion and assessment records land where compliance evidence already lives.

B) Assess competence rather than completion — Build an assessment per role that tests whether the learner can apply the material, not whether they sat through it. Include a hands-on exercise where the learner reviews a real vendor's terms and identifies the training-data position, the sub-processor chain and the model-change notification obligation, because a scenario the learner works through is the only assessment format that distinguishes recognition from capability, and recognition is what multiple-choice tests measure. Set a pass threshold, require a retake on failure, and record the pass rate per role as the practice's headline outcome metric. Publish item-level results to the curriculum owner so that a question the whole cohort fails is treated as a teaching failure rather than a learner failure, which is the usual cause. Re-assess at least annually and whenever the threat picture or the sanctioned toolset changes materially.

C) Publish the guidance library where the work happens — Publish a guidance library covering the contract clause library, the intake questionnaire keyed by archetype, and the assessment playbook per vendor archetype. The binding constraint is reachability, not completeness: guidance that requires leaving the working context to find will be guessed around, so place it where the work is, in the repository for engineers, in the workflow tool for process owners, in the service catalogue for requesters. Every guidance item names an owner and a review date, and stale guidance is withdrawn rather than left standing, because out-of-date guidance is worse than none and is trusted for longer. Publish a named route for questions the library does not answer, staffed with a response service level, and mine those questions monthly as the highest-quality signal available about where the guidance is missing or wrong.

Outcome metrics

Success criteria

• Every in-scope role among procurement and sourcing professionals, third-party risk analysts, legal and privacy counsel, and the business owners requesting the tools has a curriculum under 90 minutes covering the failure modes that role actually encounters.
• Competence is assessed by applied scenario rather than recall, with an 80% or better pass rate per role and per-role results published.
• New joiners in scope reach baseline literacy within 90 days, enrolled automatically from the joiner process.
• A guidance library sits where the work happens, with every item carrying an owner and a review date and stale items withdrawn.
• A named question route operates with a published response service level and is mined monthly for guidance gaps.

Assessment questions

1. Does every in-scope role have its own curriculum under 90 minutes covering the AI failure modes that role actually encounters in the Vendors domain, delivered through the existing learning system with a 90-day deadline for new joiners?
2. Is competence assessed by applied scenario rather than recall, with a published pass threshold, an 80% or better pass rate per role, and item-level results reviewed by the curriculum owner so cohort-wide failures are treated as teaching defects? For Vendors, the assessor should be shown the procurement curriculum and one learner's vendor-terms review exercise.
3. Does a guidance library sit where the work happens with every item carrying an owner and a review date, are stale items withdrawn rather than retained, and is there a named question route with a response service level that is mined monthly for gaps? In Vendors, ask to see the procurement curriculum and one learner's vendor-terms review exercise.

Common pitfalls

• ❌ Building one general AI security course for everyone, which is too long to finish, too general to apply, and calibrated to no actual role.
• ❌ Measuring completion rather than competence, which tells the programme who attended and nothing about what anyone can now do.
• ❌ Publishing guidance in a wiki nobody opens, when reachability from the working context is the whole of the control.
• ❌ Leaving stale guidance standing, which is worse than no guidance because it is trusted for longer.
• ❌ Treating cohort-wide assessment failures as a learner problem rather than as evidence the material taught the wrong thing.

Level 2, Comprehensive

Scale Vendors AI security literacy through tier-calibrated depth, a champions network embedded in the teams, and exercises drawn from the organisation's own incidents.

Activities

A) Calibrate curriculum depth to asset tier — Extend the L1 curricula with tier-calibrated depth keyed to the SM L2 rubric. Practitioners working on Critical- and High-tier assets take an extended curriculum with hands-on laboratory work and a higher pass threshold; those on Medium and Low take the baseline. Publish the mapping from asset tier to required curriculum so that assignment is derived rather than negotiated, and wire enrolment to the asset inventory so that a practitioner joining a Critical-tier team is enrolled automatically rather than when someone remembers. Set a currency requirement: Critical-tier practitioners re-assess annually, others every two years. Report pass rates and currency by tier rather than in aggregate, because an organisation-wide 85% pass rate is consistent with the Critical-tier cohort being the weakest, and aggregate reporting actively conceals exactly the case that matters. In this domain the extended curriculum has learners read a real vendor's current terms and locate the training-data position, the sub-processor chain and the model-change notification obligation, because most practitioners have never actually read one and assume protections that are not there.

B) Build and resource a champions network — Establish a security champion inside each team working on Critical- and High-tier assets. The champion is not a security professional and should not be treated as one: the role is to be the person who recognises when something needs the central function, to answer the routine questions locally, and to carry context the central team cannot have. Resource the role explicitly with allocated time, typically half a day a week, agreed with the engineering or business management chain rather than assumed as goodwill, because an unresourced champion network decays within two quarters and its decay is invisible until it is total. Give champions a deeper curriculum, a private channel to the central team, early access to threat and guidance updates, and visible recognition in performance processes. Meet the network monthly and mine it for what the central function cannot see. Vendor champions sit within procurement, legal and the business units requesting tools, and their distinguishing value is hearing about a tool at the point of desire rather than at the point of invoice.

C) Run exercises from the organisation's own incidents — Convert incidents and near-misses from the issue-management practice into teaching material on a defined cadence, targeting one new exercise per quarter. An exercise built from something that actually happened in this organisation carries an authority that no generic example achieves, and it converts the incident from a cost into an asset. Anonymise where necessary but keep the specifics that make it recognisable, since the recognisable detail is the part that is remembered. Run the exercises with the champions network first, then let champions run them with their teams, which both scales delivery and tests whether the champion can teach the material rather than only pass an assessment. Track which failure modes recur across exercises and feed that back to the threat assessment and guidance practices as evidence of where understanding is genuinely weak. Vendor exercises are built from real cases where an approved product silently enabled an AI feature or changed its terms at renewal, which is the failure mode practitioners most consistently disbelieve until shown.

Outcome metrics

Success criteria

• Curriculum depth is calibrated to asset tier with enrolment derived from the inventory, and pass rates are reported per tier rather than in aggregate.
• Every team working on Critical-tier assets has a trained champion with management-agreed allocated time.
• At least four exercises per year are built from the organisation's own incidents and near-misses.
• Champions resolve at least 60% of routine questions without central escalation.
• Recurring failure modes across exercises are fed back to threat assessment and guidance as evidence of genuine weakness.

Assessment questions

1. Is curriculum depth calibrated to the SM L2 asset tier with enrolment derived automatically from the inventory, and are pass rates and currency reported per tier rather than in an aggregate that could conceal a weak Critical-tier cohort? Evidence this in the Vendors domain against the procurement curriculum and one learner's vendor-terms review exercise.
2. Does every team working on Critical-tier assets have a trained champion with explicitly allocated time agreed with the management chain, and do champions resolve at least 60% of routine questions without central escalation? For Vendors, the assessor should be shown the procurement curriculum and one learner's vendor-terms review exercise.
3. Are at least four exercises per year built from the organisation's own incidents and near-misses, run first with champions and then by champions with their teams, with recurring failure modes fed back to threat assessment and guidance? In Vendors, ask to see the procurement curriculum and one learner's vendor-terms review exercise.

Common pitfalls

• ❌ Reporting one organisation-wide pass rate, which is consistent with the Critical-tier cohort being the weakest and conceals precisely that case.
• ❌ Appointing champions without allocated time, after which the network decays within two quarters and its decay is invisible until total.
• ❌ Treating champions as junior security staff rather than as embedded context-holders, which drives away the people best suited to the role.
• ❌ Using generic exercise material when the organisation's own incidents are available, discarding the authority that recognisable detail carries.
• ❌ Anonymising exercises so thoroughly that the recognisable specifics are gone, which removes the part learners actually remember.

Level 3, Industry-Leading

Make Vendors AI security learning continuous and evidence-driven, measure whether it changes behaviour rather than knowledge, and contribute curricula back to the field.

Activities

A) Drive curriculum and guidance from observed signal — Close the loop between what the organisation observes and what it teaches. Feed incident root causes, design and implementation review findings, security test failures and monitoring detections into a quarterly curriculum review, and treat any failure mode appearing three times as a mandatory curriculum change rather than a discussion item. Instrument the guidance library so that search terms returning nothing, and the questions reaching the named route, are captured as gap signals, because what practitioners look for and fail to find is the highest-quality guidance-gap data available and it is free. Publish a service level covering the interval from an observed recurring failure mode to updated guidance, targeting 30 days. Retire material that the signal shows is no longer relevant, since a curriculum that only grows becomes one nobody completes. In this domain the extended curriculum has learners read a real vendor's current terms and locate the training-data position, the sub-processor chain and the model-change notification obligation, because most practitioners have never actually read one and assume protections that are not there.

B) Measure behaviour change rather than knowledge — Move the headline measure from assessment scores to behaviour in the environment. Compare teams and cohorts before and after training on measures the other practices already produce: the rate of findings by category in implementation review, the reuse rate of sanctioned patterns, the proportion of designs arriving at review already conformant, the rate of self-reported near-misses, which should rise as psychological safety improves and is one of the few metrics where an increase is good news. Accept that attribution is imperfect and say so, rather than claiming a causal link the data cannot carry; the honest claim is that behaviour moved in the trained cohort and did not in the untrained one, over a stated period. Report behaviour measures alongside assessment results so that a high pass rate with unchanged behaviour is visible as the failure it represents. Vendor champions sit within procurement, legal and the business units requesting tools, and their distinguishing value is hearing about a tool at the point of desire rather than at the point of invoice.

C) Contribute curricula and measure capability in progression — Contribute at least four substantive artifacts per year to the field, weighted toward teaching material: role-based AI security curricula, applied exercise and laboratory scenarios, anonymised incident case studies that other organisations can teach from, and guidance patterns. Good AI security teaching material is the field's scarcest resource, far scarcer than frameworks or tooling, which makes curriculum contribution unusually high-leverage and unusually likely to be adopted. Work through OWASP education projects, sector bodies, professional associations and academic partnerships. Separately, establish AI security capability as a recognised factor in performance and progression for in-scope roles, agreed with human resources, because a capability the organisation says matters but never recognises is one practitioners correctly deprioritise. Vendor exercises are built from real cases where an approved product silently enabled an AI feature or changed its terms at renewal, which is the failure mode practitioners most consistently disbelieve until shown.

Outcome metrics

Success criteria

• Curricula and guidance update from observed incidents, findings and telemetry, with recurring failure modes reaching updated guidance within 30 days.
• Guidance gaps are detected from search misses and unanswered questions rather than from periodic review alone.
• At least three behaviour measures are tracked pre and post training, reported alongside assessment results with attribution limits stated.
• At least four externally accepted curriculum or exercise contributions per year.
• AI security capability is a recognised factor in performance and progression for every in-scope role, agreed with human resources.

Assessment questions

1. Do curricula and guidance update from incident root causes, review findings, test failures and detections, with any failure mode appearing three times triggering a mandatory curriculum change within 30 days, and are guidance gaps detected from search misses and unanswered questions? Evidence this in the Vendors domain against the procurement curriculum and one learner's vendor-terms review exercise.
2. Does the programme measure behaviour change in the environment using at least three downstream measures tracked pre and post training, reported alongside assessment results with attribution limits stated rather than a causal claim the data cannot carry? For Vendors, the assessor should be shown the procurement curriculum and one learner's vendor-terms review exercise.
3. Does the organisation contribute at least four externally accepted curriculum or exercise artifacts per year, and is AI security capability a recognised factor in performance and progression for every in-scope role? In Vendors, ask to see the procurement curriculum and one learner's vendor-terms review exercise.

Common pitfalls

• ❌ Reviewing curricula annually while the threat picture moves quarterly, so the material lags by a planning cycle permanently.
• ❌ Adding material without ever retiring any, producing a curriculum too long to complete and therefore completed by nobody.
• ❌ Claiming training caused a behaviour improvement when the data supports only correlation, which discredits the honest measures alongside it.
• ❌ Ignoring search misses and unanswered questions, which are the cheapest and highest-quality guidance-gap signal available.
• ❌ Declaring AI security capability important while never recognising it in progression, which practitioners correctly read as a statement of priority.

Business Function: Building

Threat Assessment (TA)

Objective: Publish an archetype-level threat library for the Vendors domain, tagged to the HAI TTPs and MITRE ATLAS, so threat modelling starts from a reusable baseline rather than a blank page.

A program at this level has written down what can go wrong with each vendors archetype, once, so that no team has to rediscover it. The library covers consumer generative AI, AI-embedded SaaS, AI coding assistant, AI API and foundation-model vendor, and AI agent or automation platform, and every entry carries a HAI TTP tag, an ATLAS technique where one applies, an OWASP LLM reference and the practice expected to address it. This is what makes threat modelling tractable at scale: a team adopting a known archetype inherits its threat model and spends its effort on what is genuinely novel about their case, rather than re-deriving prompt injection from first principles every time.

Level 1, Foundational

Publish an archetype-level threat library for the Vendors domain, tagged to the HAI TTPs and MITRE ATLAS, so threat modelling starts from a reusable baseline rather than a blank page.

Activities

A) Build the archetype threat library — Author one threat snapshot per archetype covering consumer generative AI, AI-embedded SaaS, AI coding assistant, AI API and foundation-model vendor, and AI agent or automation platform. For each archetype, identify the entities and trust boundaries first, being the data the organisation sends, the vendor's processing, the sub-processor chain behind it, and the outputs returned into organisational workflows, because a threat list without a boundary model is a list of worries rather than an analysis. Then enumerate the threats that follow from those boundaries, which in this domain centre on AI features silently enabled inside an approved product, training-data terms changing at renewal, sub-processor chains extending past contractual reach, and a vendor model change altering behaviour the organisation had tested. Keep each snapshot to a length a team will actually read, targeting two pages, and write each threat as a concrete failure with a named precondition rather than as a risk category. Derive the snapshots from contracts, data processing agreements, vendor documentation and the vendor inventory and validate each with one practitioner who works on that archetype daily, because a threat library authored entirely by security will describe the system security imagines rather than the one that exists.

B) Tag every threat to the shared vocabulary — Tag each threat entry with the HAI TTP it expresses, being excessive agency, agent goal hijack, tool misuse or rogue agents; the MITRE ATLAS technique where a real one applies; the OWASP LLM 2025 reference; and the compliance obligation it bears on where relevant. Use only technique identifiers that genuinely exist, because a plausible but invented identifier is worse than none, it survives review and misleads everyone downstream. Where no ATLAS technique fits, record that explicitly rather than forcing the nearest match, and treat the gap as a candidate contribution at L3. The tagging is what makes the library composable with everything else: it lets monitoring ask which threats have detections, testing ask which have tests, and requirements ask which have controls, without any of them re-reading the prose.

C) Trace every threat to a treating practice and publish the gaps — For every threat in the library, name the practice expected to address it and the specific control, requirement, pattern, test or detection that does so. Then publish the gaps, which is the activity teams most often skip and the one that gives the library its value: a threat with no named treatment is an accepted risk whether or not anyone decided to accept it, and making that visible is the point of the exercise. Report the treatment-coverage rate to the sponsor quarterly and treat an untreated threat at a high-risk archetype as an open finding with an owner rather than as a line in a register. Establish a 30-day service level for producing a threat snapshot when a new archetype enters the inventory, and review the whole library at least quarterly against external threat intelligence.

Outcome metrics

Success criteria

• A threat snapshot exists for at least 90% of inventoried vendors archetypes, each opening with entities and trust boundaries rather than a threat list.
• Every threat entry carries a HAI TTP tag and, where applicable, a genuine ATLAS technique identifier, with no invented references.
• At least 85% of threats trace to a named treating practice and specific control, with the untreated remainder published as gaps rather than omitted.
• Untreated threats at high-risk archetypes carry an owner and a remediation date.
• A 30-day snapshot service level operates for new archetypes, triggered from the inventory.

Assessment questions

1. Does a published threat snapshot exist for at least 90% of inventoried vendors archetypes, each establishing entities and trust boundaries before enumerating threats, validated with a practitioner who works on that archetype?
2. Does every threat entry carry a HAI TTP tag, a genuine MITRE ATLAS technique where one applies with gaps recorded rather than forced, an OWASP LLM reference and the relevant compliance obligation? For Vendors, the assessor should be shown the threat snapshot for the AI API vendor archetype, with its sub-processor chain.
3. Are at least 85% of threats traced to a named treating practice and specific control, with untreated threats published as gaps and those at high-risk archetypes carrying an owner and remediation date? In Vendors, ask to see the threat snapshot for the AI API vendor archetype, with its sub-processor chain.

Common pitfalls

• ❌ Writing a threat list without first drawing entities and trust boundaries, which produces a list of worries rather than an analysis anyone can act on.
• ❌ Inventing plausible ATLAS technique identifiers where none fit, which survives review and misleads every downstream practice that trusts the tag.
• ❌ Authoring the library entirely within security, which describes the system security imagines rather than the one practitioners operate.
• ❌ Producing the library and never publishing the treatment gaps, which leaves threats accepted by default without anyone deciding to accept them.
• ❌ Writing snapshots so long that teams do not read them, after which the library is maintained for an audience of nobody.

Level 2, Comprehensive

Model threats per asset at a depth calibrated to its risk tier, refresh models on material change, and keep the Vendors threat picture current against real adversary behaviour.

Activities

A) Calibrate threat-model depth to asset tier — Publish what each tier receives. Critical: a per-asset threat model, an adversarial overlay walking the relevant ATLAS tactics end to end, an explicit abuse-case set, and a named architect reviewer. High: the archetype model plus documented per-asset deltas and an ATLAS tactic walk. Medium: the archetype model with deltas recorded. Low: the archetype model unchanged. Publishing this is what stops the two failure modes of uniform depth, namely modelling everything shallowly so the Critical assets are under-analysed, or modelling everything deeply so the practice becomes a bottleneck and teams route around it. Set turnaround service levels per tier and report adherence, because a threat-modelling practice that cannot say how long it takes will be worked around by teams under delivery pressure whatever its analytical quality. Per-asset modelling in this domain means walking the data path into the vendor and the sub-processor chain behind it, because the organisation's threat surface extends to parties it has no relationship with and often cannot name, which is the specific thing an archetype model cannot capture.

B) Refresh on material change rather than on the calendar — Define material change per domain and wire the trigger to the signals that already exist. In this domain a material change includes a new capability or interface on the asset, a change to the data classes it handles, a model or provider change, a change in exposure or the population reached, and a tier increase from the SM rubric. Each triggers a re-model inside the tier's window, being 14 days at Critical and 30 at High. Annual review remains as a floor, not as the mechanism; the calendar is the least informative trigger available and relying on it means an asset that changed materially in month two carries a stale model for ten months. Record every refresh with what changed and what it changed in the model, so that a reviewer can see whether the re-model was substantive or a date stamp. In this domain the material change that most often invalidates a model is a vendor model change or a sub-processor addition, which arrive as a notification email if the contract requires one and as nothing at all if it does not.

C) Feed external intelligence in and push threat output downstream — Consume external threat intelligence on a defined cadence, drawing from MITRE ATLAS updates, OWASP LLM and Agentic project output, vendor and provider advisories, sector information-sharing bodies and published incident analyses. Assess each item for applicability to the archetypes in the library within 30 days, and record the assessment even when the answer is not applicable, because the record is what demonstrates the cadence is real. Push threat-model output downstream mechanically rather than by expectation: each Critical- and High-tier threat generates a requirement in the SR pack, a test case in the ST battery and a detection candidate in the ML set, tracked to closure. A threat model whose output is not visible in what gets tested and detected has produced a document, and the practice is then measured on documents. Vendor threat output lands most usefully in the contract clause set and the gateway filtering rules, since those are the only two places where an organisation can actually act on a threat it does not control.

Outcome metrics

Success criteria

• Threat-model depth per tier is published with turnaround service levels, and adherence is at or above 90% per tier.
• Every Critical-tier asset has a per-asset model with an adversarial ATLAS overlay and a named architect reviewer.
• Material change triggers a re-model inside the tier window, with refresh records stating what actually changed.
• External intelligence is assessed for applicability within 30 days, with negative assessments recorded.
• At least 90% of Critical- and High-tier threats generate a tracked requirement, test case and detection candidate.

Assessment questions

1. Is threat-model depth calibrated by asset tier with published turnaround service levels, such that Critical-tier assets receive a per-asset model with an adversarial ATLAS overlay and a named architect reviewer while Low-tier assets inherit the archetype snapshot? Evidence this in the Vendors domain against the threat snapshot for the AI API vendor archetype, with its sub-processor chain.
2. Does material change, including a new capability, a data-class change, a model or provider change, an exposure change or a tier increase, trigger a re-model within 14 days at Critical and 30 at High, with refresh records stating what actually changed rather than only a date? For Vendors, the assessor should be shown the threat snapshot for the AI API vendor archetype, with its sub-processor chain.
3. Is external threat intelligence assessed for applicability within 30 days with negative assessments recorded, and do at least 90% of Critical- and High-tier threats generate a tracked requirement, test case and detection candidate? In Vendors, ask to see the threat snapshot for the AI API vendor archetype, with its sub-processor chain.

Common pitfalls

• ❌ Relying on annual review as the refresh mechanism, which leaves an asset that changed in month two carrying a stale model for ten months.
• ❌ Modelling every asset at the same depth, which either under-analyses the Critical assets or turns the practice into a bottleneck teams route around.
• ❌ Recording refreshes as date stamps without stating what changed, which makes it impossible to tell a substantive re-model from a compliance gesture.
• ❌ Consuming threat intelligence without recording negative assessments, so there is no evidence the cadence is actually operating.
• ❌ Ending the threat model at the document, so nothing in testing or detection changes and the practice is judged on paperwork.

Level 3, Industry-Leading

Drive Vendors threat modelling from live signal and adversary emulation, validate models against what actually happens, and contribute observed techniques back to ATLAS and the community.

Activities

A) Validate models through adversary emulation — Run adversary emulation against Critical-tier assets at least semi-annually, scoped from the threat model and executed by a team that did not author it. Measure the two directions of error explicitly: threats the model predicted that emulation could not demonstrate, and behaviours emulation found that the model did not anticipate. The second is the more valuable and the more uncomfortable, and a practice that only reports confirmations is not learning. Record for every threat in the library whether it has been demonstrated in this environment, demonstrated elsewhere, or remains theoretical, and report the distribution, since a library where most entries are theoretical is a library built from reading rather than from testing. Feed every unanticipated finding into the archetype library, not only into the asset's model, so that one exercise improves every asset sharing that archetype. Per-asset modelling in this domain means walking the data path into the vendor and the sub-processor chain behind it, because the organisation's threat surface extends to parties it has no relationship with and often cannot name, which is the specific thing an archetype model cannot capture.

B) Update models from live signal continuously — Close the loop from operations back to the models. Feed detection telemetry, incident root causes, security test failures and near-misses into the library continuously rather than at review time, and treat any observed technique not represented in the library as a defect in the library with an owner and a date. Publish a service level covering the interval from an observed technique to an updated model, targeting 30 days. Track the share of incidents whose root cause was already present in the library as an untreated threat, which is the single most informative metric this practice produces: it separates a failure to foresee, which is hard, from a failure to act on what was already written down, which is not, and the second is far more common than most programmes admit. In this domain the material change that most often invalidates a model is a vendor model change or a sub-processor addition, which arrive as a notification email if the contract requires one and as nothing at all if it does not.

C) Contribute observed techniques back — Submit techniques observed in this environment to MITRE ATLAS, and contribute to OWASP LLM and Agentic project revisions, sector information-sharing bodies and published incident analyses. This matters more in AI security than in most fields because the public technique catalogue is young, and the gap between what practitioners observe and what is publicly documented is wide, which means a single well-documented submission can be genuinely novel rather than incremental. Target at least four substantive submissions per year, count only those accepted, anonymise and legally review every one, and coordinate disclosure with affected model providers or vendors where a technique touches their systems. Where the L1 activity recorded threats for which no ATLAS technique fit, revisit that list first: it is the highest-yield source of contribution candidates the programme has. Vendor threat output lands most usefully in the contract clause set and the gateway filtering rules, since those are the only two places where an organisation can actually act on a threat it does not control.

Outcome metrics

Success criteria

• Every Critical-tier asset is exercised by adversary emulation at least annually, by a team that did not author the model.
• Every threat in the library is classified as demonstrated here, demonstrated elsewhere or theoretical, with the theoretical share reported and falling.
• Emulation findings the model did not anticipate are fed back into the archetype library within 30 days, improving every asset sharing that archetype.
• Incidents whose root cause was an already-documented untreated threat are reported openly and falling year over year.
• At least four accepted external technique contributions per year, anonymised, legally reviewed and disclosure-coordinated.

Assessment questions

1. Is every Critical-tier asset exercised by adversary emulation at least annually by a team that did not author the model, with both error directions measured, including behaviours the model failed to anticipate? Evidence this in the Vendors domain against the threat snapshot for the AI API vendor archetype, with its sub-processor chain.
2. Is every library threat classified as demonstrated here, demonstrated elsewhere or theoretical, and is the share of incidents whose root cause was an already-documented untreated threat reported openly and falling? For Vendors, the assessor should be shown the threat snapshot for the AI API vendor archetype, with its sub-processor chain.
3. Does the organisation contribute at least four accepted techniques per year to ATLAS and the community, anonymised, legally reviewed and disclosure-coordinated, drawing first on the threats for which no existing technique fitted? In Vendors, ask to see the threat snapshot for the AI API vendor archetype, with its sub-processor chain.

Common pitfalls

• ❌ Reporting only the threats emulation confirmed, which converts a learning exercise into a validation ritual.
• ❌ Feeding emulation findings into the single asset's model rather than the archetype library, so one exercise improves one asset instead of all of them.
• ❌ Not tracking incidents whose root cause was already in the library untreated, which conceals the difference between failing to foresee and failing to act.
• ❌ Leaving the unmapped-technique list from L1 unrevisited, discarding the best contribution candidates the programme has.
• ❌ Submitting techniques that touch a provider's systems without coordinating disclosure, which damages the relationships the programme depends on.

Security Requirements (SR)

Objective: Publish a base security requirements pack per vendors archetype, every requirement carrying a testable acceptance criterion, so teams inherit requirements rather than negotiating them.

A program at this level has written down what each vendors archetype must do, once, in a form a test can check. The pack covers consumer generative AI, AI-embedded SaaS, AI coding assistant, AI API and foundation-model vendor, and AI agent or automation platform, and every requirement carries an acceptance criterion specific enough that two reviewers assessing the same asset reach the same verdict. This is the practice that converts threat-model output into something buildable: a threat without a requirement is an observation, and a requirement without a testable criterion is an aspiration. Requirements trace back to the threats they mitigate and forward to the tests that verify them.

Level 1, Foundational

Publish a base security requirements pack per vendors archetype, every requirement carrying a testable acceptance criterion, so teams inherit requirements rather than negotiating them.

Activities

A) Author the base requirements pack per archetype — Write one base pack per archetype covering consumer generative AI, AI-embedded SaaS, AI coding assistant, AI API and foundation-model vendor, and AI agent or automation platform. The pack states what any asset of that archetype must do, drawn from the TA threat library so that every requirement answers a specific threat rather than a general anxiety. In this domain the base requirements centre on a contractual training-data prohibition, model-change notification with a stated window, sub-processor notification and approval rights, incident-notification windows, data residency commitments, deletion on termination, and audit or evidence rights proportionate to the tier. Keep the base pack deliberately small, targeting fifteen to twenty-five requirements: a pack of two hundred requirements is negotiated down in practice, and the negotiation happens invisibly at the team's discretion rather than visibly at the programme's. Write each requirement as an obligation on the system rather than on the team, because a requirement phrased as an activity is satisfied by doing the activity badly. Version the packs and publish the change history so teams can see what changed and when.

B) Attach a testable acceptance criterion to every requirement — Give every requirement an acceptance criterion that states how conformance is demonstrated and by what evidence. The criterion must be specific enough that two reviewers assessing the same asset independently reach the same verdict, which is the operational test of whether a criterion is real. Prefer criteria a test can evaluate over criteria a person must judge, and where judgment is unavoidable, state the evidence the judgment rests on. Requirements that cannot be given a testable criterion are either too vague to be useful or belong in guidance rather than requirements, and moving them is better than leaving them to be argued about at every review. Map each criterion to the practice that will verify it, being implementation review, security testing or environment hardening, so that the verification path exists before the requirement is issued.

C) Establish traceability and declare the consumption envelope — Build a requirements evidence map linking each requirement backward to the TA threat it mitigates and forward to the test case, review check or detection that verifies it. Publish the map and treat a requirement with no verifying mechanism as an open gap, because an unverified requirement is a statement of intent that the programme has no way to know is met. Separately, require every archetype to declare a consumption envelope: a spend or volume cap appropriate to its expected workload, and an explicit behaviour when that cap is reached, being degrade, queue or refuse, and never silently continue. Stating the cap as a requirement rather than an operational setting is what makes it testable at per active seat and reviewable at design time, and it is the point at which unbounded consumption stops being an operational surprise and becomes a design decision.

Outcome metrics

Consumption metrics — Archetypes whose requirements pack declares a consumption cap and an explicit behaviour at the cap per per archetype requirements pack · Declared consumption envelope per unit of work, stated per archetype per per active seat · Consumption-cap requirements carrying a testable acceptance criterion per per consumption requirement

Success criteria

• A base requirements pack exists for every inventoried vendors archetype, deliberately small at fifteen to twenty-five requirements.
• Every requirement carries an acceptance criterion specific enough that independent reviewers reach the same verdict, verified by double-review calibration at 85% or better.
• A requirements evidence map traces at least 90% of requirements backward to a threat and forward to a verifying mechanism, with gaps published.
• Every archetype declares a consumption envelope with an explicit behaviour at the cap, expressed per unit of work.
• Requirements are phrased as obligations on the system rather than activities for the team.

Assessment questions

1. Does a base security requirements pack exist for every inventoried vendors archetype, deliberately small, derived from the TA threat library, and phrased as obligations on the system rather than activities for the team?
2. Does every requirement carry an acceptance criterion specific enough that two reviewers independently reach the same verdict, verified by a periodic blind double-review calibration exercise at 85% agreement or better? For Vendors, the assessor should be shown the vendor requirements pack and its training-data prohibition clause.
3. Does a requirements evidence map trace at least 90% of requirements backward to a threat and forward to a verifying mechanism with gaps published, and does every archetype declare a consumption envelope with an explicit, testable behaviour at the cap? In Vendors, ask to see the vendor requirements pack and its training-data prohibition clause.

Common pitfalls

• ❌ Writing packs of two hundred requirements, which get negotiated down invisibly at each team's discretion rather than visibly at the programme's.
• ❌ Phrasing requirements as activities for the team rather than obligations on the system, which are satisfied by performing the activity badly.
• ❌ Accepting acceptance criteria that two reviewers would read differently, which turns every review into a negotiation about what the requirement meant.
• ❌ Issuing requirements with no verifying mechanism, so the programme has no way to know whether they are met.
• ❌ Leaving the consumption cap to operations, which makes unbounded spend an operational surprise rather than a design decision.

Level 2, Comprehensive

Calibrate vendors requirements packs to asset tier, maintain a per-asset requirements evidence map, and make conformance demonstrable rather than asserted.

Activities

A) Derive requirements scope from tier — Publish the mapping from asset tier to requirements scope: which requirements are mandatory at each tier, which are recommended, and which tier-specific additions apply above the base pack. Derive the scope from the inventory tier automatically rather than letting each team select, because scope selection under delivery pressure reliably produces the minimum defensible set and does so invisibly. Add tier-specific requirements where the tier warrants them, such as an independent conformance assessment and dual approval at Critical. Where a team believes a requirement does not apply, that is a deviation and goes through the deviation process rather than through scope interpretation, which is the distinction that keeps the tier mapping meaningful rather than advisory. Tier scope in this domain keys chiefly on data exposure rather than contract value, because the two correlate weakly and a cheap tool can carry the entire customer base through it.

B) Maintain a per-asset requirements evidence map — For every Critical- and High-tier asset, maintain a requirements evidence map recording per requirement its status, the evidence that demonstrates conformance, where that evidence lives, its evidence grade from E0 to E3, and its last verification date. The map is maintained continuously as the asset is built rather than assembled at review, which is what makes design review cheap and audit response fast. Wire evidence collection to the systems that already hold it, being the code repository, the CI pipeline, the test registry and the configuration store, so that the map reflects reality rather than the last time someone updated it. Report the distribution of evidence grades per asset, since an asset whose conformance rests entirely on E0 assertions is in a materially different position from one resting on E2 telemetry, and a single conformance percentage conceals that difference entirely. Evidence for vendor requirements is largely written by the vendor, which is why contractual audit and evidence rights are themselves a requirement and their absence is a finding rather than a preference.

C) Govern deviations and verify consumption caps per tier — Operate a deviation process: any requirement not met is recorded with a rationale, a compensating control, a named approver whose authority scales with tier, and a mandatory expiry date. Review the deviation register monthly, report Critical-tier deviations to the sponsor, and treat a deviation past expiry as an open finding. Deviations are healthy when visible and corrosive when absorbed, and the register is what determines which. Separately, make consumption caps tier-proportionate and verify them rather than declaring them: each Critical- and High-tier asset's cap is expressed relative to per active seat, and the at-cap behaviour is exercised at least annually so that what happens at the limit is known before it happens in production. Record the exercise result, because a cap whose behaviour has never been observed is a cap nobody should rely on. Cap exercises in this domain are run at the broker, since vendor-side limits are often unavailable and the broker is the only enforcement point the organisation controls.

Outcome metrics

Consumption metrics — Critical- and High-tier assets whose consumption cap behaviour has been exercised and recorded per per Critical or High tier asset · Declared consumption envelope against observed consumption, per tier per per active seat · Deviations granted against a consumption requirement, with compensating control recorded per per active deviation

Success criteria

• Requirements scope is derived automatically from asset tier, and scope disagreements route through the deviation process rather than through interpretation.
• At least 95% of Critical-tier assets carry a current per-asset requirements evidence map maintained continuously rather than assembled at review.
• Evidence grades are reported per asset, with at least 60% of Critical-tier requirements resting on E2 evidence or better.
• Every deviation carries rationale, compensating control, a tier-appropriate approver and an expiry, with none past expiry.
• Every Critical-tier asset's consumption cap behaviour has been exercised within 12 months and the result recorded.

Assessment questions

1. Is requirements scope derived automatically from the SM L2 asset tier, with disagreements routed through a deviation process rather than resolved as scope interpretation under delivery pressure? Evidence this in the Vendors domain against the vendor requirements pack and its training-data prohibition clause.
2. Does every Critical- and High-tier asset carry a per-asset requirements evidence map maintained continuously from repository, CI and configuration signals, reporting an evidence grade per requirement rather than a single conformance percentage? For Vendors, the assessor should be shown the vendor requirements pack and its training-data prohibition clause.
3. Do all deviations carry rationale, compensating control, tier-appropriate approver and expiry with none past expiry, and has every Critical-tier asset's consumption cap behaviour been exercised and recorded within the last 12 months? In Vendors, ask to see the vendor requirements pack and its training-data prohibition clause.

Common pitfalls

• ❌ Letting teams select their own requirements scope, which reliably produces the minimum defensible set and does so invisibly.
• ❌ Assembling evidence maps at review time, which makes every review expensive and every audit a project.
• ❌ Reporting a single conformance percentage, which conceals whether conformance rests on assertion or on telemetry.
• ❌ Granting deviations without expiry, after which the deviation register becomes the real requirements set.
• ❌ Declaring consumption caps without ever exercising them, so their behaviour at the limit is first learned during an incident.

Level 3, Industry-Leading

Express vendors requirements as executable policy evaluated continuously in the pipeline, so conformance is enforced at merge rather than assessed at review.

Activities

A) Express requirements as executable policy in the pipeline — Convert every machine-evaluable requirement into a policy check that runs in the integration delivery pipeline and broker configuration path, evaluating the change rather than the documentation. Blocking severity follows asset tier: a Critical-tier breach fails the integration change, a Low-tier breach warns. Agree the blocking behaviour with engineering before enabling it, because a security check that blocks the integration change without an agreed escalation path is a check that gets disabled within a month and takes the practice's credibility with it. Publish the list of requirements that are not machine-evaluable and review it quarterly with the aim of shrinking it, since that list is the honest measure of how far enforcement actually reaches. Provide a documented break-glass path with automatic notification and mandatory post-hoc review, because emergencies happen and an unbreakable gate gets bypassed in ways nobody sees. Vendor requirements are the least machine-evaluable in the model, because the obligations live in contracts rather than configuration; what can be checked is the integration path, the payload fields and the gateway route, and the rest must be listed as unevaluable rather than assumed.

B) Evolve requirements from operational evidence — Close the loop from operations to requirements. Feed incident root causes, security test failures, implementation review findings and monitoring detections into a quarterly requirements review, and treat any failure mode appearing three times without a covering requirement as a mandatory pack change. Publish a service level covering the interval from a recurring failure mode to a new or amended requirement, targeting 60 days including the policy implementation. Retire requirements that operational evidence shows are no longer load-bearing, because a pack that only grows becomes one teams route around; the willingness to remove requirements is what keeps the pack small enough to be respected. Record for each requirement how many times its policy check has actually fired, which distinguishes requirements that are doing work from those that are decoration. The recurring failure modes that drive new vendor requirements are a field quietly added to a vendor payload and a direct integration added beside the gateway for latency, both introduced by engineers solving a real problem.

C) Contribute executable requirement packs back — Contribute at least four substantive artifacts per year, weighted toward executable form: requirement packs with their policy implementations, acceptance criteria that others can adopt directly, and conformance check libraries. Executable contributions are worth substantially more than prose in this practice, because a peer organisation can run a policy check and merely read a requirement, and the adoption difference between the two is large. Work through OWASP projects, OpenSSF AI, sector bodies and the relevant standards communities. Count only accepted or published artifacts, review each legally before release, and prefer contributing the checks that fire most often in this environment, since those are the ones carrying the most operational evidence behind them. Vendor requirement packs contribute well as clause libraries, which peers can take to their own negotiations, and this is one of the few places where sharing materially shifts supplier behaviour across an industry.

Outcome metrics

Consumption metrics — Consumption-cap requirements enforced as pipeline policy rather than asserted in a document per per consumption requirement · Changes blocked at merge for breaching a declared consumption envelope per per active seat · Consumption-cap policy checks that have never fired per per consumption policy check

Success criteria

• At least 80% of machine-evaluable requirements run as pipeline policy checks, with the non-evaluable remainder listed explicitly and shrinking.
• Critical-tier breaches are blocked at the integration's merge gate and at the broker configuration change, with a documented break-glass path carrying automatic notification and mandatory post-hoc review within 5 business days.
• Recurring failure modes reach a new or amended requirement, including its policy implementation, within 60 days.
• Requirements are retired as well as added, with firing counts reviewed annually and never-firing requirements examined.
• At least four accepted external contributions per year, the majority in executable rather than prose form.

Assessment questions

1. Do at least 80% of machine-evaluable requirements run as pipeline policy checks with tier-proportionate blocking severity agreed with engineering, and is the non-machine-evaluable remainder listed explicitly and reviewed quarterly for reduction? Evidence this in the Vendors domain against the vendor requirements pack and its training-data prohibition clause.
2. Do recurring failure modes from incidents, test failures and review findings reach a new or amended requirement including its policy implementation within 60 days, and are requirements retired as well as added based on firing counts? For Vendors, the assessor should be shown the vendor requirements pack and its training-data prohibition clause.
3. Is there a documented break-glass path with automatic notification and mandatory post-hoc review within 5 business days, and does the organisation contribute at least four accepted artifacts per year, the majority executable rather than prose? In Vendors, ask to see the vendor requirements pack and its training-data prohibition clause.

Common pitfalls

• ❌ Enabling blocking checks without an agreed escalation path, after which the checks are disabled within a month and take the practice's credibility with them.
• ❌ Automating the easy requirements and leaving the non-evaluable list unpublished, so nobody knows how far enforcement actually reaches.
• ❌ Adding requirements without ever retiring any, producing a pack large enough that teams route around it.
• ❌ Tolerating high false-positive rates on policy checks, which trains engineers to bypass rather than to fix.
• ❌ Contributing requirement prose rather than executable checks, which peers read and do not adopt.

Secure Architecture (SA)

Objective: Publish reference architecture patterns and an antipattern catalogue for each vendors archetype, so the secure design is the default path rather than an achievement.

A program at this level has drawn the secure shape of each vendors archetype once, so teams adopt rather than invent. The library covers the brokered-access pattern routing vendor AI through an organisational gateway, the data-minimisation pattern filtering before egress to the vendor, the fallback pattern surviving vendor unavailability, and the exit pattern keeping data portable by construction, and it is paired with an antipattern catalogue naming the shapes that recur and what to replace them with. An antipattern catalogue is the more useful half and the more commonly missing one: teams recognise what they are building far more readily than they recognise what they should have built. Every pattern states which threats it mitigates, so adoption is an argument rather than an instruction.

Level 1, Foundational

Publish reference architecture patterns and an antipattern catalogue for each vendors archetype, so the secure design is the default path rather than an achievement.

Activities

A) Publish reference patterns per archetype — Author one reference pattern per archetype, covering the brokered-access pattern routing vendor AI through an organisational gateway, the data-minimisation pattern filtering before egress to the vendor, the fallback pattern surviving vendor unavailability, and the exit pattern keeping data portable by construction. Each pattern states the components and their boundaries, the controls at each boundary, the threats it mitigates by identifier from the TA library, the requirements it satisfies from the SR pack, and the conditions under which it does not apply. Keep each pattern to a length a team reads before designing rather than after, targeting three pages with a diagram. Validate each with an engineer who has built that archetype, because a pattern authored entirely within security describes a system that is secure and unbuildable, and teams respond to that by ignoring it rather than by arguing. Publish patterns in the repository where the work happens rather than in a separate architecture space.

B) Publish the antipattern catalogue — Author an antipattern catalogue naming the insecure shapes that actually recur in this domain, being direct vendor API access from application code with credentials embedded, unfiltered organisational data sent to a vendor endpoint, and a vendor dependency with no fallback and no tested exit. For each antipattern, state why it is attractive, since it is always attractive for a reason and a catalogue that does not acknowledge that reads as naive; what specifically goes wrong; and which reference pattern replaces it. The antipattern half of the library is the more useful and the more commonly omitted, because teams recognise the thing they are building far more readily than they recognise the thing they should have built instead. Draw the catalogue from real designs seen in review rather than from literature, and add to it whenever review encounters a new recurring shape.

C) Establish the untrusted-content boundary and a design-time self-check — Document explicitly where untrusted content enters each archetype and how it is kept separate from instruction, control and identity. This is the single most consequential architectural decision in AI systems and the one most often left implicit, because the failure it prevents, content becoming instruction, does not look like a boundary violation in any conventional sense. State the pattern for isolation, for provenance marking, and for what the system must never do with content it did not author. Publish a short design-time self-check, at most ten questions, that a team can answer before requesting review, which both raises the quality of what arrives at review and tells the team what the review will ask, removing the asymmetry that makes design review feel arbitrary.

Outcome metrics

Oversight metrics — Share of designs where the reviewer recorded a required change before approval (detects: rubber-stamping at design review, where approval is recorded without the review having altered anything, which is the default state of a review with no capacity to say no) · Designs approved with a human gate declared on a new data class flowing to a vendor but no stated refusal authority (detects: a gate that exists structurally while the person holding it cannot actually refuse, which satisfies the control on paper and not in fact)

Success criteria

• A reference pattern exists for at least 85% of inventoried vendors archetypes, each validated with an engineer who has built that archetype.
• Every pattern states the threats it mitigates by TA identifier and the requirements it satisfies.
• An antipattern catalogue of at least eight entries is published, each drawn from a design actually seen in review and naming its replacement pattern.
• Every archetype ingesting content the organisation does not author has a documented untrusted-content boundary.
• A design-time self-check of at most ten questions is published and completed by at least 70% of designs arriving at review.

Assessment questions

1. Does a reference pattern exist for at least 85% of inventoried vendors archetypes, each stating components, boundaries, controls, the threats it mitigates by TA identifier, the requirements it satisfies and where it does not apply, validated with an engineer who has built that archetype?
2. Is there an antipattern catalogue of at least eight entries drawn from designs actually seen in review, each stating why the shape is attractive, what goes wrong, and which reference pattern replaces it? For Vendors, the assessor should be shown the brokered-access pattern and the data-minimisation filter.
3. Does every archetype ingesting content the organisation does not author have a documented untrusted-content boundary, and is a design-time self-check published and completed by at least 70% of designs arriving at review? In Vendors, ask to see the brokered-access pattern and the data-minimisation filter.

Common pitfalls

• ❌ Authoring patterns entirely within security, producing designs that are secure and unbuildable, which teams ignore rather than contest.
• ❌ Publishing reference patterns without an antipattern catalogue, omitting the half teams actually recognise themselves in.
• ❌ Writing an antipattern catalogue that does not acknowledge why each shape is attractive, which reads as naive and is dismissed.
• ❌ Leaving the untrusted-content boundary implicit, which is the most consequential architectural decision in AI systems and the least conventional-looking.
• ❌ Publishing patterns in a separate architecture space rather than where the work happens, after which they are found only by people already looking.

Level 2, Comprehensive

Calibrate the vendors reference-pattern set by risk tier and scale the autonomy and oversight controls in each pattern with the asset's capability tier.

Activities

A) Declare tier applicability across the pattern library — Annotate every pattern in the library with the tiers it applies to and the tiers at which it is mandatory rather than recommended. Publish the mapping so that a team's pattern set is derived from the asset's tier rather than assembled by preference. At Critical tier, pattern adherence is mandatory and any deviation requires a full-lane design review with a named architect; at Low tier the pattern is recommended and deviation needs only a recorded note. Stating this removes the ambiguity that otherwise resolves in favour of whatever is quickest to build. Re-derive the applicable pattern set automatically when an asset's tier changes, because a Medium-to-Critical promotion that does not change the pattern obligations has changed nothing that matters.

B) Scale autonomy and oversight controls with capability tier — Within each pattern, define how its controls tighten as the asset's capability tier rises. Bound blast radius more tightly, shorten the autonomy budget between human checkpoints, and require a human gate on a new data class flowing to a vendor at higher capability tiers where lower tiers need only logging. Specify each gate numerically rather than in principle: who holds the authority to refuse, what information they receive, and how long they have. A gate specified as a principle is compressed under delivery pressure until scrutiny is impossible, and because nothing in the specification objects, the erosion is invisible. Publish a reviewer-load ceiling per tier above which review capacity must be added rather than reviewers exhorted, since a gate held by an overloaded reviewer is a gate in name only.

C) Run the incident-to-pattern learning loop — Establish a standing review that takes every incident reaching a root cause and asks two questions: was there a pattern that should have prevented this, and if so why was it not followed, or if not what pattern should now exist. Record the answer for every incident, including the answer that no pattern change is warranted, because the discipline of recording the negative case is what keeps the loop honest rather than performative. Where a pattern existed and was not followed, the finding belongs to the review and adoption practices rather than to the library. Where no pattern existed, author one and publish it within 60 days. Track the share of incidents attributable to a pattern that existed and was not followed, which separates a library problem from an adoption problem and is the single most useful number this practice produces.

Outcome metrics

Oversight metrics — Reviewer override rate on pattern deviation requests, paired with sustained-override rate (detects: deviation approval becoming automatic under delivery pressure, where pairing with sustained rate counters the gaming that follows once the rate becomes a target) · Design reviews per reviewer per week against the published ceiling (detects: reviewer overload, where architectural scrutiny cannot survive the volume regardless of reviewer skill, which is a capacity defect rather than a personnel one) · Critical-tier designs where the human gate on a new data class flowing to a vendor specifies authority, information and a time budget numerically (detects: oversight specified as a principle rather than a number, which is compressed under delivery pressure until scrutiny is impossible and nothing objects)

Success criteria

• Every pattern declares its tier applicability and whether it is mandatory or recommended at each tier.
• Controls within each pattern scale with capability tier, with every human gate specified numerically in authority, information and time.
• At least 95% of Critical-tier assets either follow a reference pattern or carry a full-lane deviation review, with no third category.
• Every incident reaching root cause is checked against the library, with negative findings recorded.
• A per-tier reviewer-load ceiling is published and monitored, and breaching it adds capacity rather than exhortation.

Assessment questions

1. Does every pattern declare its tier applicability and mandatory-versus-recommended status, with the applicable set re-derived automatically when an asset's tier changes? Evidence this in the Vendors domain against the brokered-access pattern and the data-minimisation filter.
2. Do the controls inside each pattern tighten as capability rises, with the human gate on a new data class flowing to a vendor specified numerically in authority, information and time budget rather than as a principle, and is a per-tier reviewer-load ceiling published?
3. Is every incident reaching a root cause checked against the library for a preventing pattern with negative findings recorded, and is the share attributable to an existing pattern not followed tracked separately from gaps in the library? In Vendors, ask to see the brokered-access pattern and the data-minimisation filter.

Common pitfalls

• ❌ Publishing patterns without tier applicability, which leaves adoption scope to resolve in favour of whatever is quickest to build.
• ❌ Specifying human gates as principles rather than numbers, so they compress invisibly under delivery pressure until scrutiny is impossible.
• ❌ Recording only the incidents that produced a pattern change, which makes the learning loop performative rather than honest.
• ❌ Conflating a pattern that was missing with a pattern that was ignored, which sends improvement effort to the wrong practice.
• ❌ Responding to reviewer overload with exhortation rather than capacity, which converts the gate into a formality.

Level 3, Industry-Leading

Publish vendors patterns as executable artifacts adopted by default, verify conformance continuously, and contribute the pattern library back to the field.

Activities

A) Ship patterns as executable artifacts — Convert each reference pattern into something a team runs rather than reads: infrastructure modules, project scaffolds, libraries or configuration bundles that instantiate the pattern correctly by default. This is the level's central move and the one that changes adoption economics, because a pattern that is faster to adopt than to circumvent is adopted without enforcement, and no amount of mandate achieves what convenience achieves. Version the artifacts, publish their change history, and provide a migration path when a pattern changes so that existing adopters are not stranded on an old version, which is the most common reason executable pattern libraries decay. Measure adoption of the artifact rather than conformance to the document, since the artifact is now the pattern. In this domain the executable artifact is a broker or gateway configuration that routes vendor access, filters the payload and applies limits, so using the vendor correctly is easier than integrating directly.

B) Verify conformance continuously against the running system — Verify that what is running still matches the pattern that was approved, continuously rather than at review. Draw the signals from configuration state, runtime telemetry and the deployment pipeline, and evaluate each pattern's invariants as checks that can fail. Architecture drift is the defining failure of this practice at scale: designs are reviewed once and systems change continuously, so a library verified only at design time describes the past. Publish a service level covering detection of a Critical-tier conformance breach within 5 business days, and route breaches to the issue-management practice with tier-appropriate severity rather than to a report nobody owns. Where an invariant cannot be checked automatically, list it explicitly so the verification's real reach is known. Conformance verification reads the integration path and payload schema against the pattern, which catches the direct integration added for latency beside the sanctioned broker.

C) Contribute the executable pattern library back — Contribute at least four substantive artifacts per year, weighted toward executable form: reference implementations, infrastructure modules, scaffolds and the conformance checks that verify them. Executable contributions carry disproportionate value in this practice, because a peer organisation can adopt a module and can only read a diagram, and the adoption gap between the two is the difference between influencing the field and describing it. Work through OpenSSF AI, CNCF where infrastructure applies, OWASP projects and sector bodies. Count only accepted artifacts, review each legally before release, and prefer contributing the patterns with the highest internal adoption, since those carry the most operational evidence and are the most likely to survive contact with another organisation's environment. Vendor patterns contribute well as broker configurations and payload-filtering rules, which peers can adopt against the same small set of major providers.

Outcome metrics

Oversight metrics — Pattern deviations later implicated in an incident, as a share of deviations approved (detects: deviation approval that is procedurally correct and substantively wrong, which only post-hoc linkage to incidents can reveal) · Automated pattern conformance checks overridden by a human, with the override sustained (detects: automation override becoming the routine path around a check that is inconvenient rather than wrong, which is how enforced patterns quietly become advisory) · Reviewer disagreement rate on a blind double-review sample of Critical-tier designs (detects: reviewer drift, where the same design would be approved by one architect and refused by another, making the gate a lottery rather than a control)

Success criteria

• At least 80% of reference patterns ship as executable artifacts with versioning and a migration path on breaking change.
• At least 75% of new Critical- and High-tier assets adopt the executable artifact at creation.
• At least 85% of pattern invariants are verified continuously against the running system, with unverifiable invariants listed explicitly.
• Critical-tier conformance breaches are detected within 5 business days and routed to issue management with tier-appropriate severity.
• At least four accepted external contributions per year, the majority executable rather than documentary.

Assessment questions

1. Do at least 80% of reference patterns ship as executable artifacts with versioning and a migration path, and do at least 75% of new Critical- and High-tier assets adopt the artifact at creation rather than adopting the document? Evidence this in the Vendors domain against the brokered-access pattern and the data-minimisation filter.
2. Are at least 85% of pattern invariants verified continuously against the running system with unverifiable invariants listed explicitly, and are Critical-tier breaches detected within 5 business days and routed to issue management? For Vendors, the assessor should be shown the brokered-access pattern and the data-minimisation filter.
3. Does the organisation contribute at least four accepted artifacts per year in majority-executable form, prioritising the patterns with the highest internal adoption and therefore the most operational evidence? In Vendors, ask to see the brokered-access pattern and the data-minimisation filter.

Common pitfalls

• ❌ Shipping executable patterns without a migration path, which strands existing adopters on old versions and is the usual cause of library decay.
• ❌ Verifying conformance only at design time, which describes the past while the running system changes continuously.
• ❌ Measuring conformance to the document after the artifact has become the real pattern, which measures the wrong thing.
• ❌ Leaving unverifiable invariants unlisted, so the verification's actual reach is unknown and overstated.
• ❌ Contributing diagrams rather than modules, which peers read and do not adopt.

Business Function: Verification

Design Review (DR)

Objective: Establish a two-lane design review for vendors AI changes with a published checklist and service level, so review is predictable enough that teams route through it.

A program at this level reviews vendors AI designs against a published checklist, in one of two lanes, with a stated turnaround. The fast lane clears low-risk changes against a standard checklist in days; the full lane handles novel or high-risk designs with a named architect. Publishing both the checklist and the service level is what makes the practice work: a review whose criteria and duration are unknown is a review teams plan around rather than through, and the routing-around is invisible until an incident reveals it.

Level 1, Foundational

Establish a two-lane design review for vendors AI changes with a published checklist and service level, so review is predictable enough that teams route through it.

Activities

A) Establish two review lanes with published service levels — Define a fast lane and a full lane, and publish the routing rule and the service level for each. The fast lane handles designs adopting a sanctioned reference pattern without material deviation, cleared against a standard checklist within 5 business days. The full lane handles novel architectures, pattern deviations and high-risk designs, with a named architect reviewer and a decision within 15 business days. Publishing both numbers is the practice's single most important act: a review of unknown duration is a review teams plan around, and they do so quietly. Measure and report adherence per lane, and treat a persistent breach as a capacity problem to be resourced rather than a communication problem to be explained.

B) Publish the checklist and make decisions written — Publish the review checklist so teams can self-assess before submitting. In this domain the checklist covers the training-data position established contractually, sub-processor chain enumerated, data classes to be shared justified and minimised, residency and transfer basis stated, model-change notification obligation present, fallback for vendor unavailability, and a data-export path for exit. Every review produces a written decision of approve, approve with conditions, or refuse, naming the reviewer and stating the rationale. Conditions carry an owner and a date and are tracked to closure, because an approval with untracked conditions is an approval, and the conditions are the part that mattered. Record the checklist items that most often fail, and feed that back to the education and architecture practices as the clearest available evidence of where design understanding is weak.

C) Define review triggers and detect what bypassed review — Define what triggers a review rather than relying on teams to request one. In this domain the triggers are a new data class shared with the vendor, a vendor model change, a sub-processor addition, or a contract renewal, plus any new asset entering the inventory. Wire the triggers to the signals that already exist, being inventory changes, deployment events and change records, so that a review obligation is raised by the system rather than remembered by a person. Then detect bypass: reconcile assets reaching production against review decisions, and treat a production asset with no review record as a finding with an owner. Bypass detection is what converts design review from a service teams may use into a gate whose absence is visible, and without it the practice measures only the designs it happened to see.

Outcome metrics

Success criteria

• Two lanes operate with published routing rules and service levels of 5 and 15 business days, with adherence at or above 90%.
• The review checklist is published and teams self-assess against it before submitting.
• Every decision is written with a named reviewer and rationale, and every condition carries an owner, a date and closure tracking.
• Review triggers fire from inventory, deployment and change signals rather than relying on team initiative.
• Bypass reconciliation runs monthly and every unreviewed production asset is raised as a finding.

Assessment questions

1. Do two published review lanes operate with stated routing rules and service levels of 5 and 15 business days, with adherence measured and reported per lane and persistent breaches treated as a capacity problem? Evidence this in the Vendors domain against a full-lane decision on a new data class flowing to a vendor.
2. Is the review checklist published so teams self-assess before submitting, covering the training-data position established contractually, sub-processor chain enumerated, data classes to be shared justifie, and does every review produce a written decision with named reviewer, rationale and tracked conditions?
3. Do review triggers fire from inventory, deployment and change signals rather than team initiative, and does monthly bypass reconciliation raise every unreviewed production asset as a finding with an owner? In Vendors, ask to see a full-lane decision on a new data class flowing to a vendor.

Common pitfalls

• ❌ Running review without a published service level, which teams respond to by planning around review rather than through it, invisibly.
• ❌ Approving with conditions that are never tracked, which converts a conditional approval into an unconditional one.
• ❌ Relying on teams to request review, which means the practice only ever sees the designs of teams already inclined to ask.
• ❌ Never reconciling production against review records, so the practice measures the designs it saw and not the ones it missed.
• ❌ Treating persistent service-level breaches as a communication problem rather than as evidence that review capacity is short.

Level 2, Comprehensive

Calibrate vendors design review depth and reviewer authority to asset tier, and make review decisions traceable to the threats and requirements they turned on.

Activities

A) Derive review depth and reviewer authority from tier — Publish what each tier receives: at Critical, a full-lane review with a named architect, mandatory threat-model and requirements-evidence review, and a decision recorded where the sponsor can see it; at High, full lane where a deviation exists and fast lane otherwise; at Medium, fast lane; at Low, a checklist pass with sampling. Derive the lane from the inventory tier automatically rather than from reviewer judgment at intake, because intake routing under queue pressure drifts toward the fast lane and does so without anyone deciding to lower the bar. Publish a reviewer-load ceiling per tier, and when it is breached add reviewers rather than shortening reviews, since the alternative is a gate that approves at the rate the queue demands. In this domain the designs that genuinely need judgment are new data classes flowing to a vendor and new sub-processor chains, while an additional seat on a sanctioned tool does not.

B) Make decisions traceable to threats and requirements — Require every full-lane decision to cite the specific threat identifiers from the TA library and requirement identifiers from the SR pack that it assessed, and to state which ones drove each condition or refusal. This converts review from personal judgment into reconstructable reasoning, which matters for three reasons: a second reviewer can reach the same conclusion, an auditor can follow it, and the practice can tell which threats and requirements are actually doing work in review. Report the threats and requirements most frequently driving conditions, since that is direct evidence of where designs are systematically weak and belongs in the architecture and education practices rather than only in the review log. Automated design criteria read the payload schema and integration path, but the contractual dimension cannot be automated and must be listed explicitly as reviewer work rather than quietly omitted.

C) Govern deviations and make Critical decisions visible — Operate deviation approval as a distinct decision with its own record: what pattern or requirement is not met, why, what compensating control applies, who approved it at an authority proportionate to tier, and when it expires. Review the deviation register monthly and report Critical-tier deviations to the sponsor alongside the approvals, because a review practice that reports only what it approved and not what it excused presents a misleading picture of the estate. Track deviations later implicated in incidents as a distinct measure, which is the only honest feedback on whether deviation judgment is sound rather than merely procedurally correct. Forward tracing in this domain most often implicates approvals that relied on a vendor assurance rather than a contractual term.

Outcome metrics

Success criteria

• Review lane and reviewer seniority are derived from asset tier, with at least 95% of routing tier-derived rather than judged at intake.
• Every Critical-tier design receives a full-lane review with a named architect and a sponsor-visible decision.
• At least 90% of full-lane decisions cite the specific threat and requirement identifiers that drove each condition or refusal.
• Deviation approvals carry a compensating control, tier-appropriate authority and an expiry, with none past expiry.
• A per-tier reviewer-load ceiling is published and breaching it adds capacity rather than shortening reviews.

Assessment questions

1. Is review lane and reviewer seniority derived automatically from asset tier rather than judged at intake under queue pressure, with at least 95% tier-derived routing and a published per-tier reviewer-load ceiling? Evidence this in the Vendors domain against a full-lane decision on a new data class flowing to a vendor.
2. Do at least 90% of full-lane decisions cite the specific TA threat identifiers and SR requirement identifiers that drove each condition or refusal, making the reasoning reconstructable by a second reviewer and an auditor? For Vendors, the assessor should be shown a full-lane decision on a new data class flowing to a vendor.
3. Do deviation approvals carry a compensating control, tier-appropriate approval authority and an expiry with none past expiry, and are Critical-tier deviations reported to the sponsor alongside approvals? In Vendors, ask to see a full-lane decision on a new data class flowing to a vendor.

Common pitfalls

• ❌ Routing at intake by reviewer judgment, which under queue pressure drifts toward the fast lane without anyone deciding to lower the bar.
• ❌ Recording decisions without citing what drove them, which makes review personal judgment and leaves a second reviewer unable to reproduce it.
• ❌ Reporting approvals to the sponsor without reporting deviations, which presents a systematically flattering picture of the estate.
• ❌ Responding to reviewer overload by shortening reviews rather than adding reviewers, producing a gate that approves at queue rate.
• ❌ Never tracking whether deviations were later implicated in incidents, which leaves deviation judgment unevaluated.

Level 3, Industry-Leading

Shift vendors design review left into the pipeline, reserve human review for genuine novelty, and measure whether review decisions were right rather than only timely.

Activities

A) Automate the machine-checkable criteria — Convert every review criterion that can be evaluated from an artifact into an automated check running against the design representation, whether that is infrastructure-as-code, a configuration manifest, a service definition or a system card. A design meeting all automated criteria and adopting a sanctioned pattern clears without human review at Medium and Low tier, and arrives at Critical and High review pre-verified so the reviewer spends attention on what is novel rather than on confirming what is conformant. Publish the list of criteria that cannot be automated and review it quarterly for reduction, because that list is the honest boundary of the automation. Keep a human veto and sampling at every tier, since a wholly automated gate encodes yesterday's criteria permanently and nothing in the system notices. In this domain the designs that genuinely need judgment are new data classes flowing to a vendor and new sub-processor chains, while an additional seat on a sanctioned tool does not. Novelty in this domain means a new vendor archetype or an unfamiliar sub-processor arrangement, not another tool in a category already assessed.

B) Measure decision quality, not throughput — Trace review decisions forward. For every approval, record whether the asset subsequently generated an incident, an implementation-review finding or a security-test failure attributable to something the review should have caught. Report false-negative rate, being approvals later implicated, and investigate false-positive pressure, being conditions imposed that evidence later showed unnecessary, since a gate that over-conditions trains teams to argue rather than comply. Throughput and service-level adherence remain as health metrics but are explicitly not the practice's outcome measure, because a review function optimising for turnaround will achieve it by approving, and the metric will look excellent throughout. Publish decision-quality measures to the sponsor alongside throughput. Automated design criteria read the payload schema and integration path, but the contractual dimension cannot be automated and must be listed explicitly as reviewer work rather than quietly omitted.

C) Calibrate reviewers and concentrate attention on novelty — Run blind double-review on a sample of Critical-tier designs at least quarterly and report the disagreement rate, because a gate whose outcome depends on which architect drew the review is a lottery rather than a control, and disagreement is invisible without deliberately measuring it. Use the disagreement cases to refine criteria rather than to rank reviewers. Reserve human review explicitly for novelty: designs using an unestablished pattern, a new archetype, a new model capability class or an unfamiliar threat surface, and route everything else to automated verification with sampling. Track the share of reviewer hours spent on novel designs, which should rise steadily and is the clearest measure that the automation is doing its job. Forward tracing in this domain most often implicates approvals that relied on a vendor assurance rather than a contractual term.

Outcome metrics

Success criteria

• At least 75% of machine-checkable criteria run automatically, with the non-automatable remainder listed and shrinking.
• Medium- and Low-tier conformant designs clear without human review while sampling continues, and a human veto is retained at every tier.
• Decision quality is measured by forward tracing to incidents and findings, and reported to the sponsor alongside throughput.
• Blind double-review runs quarterly on Critical-tier designs with a disagreement rate at or below 15%, used to refine criteria rather than rank reviewers.
• At least 60% of reviewer hours are spent on genuinely novel designs, and rising.

Assessment questions

1. Are at least 75% of machine-checkable review criteria evaluated automatically with the non-automatable remainder listed explicitly, and do conformant Medium- and Low-tier designs clear without human review while sampling and a human veto are retained? Evidence this in the Vendors domain against a full-lane decision on a new data class flowing to a vendor.
2. Is decision quality measured by tracing approvals forward to incidents, implementation findings and test failures, reported to the sponsor alongside throughput and explicitly rather than throughput as the outcome measure? For Vendors, the assessor should be shown a full-lane decision on a new data class flowing to a vendor.
3. Does blind double-review run at least quarterly on Critical-tier designs with disagreement at or below 15% used to refine criteria rather than rank reviewers, and is at least 60% of reviewer time spent on genuinely novel designs? In Vendors, ask to see a full-lane decision on a new data class flowing to a vendor.

Common pitfalls

• ❌ Optimising the practice for turnaround, which a review function achieves by approving while the metric looks excellent throughout.
• ❌ Removing the human veto and sampling once automation is in place, which encodes yesterday's criteria permanently with nothing to notice.
• ❌ Never measuring reviewer disagreement, leaving a gate whose outcome depends on which architect drew the review.
• ❌ Using double-review disagreement to rank reviewers, which produces defensive convergence rather than better criteria.
• ❌ Leaving the non-automatable criteria list unpublished, so the boundary of the automation is unknown and assumed wider than it is.

Implementation Review (IR)

Objective: Review what was actually built against what was designed and required for vendors AI assets, at go-live, with findings tracked to closure.

A program at this level checks the built system rather than the intended one. Every vendors AI asset gets an implementation review before go-live, working from a published checklist that examines the data actually sent to the vendor against what was approved, the integration path using the organisational gateway rather than direct credentials, contractual controls reflected in configuration such as training opt-out flags, fallback behaviour present in code, and export capability genuinely working. The distinction from design review is the entire point: design review assesses what a team intends, implementation review assesses what exists, and the gap between the two is where most AI security failures actually live. Findings carry an owner, a severity and a date, and go-live is conditional on the blocking ones being closed.

Level 1, Foundational

Review what was actually built against what was designed and required for vendors AI assets, at go-live, with findings tracked to closure.

Activities

A) Publish the implementation review checklist and run it at go-live — Publish a checklist per archetype that examines the built artifact rather than its documentation. In this domain the review looks at the data actually sent to the vendor against what was approved, the integration path using the organisational gateway rather than direct credentials, contractual controls reflected in configuration such as training opt-out flags, fallback behaviour present in code, and export capability genuinely working. The discipline that makes this practice worth running is verifying presence and effect rather than presence alone: a logging configuration that exists but emits nothing, an allowlist enforced in audit mode, a kill switch wired to nothing, all pass a documentation check and fail a real one. Run the review before go-live for every asset, and make go-live conditional on blocking findings being closed. Where the review cannot verify a control because the environment does not expose it, record that explicitly as unverifiable rather than as passed, since silence is not a pass.

B) Compare the built system against the approved design — Reconcile what was built against the design review decision and the requirements pack: every requirement in scope for the asset's archetype gets a verified, not-verified or unverifiable disposition, and every condition attached to the design approval gets confirmed as met. Record the design-to-implementation gap as a measure in its own right, because a gap that is never measured is assumed to be zero and never is. Where implementation diverges from the approved design, treat it as a finding against the change rather than as a re-opening of the design decision, and route material divergence back to design review rather than absorbing it in the implementation record.

C) Track findings to closure with severity and ownership — Give every finding a severity, an owner, a remediation date and a disposition of remediated, accepted or deferred. An accepted finding requires a named acceptor with authority proportionate to the severity, because a finding that is neither fixed nor formally accepted leaves residual risk that nobody owns, which is the most common way implementation review findings evaporate. Feed all findings into the issue-management backlog rather than maintaining a separate review register, so that AI findings age and escalate on the same rails as everything else. Report finding density by archetype and by checklist item quarterly, which tells the architecture and education practices where implementations systematically diverge rather than where reviewers happened to look.

Outcome metrics

Oversight metrics — Share of implementation reviews recording at least one finding before sign-off (detects: rubber-stamping at implementation review, where sign-off is recorded without the review having examined anything, which is the default outcome when review is a release-checklist item) · Findings accepted as risk at sign-off rather than remediated, with a named acceptor (detects: silent acceptance, where a finding is noted and neither fixed nor formally accepted, so nobody owns the residual risk)

Success criteria

• Every AI asset reaching production has an implementation review against the published per-archetype checklist.
• Controls are verified as present and effective, with controls the environment cannot expose recorded as unverifiable rather than passed.
• Every in-scope requirement receives a verified, not-verified or unverifiable disposition at review.
• Blocking findings are closed before go-live, and every accepted finding has a named acceptor with proportionate authority.
• Findings flow into the shared issue backlog rather than a separate review register.

Assessment questions

1. Does every vendors AI asset receive an implementation review before go-live against a published per-archetype checklist that verifies controls are present and effective rather than documented, with controls the environment cannot expose recorded as unverifiable rather than passed?
2. Is the built system reconciled against the approved design and requirements pack, with every in-scope requirement given a verified, not-verified or unverifiable disposition and the design-to-implementation gap measured as a metric in its own right? For Vendors, the assessor should be shown the go-live review record showing the actual payload fields sent to the vendor.
3. Do all findings carry an owner, severity, date and disposition in the shared issue backlog, with blocking findings closed before go-live and every accepted finding carrying a named acceptor with proportionate authority? In Vendors, ask to see the go-live review record showing the actual payload fields sent to the vendor.

Common pitfalls

• ❌ Verifying that a control is configured rather than that it works, which passes logging that emits nothing and allowlists running in audit mode.
• ❌ Recording a control the environment cannot expose as passed, which converts an observability gap into a false assurance.
• ❌ Keeping findings in a separate review register, where they age without the escalation the shared backlog would give them.
• ❌ Allowing findings to be neither fixed nor formally accepted, which leaves residual risk owned by nobody.
• ❌ Treating implementation divergence as a chance to re-open the design decision rather than as a finding against the change.

Level 2, Comprehensive

Calibrate vendors implementation review cadence to asset tier and detect configuration drift continuously, so controls verified at go-live stay verified.

Activities

A) Set re-review cadence by tier and material change — Publish the re-review cadence per tier: Critical at go-live, semi-annually and on every material change; High at go-live, annually and on material change; Medium at go-live and annually; Low at go-live. Define material change concretely for this domain, being a new field added to the vendor payload, a direct integration added beside the gateway for latency, a training opt-out flag reset by a vendor configuration change, and a fallback path removed during refactoring, and wire the triggers to change records and deployment events so the obligation is raised by the system rather than remembered. The calendar is the weakest trigger available and should be the floor rather than the mechanism, because an asset that changed materially in month two carries a verified-once record for ten months and the record looks identical to an accurate one.

B) Detect configuration drift continuously on high-tier assets — Stand up continuous drift detection for Critical- and High-tier assets covering the controls whose silent disablement would matter most. In this domain the drift that recurs is a new field added to the vendor payload, a direct integration added beside the gateway for latency, a training opt-out flag reset by a vendor configuration change, and a fallback path removed during refactoring. Detect it from configuration state, infrastructure-as-code divergence, policy enforcement mode and runtime telemetry, and alert on change rather than on schedule. Drift detection is what converts implementation review from a point-in-time assertion into a continuing one, and it is the single highest-value activity at this level, because implementations rarely fail review, they pass and then change. Publish a detection service level of 5 business days at Critical tier and route detections into the issue backlog with tier-appropriate severity rather than into a report.

C) Apply tier-proportionate service levels and deferral authority — Set remediation service levels by finding severity and asset tier, tighter at Critical, and report adherence rather than only backlog size. Require deferral authority to rise with tier and severity, so a deferred Critical-tier finding needs an executive acceptance with an expiry and a quarterly review, while a Low-tier one needs the practice lead. Report deferred Critical-tier findings to the sponsor alongside remediated ones, because a practice that reports only closures presents a picture that flatters it. Track findings that were deferred and later implicated in an incident as a distinct measure, since that is the only feedback available on whether deferral judgment is sound rather than merely authorised.

Outcome metrics

Oversight metrics — Reviewer finding rate per review, compared across reviewers on comparable assets (detects: reviewer drift, where the same implementation would pass with one reviewer and fail with another, making sign-off depend on assignment rather than on the code) · Implementation reviews per reviewer per week against the published ceiling (detects: reviewer overload, where depth falls as volume rises and the review continues to be recorded as complete) · Critical-tier findings deferred past their remediation window with a recorded acceptance authority (detects: deferral becoming the default disposition under release pressure, which converts a blocking finding into an advisory one without a decision)

Success criteria

• Re-review cadence is published per tier and driven by material-change triggers wired to change and deployment events.
• At least 90% of Critical-tier assets are under continuous drift detection, with detection within 5 business days.
• Findings carry tier and severity service levels with adherence at or above 90% per tier.
• Deferral authority rises with tier and severity, and deferred Critical findings carry executive acceptance with expiry.
• Deferred findings later implicated in incidents are tracked as a distinct measure of deferral judgment.

Assessment questions

1. Is re-review cadence published per tier and driven by material-change triggers wired to change records and deployment events, with the calendar serving as a floor rather than the mechanism? Evidence this in the Vendors domain against the go-live review record showing the actual payload fields sent to the vendor.
2. Are at least 90% of Critical-tier assets under continuous configuration drift detection with a 5 business day detection service level, and are detections routed into the shared backlog with tier-appropriate severity? For Vendors, the assessor should be shown the go-live review record showing the actual payload fields sent to the vendor.
3. Do remediation service levels vary by tier and severity with adherence at or above 90% per tier, does deferral authority rise with tier, and are deferred findings later implicated in incidents tracked as a distinct measure? In Vendors, ask to see the go-live review record showing the actual payload fields sent to the vendor.

Common pitfalls

• ❌ Relying on annual re-review, which leaves an asset that changed in month two carrying a record indistinguishable from an accurate one for ten months.
• ❌ Detecting drift on a schedule rather than on change, which reintroduces exactly the point-in-time weakness the activity exists to remove.
• ❌ Routing drift detections into a report rather than the backlog, where they age without escalation and without an owner.
• ❌ Reporting closures to the sponsor without deferrals, which systematically flatters the practice.
• ❌ Authorising deferral without ever checking whether deferred findings later caused incidents, leaving the judgment unevaluated.

Level 3, Industry-Leading

Verify vendors implementation continuously in the pipeline and at runtime, and measure whether review judgment was right rather than only whether review happened.

Activities

A) Assert controls continuously in pipeline and runtime — Express each verifiable control as an assertion that runs both in the integration delivery pipeline and broker configuration path against the change and against the running system on a continuous schedule. Pipeline assertions prevent a regression from merging; runtime assertions catch what was changed outside the pipeline, and both are needed because the second is where drift actually originates. Give each assertion a tier-proportionate severity, blocks the integration change on Critical-tier failures, and route runtime failures to the backlog automatically. Publish the list of controls that cannot be asserted and review it quarterly with a reduction target, because that list is the true boundary of the verification and is otherwise assumed to be empty. Assertions in this domain read the integration path, payload schema, gateway route and opt-out flag state, while the contractual controls remain unassertable and must be listed as such.

B) Concentrate human review on the unassertable — Reserve human implementation review for the controls automation cannot assert and for the judgment calls automation cannot make, such as whether a control's implementation is adequate to its intent rather than merely present. Retain sampling on automatically verified assets so that the assertions themselves are checked, because an assertion that has silently stopped evaluating reports success indistinguishably from one that is passing. Track the share of reviewer hours spent on unassertable controls, which should rise as automation extends, and use it as the measure that the automation is releasing capacity rather than merely adding a second process alongside the first. What automation cannot assert here is whether a vendor is honouring a term it has agreed to, which is why audit and evidence rights matter and why their absence is a finding rather than a preference.

C) Measure review judgment by forward tracing — Trace every sign-off forward. Where a production incident is attributable to something implementation review examined and passed, record it as a review false negative and investigate whether the checklist, the assertion or the judgment was at fault. Report the false-negative rate to the sponsor alongside coverage and throughput, and treat it as the practice's primary quality measure, because coverage and timeliness are both achievable by a review function that examines nothing carefully. Run blind double-review on a sample of Critical-tier implementations quarterly and report the disagreement rate, using disagreements to refine checklists and assertions rather than to rank reviewers. Forward tracing in this domain most often implicates sign-offs that verified a configuration flag without verifying the vendor actually acts on it.

Outcome metrics

Oversight metrics — Automated verification failures overridden by a human, with the override sustained across builds (detects: override becoming the routine path around a check that is inconvenient rather than wrong, which is how automated verification quietly becomes advisory) · Sign-offs later contradicted by a production incident attributable to what the review examined (detects: review that is procedurally complete and substantively wrong, which only forward tracing to incidents can reveal) · Blind double-review disagreement rate on Critical-tier implementations (detects: reviewer drift at the tier where it matters most, invisible unless deliberately sampled)

Success criteria

• At least 80% of verifiable controls carry continuous assertions in both pipeline and runtime, with unassertable controls listed and shrinking.
• Critical-tier assertion failures blocks the integration change or reach an owner within the day.
• Review false-negative rate is measured by forward tracing to incidents and reported to the sponsor as the practice's primary quality measure.
• At least 60% of reviewer hours go to controls automation cannot assert, and rising.
• Sampling continues on automatically verified assets so that a silently stopped assertion is detectable.

Assessment questions

1. Do at least 80% of verifiable controls carry continuous assertions running both in the pipeline and against the running system, with the unassertable remainder listed explicitly and reviewed quarterly for reduction? Evidence this in the Vendors domain against the go-live review record showing the actual payload fields sent to the vendor.
2. Is review false-negative rate measured by tracing sign-offs forward to incidents and reported to the sponsor as the primary quality measure, rather than coverage and timeliness which a review examining nothing carefully can also achieve? For Vendors, the assessor should be shown the go-live review record showing the actual payload fields sent to the vendor.
3. Is sampling maintained on automatically verified assets so that an assertion which has silently stopped evaluating is detectable, and does at least 60% of reviewer time go to controls automation cannot assert? In Vendors, ask to see the go-live review record showing the actual payload fields sent to the vendor.

Common pitfalls

• ❌ Asserting only in the pipeline, which misses the drift that originates outside it and is where most silent disablement happens.
• ❌ Assuming a passing assertion is evaluating, when a stopped assertion reports success indistinguishably.
• ❌ Dropping sampling once automation is in place, removing the only check on the checks.
• ❌ Reporting coverage and timeliness as quality, both of which a review function that examines nothing carefully achieves easily.
• ❌ Leaving the unassertable control list unpublished, so the verification boundary is assumed empty and is not.

Security Testing (ST)

Objective: Baseline a foundational security test battery per vendors archetype that asserts the controls actually hold, and run it before every release.

A program at this level tests the AI-specific controls rather than assuming them. Each archetype has a foundational battery, and every test asserts a control holds rather than that a feature works. The battery covers a data-minimisation test asserting that only approved fields leave for the vendor, a gateway-path test asserting that no direct vendor integration bypasses the organisational broker, a training opt-out verification test confirming the flag is set and honoured where verifiable, a fallback test asserting the system degrades safely when the vendor is unavailable, an export test asserting data can actually be retrieved in a usable form, and a response-handling test asserting vendor output is treated as untrusted. The discipline that distinguishes this practice from general quality assurance is that each test must be able to fail for the right reason: a test that passes because the condition it checks never arises provides assurance that is indistinguishable from real assurance on every dashboard.

Level 1, Foundational

Baseline a foundational security test battery per vendors archetype that asserts the controls actually hold, and run it before every release.

Activities

A) Build the foundational battery per archetype — Author a foundational test battery per archetype covering a data-minimisation test asserting that only approved fields leave for the vendor, a gateway-path test asserting that no direct vendor integration bypasses the organisational broker, a training opt-out verification test confirming the flag is set and honoured where verifiable, a fallback test asserting the system degrades safely when the vendor is unavailable, an export test asserting data can actually be retrieved in a usable form, and a response-handling test asserting vendor output is treated as untrusted. Write each test as an assertion about a control rather than as a check on behaviour, and verify that each test can fail: introduce the condition deliberately once and confirm the test catches it, which is the only way to distinguish a test that asserts something from one that passes because its condition never arises. Keep the battery small enough to run on every integration change and on every vendor-change event at the tiers that need it, because a battery too slow to run continuously will be moved to a nightly job and then to a weekly one and then to nowhere. Version the battery alongside the reference patterns so that a pattern change and its test change travel together.

B) Map tests to threats and publish the coverage gaps — Map each test to the threat identifiers from the TA library that it exercises, and publish the threats that have no test. The coverage map is what converts a test suite into an assurance argument: without it the battery asserts that the tests pass, which is not the same claim as that the threats are addressed. Report threat coverage rather than test count, since test count rises with effort while coverage rises only with effect. Treat an untested threat at a high-risk archetype as an open finding with an owner rather than as a backlog item, and feed coverage gaps back to the threat assessment practice so that the library and the battery evolve together.

C) Run before release, with a governed waiver path — Run the battery before every release, and trigger it on the changes that actually invalidate prior results, being a vendor model change, a contract renewal, a new data field in the payload, and a vendor API version change. Failures block release for Critical- and High-tier assets and warn for the rest. Provide a governed waiver path, because an unwaivable gate is bypassed in ways nobody sees: a waiver needs a named authority, a stated reason, an expiry and automatic notification. Review waivers weekly and treat a test waived across more than one release cycle as a finding, since the common failure here is not a single waiver but a test quietly disabled to unblock a release and never re-enabled, which reduces coverage without anyone deciding to reduce it.

Outcome metrics

Oversight metrics — Failing tests waived to permit a release, with a named waiver authority (detects: waiver becoming the routine release path, where a failing security test is treated as a scheduling obstacle rather than a result) · Tests marked as skipped or quarantined for more than one release cycle (detects: silent test decay, where a test is disabled to unblock a release and never re-enabled, so coverage falls without any decision being taken)

Success criteria

• A foundational battery exists for at least 85% of inventoried vendors archetypes and runs before every release.
• Every test has been verified as able to fail by deliberately introducing its condition once.
• At least 75% of threats from the library are exercised by a test, with untested threats published as gaps carrying owners.
• Failures block release for Critical- and High-tier assets, with a governed waiver path carrying authority, reason and expiry.
• A test waived across more than one release cycle is raised as a finding rather than tolerated.

Assessment questions

1. Does a foundational security test battery exist for at least 85% of inventoried vendors archetypes, with each test written as an assertion about a control and verified as able to fail by deliberately introducing its condition once?
2. Is every test mapped to the TA threat identifiers it exercises, with threats having no test published as gaps and threat coverage reported rather than test count? For Vendors, the assessor should be shown the data-minimisation test and the fallback test result.
3. Does the battery run before every release and on the change types that invalidate prior results, with failures blocking Critical- and High-tier releases and a governed waiver path carrying named authority, reason and expiry? In Vendors, ask to see the data-minimisation test and the fallback test result.

Common pitfalls

• ❌ Writing tests that pass because their condition never arises, which are indistinguishable from real assurance on every dashboard.
• ❌ Reporting test count rather than threat coverage, since count rises with effort while coverage rises only with effect.
• ❌ Building a battery too slow to run on every integration change and on every vendor-change event, which migrates to nightly, then weekly, then nowhere.
• ❌ Providing no waiver path, which produces bypasses nobody sees rather than compliance.
• ❌ Tolerating tests quietly disabled to unblock a release, which reduces coverage without any decision being taken.

Level 2, Comprehensive

Calibrate the vendors test battery to asset tier, run it continuously in CI, and add adversarial testing where the automated battery cannot reach.

Activities

A) Calibrate battery depth and gating to tier — Publish what each tier receives: Critical takes the full battery against the integration on every change and on each vendor model or term change, with failures blocking the integration, plus quarterly adversarial testing; High takes the full battery in the integration pipeline and against each vendor release with failures blocking; Medium takes a defined subset; Low takes spot checks. Derive the applicable battery from the inventory tier automatically rather than letting teams select, because selection under release pressure converges on the fastest subset. Publish gating behaviour per tier so that nobody negotiates it at release time. Where battery runtime makes per-build execution impractical at Critical tier, treat that as a test-performance problem to be solved rather than a reason to reduce frequency, since reduced frequency is how continuous testing becomes periodic testing. Adversarial work in this domain is constrained by what the contract permits, so scope must be agreed with the vendor in advance and the constraint recorded, because an untested area is not the same as a safe one and the difference should be visible.

B) Add adversarial testing where automation cannot reach — Schedule adversarial testing by people against Critical-tier assets at least quarterly, scoped from the threat model and executed by testers who did not build the asset. The purpose is specifically to find what the automated battery cannot: an automated suite tests the techniques it encodes and therefore converges, passing continuously while the attack surface moves underneath it. Measure the adversarial exercise by what it found that the battery missed, and feed every such finding back into the battery as a new test, so that the exercise permanently raises the automated floor rather than producing a report. Scope must be set by the testing party rather than the tested team, because a team that steers scope away from what it knows is weak produces a clean result that means nothing. The regression trigger that matters most here is a vendor model change, which alters behaviour the organisation validated and often arrives with little notice.

C) Keep corpora fresh and remove tests that assert nothing — Refresh regression corpora on a defined cadence, targeting 90 days, drawing new cases from published research, incidents, adversarial findings and threat-intelligence updates. A prompt-injection or adversarial-input corpus that has not changed in a year tests the techniques of a year ago and passes with increasing confidence as the real surface moves. Separately, audit the battery for tests that have never failed across a full quarter and examine each: some are genuinely stable controls and some assert nothing at all, and the two are indistinguishable from a dashboard. Remove or strengthen the latter. Trigger regression runs on a vendor model change, a contract renewal, a new data field in the payload, and a vendor API version change in addition to the scheduled cadence. Vendor test content contributes well as payload-filtering and opt-out verification cases, which peers can run against the same providers.

Outcome metrics

Oversight metrics — Critical-tier test waivers, with the waiver authority and expiry recorded (detects: waiver authority drifting downward under release pressure, so the decision to ship with a failing security test is taken by whoever is available) · Waived tests later implicated in a production incident (detects: waiver judgment that is procedurally correct and substantively wrong, which only forward tracing can reveal) · Test suites whose pass rate is 100% across every run for a full quarter (detects: tests that assert nothing, which are indistinguishable from passing tests on every dashboard and provide false assurance at exactly the tiers that need real assurance)

Success criteria

• Battery depth and gating behaviour are published per tier and derived automatically from the inventory.
• Every Critical-tier asset is exercised by adversarial testing at least annually, scoped by the testing party rather than the tested team.
• Every adversarial finding becomes a new automated test within 30 days, raising the automated floor permanently.
• Regression corpora are refreshed within 90 days, and triggered by the change types that invalidate prior results.
• Tests that have never failed across a quarter are audited for assertion strength and removed or strengthened.

Assessment questions

1. Is battery depth and gating behaviour published per tier and derived automatically from the inventory rather than selected by teams under release pressure? Evidence this in the Vendors domain against the data-minimisation test and the fallback test result.
2. Is every Critical-tier asset exercised by adversarial testing at least annually with scope set by the testing party, and does every adversarial finding become a new automated test within 30 days so the automated floor rises permanently? For Vendors, the assessor should be shown the data-minimisation test and the fallback test result.
3. Are regression corpora refreshed within 90 days and triggered by invalidating change types, and are tests that have never failed across a quarter audited for assertion strength rather than assumed to be stable controls? In Vendors, ask to see the data-minimisation test and the fallback test result.

Common pitfalls

• ❌ Letting the tested team scope the adversarial exercise, which produces a clean result that means nothing.
• ❌ Treating adversarial findings as a report rather than as new automated tests, so the same class of finding recurs next cycle.
• ❌ Leaving regression corpora static, which tests last year's techniques with rising confidence as the real surface moves.
• ❌ Reducing test frequency when the battery gets slow, rather than treating runtime as a performance problem to solve.
• ❌ Assuming a test that never fails is a stable control, when it is indistinguishable from one that asserts nothing.

Level 3, Industry-Leading

Run continuous adversarial testing against vendors AI assets, measure the battery against what people can still find, and contribute test corpora back to the field.

Activities

A) Run adversarial testing continuously — Move from scheduled engagements to continuous adversarial operation against Critical-tier assets, whether by a standing internal team, a contracted continuous service or an automated adversarial harness that evolves its own cases. Measure the practice by time-to-find rather than by engagement count, and track the surprise rate, being the share of findings the automated battery should have caught but did not. A falling surprise rate indicates the battery is converging on reality; a flat one at high volume indicates the battery has converged on itself. Feed every finding into the battery within days rather than within the 30-day window that L2 allowed, since continuous operation makes the slower loop the binding constraint. Adversarial work in this domain is constrained by what the contract permits, so scope must be agreed with the vendor in advance and the constraint recorded, because an untested area is not the same as a safe one and the difference should be visible.

B) Close the automation gap deliberately — Treat the gap between what automation catches and what humans still find as the practice's primary work item rather than as an inevitable residue. For each human finding, ask specifically why the battery missed it: a missing case, a missing technique class, an assertion too weak, or a surface the harness cannot reach. Record the category, because the distribution tells you whether to extend the corpus, strengthen assertions or rebuild the harness, and these are very different investments. Publish the gap measure alongside coverage so that a rising coverage number with a static gap is visible as the illusion it represents. The regression trigger that matters most here is a vendor model change, which alters behaviour the organisation validated and often arrives with little notice.

C) Contribute corpora and harnesses back — Contribute at least four substantive artifacts per year, weighted toward runnable form: adversarial corpora, test harnesses, assertion libraries and reproducible technique demonstrations. Shared adversarial corpora are the scarcest resource in AI security and the one where contribution most directly raises the whole field's floor, because a corpus a peer can run is worth far more than a paper describing what it contains. Work through OWASP projects, MITRE ATLAS, OpenSSF AI and sector bodies. Anonymise, legally review and coordinate disclosure with affected providers before release. Count only accepted artifacts, and prioritise contributing the cases that most often catch real defects internally, since those carry the most evidence. Vendor test content contributes well as payload-filtering and opt-out verification cases, which peers can run against the same providers.

Outcome metrics

Oversight metrics — Adversarial findings that the automated battery should have caught but did not (detects: a battery that has converged on what it already tests, which passes continuously while the real attack surface moves) · Automated test results overridden at merge, with the override sustained across builds (detects: override becoming the normal path around an inconvenient check, which converts a gate into advice without any decision) · Share of red-team scope determined by the team being tested (detects: scope capture, where the tested team steers the exercise away from what it knows is weak, which makes a clean result meaningless)

Success criteria

• Critical-tier assets are under continuous adversarial testing, measured by time-to-find rather than engagement count.
• The surprise rate is tracked and falling, and reported alongside coverage so that rising coverage with a static gap is visible.
• Human findings reach the automated battery within 5 business days.
• New public techniques reach the corpus within 14 days at the 90th percentile.
• At least four accepted external contributions per year, the majority runnable corpora or harnesses, disclosure-coordinated.

Assessment questions

1. Is adversarial testing continuous against Critical-tier assets rather than scheduled, measured by time-to-find and surprise rate rather than by engagement count? Evidence this in the Vendors domain against the data-minimisation test and the fallback test result.
2. Is the automation gap treated as the primary work item, with each human finding categorised as a missing case, missing technique class, weak assertion or unreachable surface so that investment goes to the right remedy? For Vendors, the assessor should be shown the data-minimisation test and the fallback test result.
3. Do new public techniques reach the corpus within 14 days at the 90th percentile, and does the organisation contribute at least four accepted artifacts per year in majority-runnable form with disclosure coordinated? In Vendors, ask to see the data-minimisation test and the fallback test result.

Common pitfalls

• ❌ Measuring adversarial testing by engagement count, which rewards activity and says nothing about whether defects are found faster.
• ❌ Treating the automation gap as an inevitable residue rather than as the work, which leaves the battery converged on itself.
• ❌ Reporting coverage without the surprise rate, so a rising coverage number conceals a static real-world gap.
• ❌ Contributing papers rather than runnable corpora, which peers read and cannot execute.
• ❌ Releasing technique demonstrations touching a provider's systems without coordinating disclosure first.

Business Function: Operations

Environment Hardening (EH)

Objective: Publish and apply a hardening baseline for each vendors AI archetype, so the operating environment enforces the controls rather than documenting them.

A program at this level has a published hardening baseline per archetype and applies it. The baseline covers single sign-on enforced for every sanctioned AI vendor with no local accounts, access brokered through an organisational gateway rather than direct credentials, data-loss prevention on the egress path to the vendor, training opt-out configured where available, tenant-level administrative controls locked to a named owner, and audit logging enabled and exported. The distinction that matters is enforcement: a control in enforcing mode changes what can happen, and a control in audit mode changes only what is recorded, and the two are frequently confused in reporting. Consumption limits are part of the baseline rather than an operational afterthought, because an environment with no limit has no defence against the one attack that leaves every availability signal green.

Level 1, Foundational

Publish and apply a hardening baseline for each vendors AI archetype, so the operating environment enforces the controls rather than documenting them.

Activities

A) Publish the hardening baseline per archetype — Author a hardening baseline per archetype covering single sign-on enforced for every sanctioned AI vendor with no local accounts, access brokered through an organisational gateway rather than direct credentials, data-loss prevention on the egress path to the vendor, training opt-out configured where available, tenant-level administrative controls locked to a named owner, and audit logging enabled and exported. Express the baseline as configuration that can be applied and checked rather than as prose, and publish it alongside the reference architecture patterns so that the secure configuration travels with the secure design. State for every control whether it is enforcing or advisory, and default to enforcing, because a baseline whose enforcement status is unstated is assumed to be enforcing by everyone reporting on it and is frequently not. Where a control genuinely cannot be enforced in the current environment, record it as a known limitation rather than listing it in the baseline as though it were active.

B) Apply the baseline and measure conformance against the running environment — Apply the baseline to every asset of the archetype and measure conformance from the environment rather than from a checklist. Read configuration state, policy enforcement mode and runtime behaviour, and report conformance per control rather than as a single score, because one aggregate number lets a fully conformant set of easy controls conceal a missing hard one. Report separately the count of controls running in audit rather than enforcing mode, which is the most common way a baseline is nominally met and practically absent. Reconcile monthly and treat any asset with no baseline applied as a finding with an owner rather than as an onboarding backlog item.

C) Configure consumption limits and govern exceptions — Configure per-tenant and per-user consumption limits configured at the broker as part of the baseline for every asset, expressed relative to per active seat so that the limit bears a defensible relationship to the work the asset actually does. Declare the behaviour at the limit explicitly, being degrade, queue or refuse, and never silently continue, because a limit that is exceeded without effect is a reporting artefact rather than a control. Operate a baseline exception process: any control not applied requires an owner, a compensating control, an approver and an expiry, reviewed monthly, with expired exceptions treated as findings. Exceptions are healthy when visible and corrosive when absorbed into the environment as the new normal.

Outcome metrics

Oversight metrics — Baseline exceptions approved without a compensating control recorded (detects: exception approval becoming a formality, where the control is waived and nothing replaces it, which is indistinguishable from not having the control at all) · Controls switched from enforcing to audit mode, with the change approved and time-bounded (detects: silent downgrade, where a control is moved to audit mode to resolve an operational problem and left there, so the baseline reports as met while enforcing nothing)

Consumption metrics — Assets with a consumption limit configured and a declared at-limit behaviour per per production asset · Configured limit against observed consumption per per active seat · Limit breaches where the declared at-limit behaviour did not occur per per limit breach event

Success criteria

• A hardening baseline is published per archetype as applicable configuration rather than prose, and applied to at least 90% of production vendors AI assets.
• At least 90% of baseline controls run in enforcing mode, with every audit-mode control recorded as an exception with an owner.
• Every production asset has a consumption limit configured with an explicit, declared at-limit behaviour.
• Conformance is reported per control rather than as an aggregate score that can conceal a missing hard control.
• Every baseline exception carries an owner, compensating control, approver and expiry, with none past expiry.

Assessment questions

1. Is a hardening baseline published per vendors archetype as applicable configuration rather than prose, applied to at least 90% of production assets, and measured against the running environment rather than a checklist?
2. Are at least 90% of baseline controls in enforcing rather than audit mode, with audit-mode controls recorded as exceptions, and is conformance reported per control rather than as an aggregate that can conceal a missing hard control? For Vendors, the assessor should be shown the broker configuration showing single sign-on enforced and limits applied.
3. Does every production asset have per-tenant and per-user consumption limits configured at the broker configured with an explicit declared at-limit behaviour, and does every baseline exception carry an owner, compensating control, approver and expiry?

Common pitfalls

• ❌ Leaving enforcement status unstated, so controls in audit mode are reported and believed to be enforcing.
• ❌ Reporting a single aggregate conformance score, which lets a set of easy controls conceal a missing hard one.
• ❌ Measuring conformance from a checklist rather than from the running environment, which records intent rather than state.
• ❌ Configuring a limit without declaring the behaviour at the limit, producing a threshold that is crossed without effect.
• ❌ Allowing exceptions without compensating controls, which is indistinguishable from not having the control.

Level 2, Comprehensive

Calibrate vendors hardening intensity to asset tier, detect and remediate baseline drift continuously, and make consumption limits tier-proportionate and tested.

Activities

A) Publish tier-calibrated baselines — Publish a baseline variant per tier rather than one baseline with exceptions. Critical adds per-tenant isolation, no standing human access with time-bound elevation, shorter secret rotation intervals, stricter egress allowlisting and full logging; Low takes single sign-on and the essential controls. Deriving the applicable variant from the inventory tier removes the negotiation that otherwise happens per asset and resolves toward the lighter variant. Publish the reasoning alongside the variants so that a team subject to the Critical baseline understands why, because a baseline that arrives without justification is complied with minimally and circumvented where possible. Drift in this domain most often appears as a training opt-out flag reset by a vendor configuration change and a direct integration added beside the broker, neither of which the organisation initiates.

B) Detect and remediate drift continuously — Stand up continuous drift detection across the baseline controls for Critical- and High-tier assets, reading configuration state, policy enforcement mode and runtime behaviour, and alerting on change rather than on schedule. Publish remediation windows by tier and severity, tighter at Critical, and route detections into the shared issue backlog with tier-appropriate severity rather than into a dashboard. Drift is the defining failure of hardening at scale because environments are hardened once and then changed for operational reasons that are individually reasonable: a rule widened for a debugging session, a quota raised under load, an enforcement mode relaxed to unblock a deployment. Each is defensible and none is reverted unless something notices. Limit exercises in this domain drive consumption to the broker-side ceiling, since vendor-side limits are frequently unavailable and the broker is the only enforcement point the organisation controls.

C) Make limits tier-proportionate and exercise them — Set consumption limits proportionate to tier and to the asset's observed workload, expressed relative to per active seat. Exercise the at-limit behaviour at least annually for Critical- and High-tier assets by driving the asset to its limit in a controlled test and confirming that the declared behaviour actually occurs, because a limit whose behaviour has never been observed is an assumption. Record the exercise result. Treat a cost-driven change to a limit, a logging level, a retention period or an enforcement mode on a Critical-tier asset as a material change requiring approval, since these are precisely the controls that cost pressure erodes and the erosion is invisible on any security dashboard. Cost pressure in this domain lands first on seat counts, where reclaiming unused seats is genuinely good and cutting logging exports is not.

Outcome metrics

Oversight metrics — Time-bound elevations to Critical-tier environments that expired without being reviewed (detects: elevation becoming effectively standing access, where a time bound exists on paper and is renewed automatically without anyone assessing whether it is still needed) · Drift remediation deferrals at Critical tier, with a named acceptance authority (detects: deferral becoming the default response to drift under operational pressure, which converts continuous detection into continuous reporting) · Enforcement-mode relaxations approved without an expiry (detects: a temporary relaxation becoming permanent because nothing in the approval obliges anyone to revisit it)

Consumption metrics — Critical- and High-tier assets whose at-limit behaviour has been exercised and recorded per per Critical or High tier asset · Limit headroom: configured limit against observed peak consumption, by tier per per active seat · Cost-driven relaxations of limits, logging or retention on Critical-tier assets per per approved change

Success criteria

• A baseline variant per tier is published with its reasoning and derived automatically from the inventory.
• At least 95% of Critical-tier assets are under continuous drift detection, with detection within 3 business days.
• Standing human access to Critical-tier AI environments is zero, with time-bound elevation only.
• Every Critical-tier asset's at-limit behaviour has been exercised within 12 months and the result recorded.
• Cost-driven changes to limits, logging, retention or enforcement mode at Critical tier require approval as material changes.

Assessment questions

1. Is a baseline variant published per tier with its reasoning and derived automatically from the inventory, rather than one baseline with per-asset exceptions that resolve toward the lighter option? Evidence this in the Vendors domain against the broker configuration showing single sign-on enforced and limits applied.
2. Are at least 95% of Critical-tier assets under continuous drift detection with a 3 business day detection window, with detections routed into the shared backlog rather than a dashboard, and is standing human access to Critical-tier environments zero? For Vendors, the assessor should be shown the broker configuration showing single sign-on enforced and limits applied.
3. Has every Critical-tier asset's at-limit behaviour been exercised within 12 months, and do cost-driven changes to limits, logging, retention or enforcement mode require approval as material changes? In Vendors, ask to see the broker configuration showing single sign-on enforced and limits applied.

Common pitfalls

• ❌ Running one baseline with per-asset exceptions rather than tier variants, which turns every asset into a negotiation resolving toward the lighter option.
• ❌ Detecting drift on a schedule rather than on change, which reintroduces the point-in-time weakness the activity exists to remove.
• ❌ Renewing time-bound elevations automatically, which produces standing access with a time bound written on it.
• ❌ Declaring limits without exercising them, so their behaviour is first learned during the event they exist for.
• ❌ Exempting cost-driven changes from approval, which is how hardened environments degrade with everyone's agreement.

Level 3, Industry-Leading

Make vendors AI environments self-correcting, with baselines applied at provisioning and drift remediated automatically where it is safe to do so.

Activities

A) Apply baselines at provisioning — Move baseline application to provisioning time so that an asset is conformant from creation rather than hardened afterwards. Implement the baseline as infrastructure-as-code modules, admission policies and platform defaults, so the conformant configuration is the default path and a non-conformant asset requires deliberate effort to create. Block provisioning of Critical-tier assets that would not meet baseline. Measure the time an asset spends non-conformant after creation and drive it to zero, since that window is the one period in an asset's life when it is both unhardened and unmonitored, and it is the window most exploitation of new infrastructure actually uses. Provisioning-time conformance in this domain means a vendor integration cannot be created outside the broker, which is the only structural defence against direct integrations added for latency.

B) Automate remediation within enumerated bounds — Enumerate explicitly the drift classes that may be remediated automatically, being those where the correct state is unambiguous and reversion carries no operational risk, such as re-enabling logging, restoring an enforcement mode, or closing an egress rule not in the allowlist. Everything outside that list alerts to a human. Log every automated action with what changed, what it was reverted to and why, and make the log reviewable, because self-correction that is silent can mask a genuine operational need indefinitely and the team whose change was reverted will simply make it again. Track repeated remediation of the same control on the same asset as a signal that the baseline is wrong or the operational need is real, and route it to review rather than continuing to revert. Safe automatic remediation in this domain covers re-setting a training opt-out flag reset by the vendor, which is one of the few actions where the organisation can correct a change it did not make.

C) Govern adaptive limits and publish the automation boundary — Allow consumption limits to adapt to observed workload within governed bounds, so that a limit tracks the asset's real demand rather than remaining static and becoming either irrelevant or obstructive. Set a hard ceiling that adaptation cannot cross without human approval, because an adaptive limit with no ceiling accommodates an attack as readily as it accommodates growth, and that is precisely the failure mode denial-of-wallet exploits. Express bounds relative to per active seat. Publish the list of controls that cannot be automatically applied or remediated and review it quarterly for reduction, so that the boundary of the automation is known rather than assumed to be complete. Adaptive limits in this domain track consumption per active seat, since per-seat pricing hides the user whose automation consumes at a rate no human could.

Outcome metrics

Oversight metrics — Automated remediations reverted by a human within 24 hours (detects: automation acting against a genuine operational need, which indicates the enumerated safe class was wrongly scoped and is invisible unless reversions are tracked) · Adaptive limit ceiling increases approved, with the approver and justification recorded (detects: ceiling creep, where each individual increase is justified and the cumulative effect is an unbounded limit nobody decided to create) · Alerts for non-automatable drift acknowledged but not remediated within the tier window (detects: alert fatigue on the drift classes that specifically require human judgment, which are the ones automation deliberately left to people)

Consumption metrics — Adaptive limits operating within a governed hard ceiling per per asset with an adaptive limit · Cumulative ceiling increase over the trailing year, per asset per per active seat · Drift events on consumption-limit configuration, and time spent drifted per per limit configuration drift event

Success criteria

• At least 95% of new assets are provisioned already conformant, with non-conformance windows at or under one hour.
• The drift classes eligible for automatic remediation are enumerated explicitly, and everything outside alerts to a human.
• Every automated remediation is logged with change, reversion and reason, and is reviewable.
• Repeated remediation of the same control is treated as a signal and routed to review rather than continuing to revert.
• Adaptive limits operate within a governed hard ceiling that requires human approval to cross.

Assessment questions

1. Are at least 95% of new assets provisioned already conformant with non-conformance windows at or under one hour, and is provisioning of non-conformant Critical-tier assets blocked? Evidence this in the Vendors domain against the broker configuration showing single sign-on enforced and limits applied.
2. Are the drift classes eligible for automatic remediation enumerated explicitly with everything else alerting to a human, and is every automated action logged with change, reversion and reason and reviewable rather than silent? For Vendors, the assessor should be shown the broker configuration showing single sign-on enforced and limits applied.
3. Is repeated automatic remediation of the same control treated as a signal routed to baseline or operational review, and do adaptive limits operate within a governed hard ceiling requiring human approval to cross? In Vendors, ask to see the broker configuration showing single sign-on enforced and limits applied.

Common pitfalls

• ❌ Hardening after provisioning rather than at it, leaving a window where the asset is both unhardened and unmonitored.
• ❌ Automating remediation without enumerating the safe classes, which eventually reverts something that mattered operationally.
• ❌ Making self-correction silent, so the team whose change was reverted simply makes it again and nobody learns.
• ❌ Treating repeated automatic remediation as a success metric rather than as evidence the baseline or the operational need is wrong.
• ❌ Allowing adaptive limits without a hard ceiling, which accommodates an attack as readily as growth.

Issue Management (IM)

Objective: Run one backlog for vendors AI issues with an AI-aware severity taxonomy and published service levels, so AI findings age and escalate like everything else.

A program at this level handles AI issues on the same rails as every other security issue, with a severity taxonomy extended to cover the failure modes AI actually produces. The taxonomy names a vendor breach affecting organisational data, a training-data term changed without notice, a vendor model change altering tested behaviour, an unsanctioned vendor found in production use, and a sub-processor added without approval, because a severity scheme built for classical vulnerabilities rates these either as nothing or as whatever the assessor improvises. One backlog matters more than it sounds: AI findings kept in a separate register age quietly, escalate to nobody, and are rediscovered at the next assessment.

Level 1, Foundational

Run one backlog for vendors AI issues with an AI-aware severity taxonomy and published service levels, so AI findings age and escalate like everything else.

Activities

A) Extend the severity taxonomy to AI failure modes — Extend the existing severity taxonomy rather than writing a new one, adding the AI-specific classes that a classical scheme cannot rate: a vendor breach affecting organisational data, a training-data term changed without notice, a vendor model change altering tested behaviour, an unsanctioned vendor found in production use, and a sub-processor added without approval. For each class, state what makes it Critical, High, Medium or Low, using the dimensions that actually vary, being the data exposed, the reversibility of the effect, the population affected and whether the asset acts or advises. Include denial-of-wallet as a named class with its own criteria, because it degrades no availability signal and a scheme keyed to availability rates it as nothing while the budget drains. Publish worked examples per class, since a taxonomy without examples is applied inconsistently and its inconsistency is invisible in the aggregate.

B) Run one backlog with wired-in sources and service levels — Route every AI finding into the shared security backlog rather than a separate AI register. Wire the sources so findings arrive automatically: implementation review, security testing, monitoring detections, adversarial exercises, vendor advisories and self-reports. Assign a severity, an owner and a service level at creation rather than at triage, because an issue that waits for triage to acquire a service level is an issue whose clock has not started. Publish service levels by severity and report adherence rather than backlog size, since backlog size is a function of discovery rate as much as of remediation and improves when discovery stops.

C) Document containment for AI-specific incidents — Write containment actions for each AI-specific incident class before they are needed, covering suspending the vendor integration at the gateway, revoking single sign-on access, invoking the fallback path, and triggering contractual notification obligations. AI containment differs from classical containment in a way that catches teams out: the fastest effective action is usually to remove the system's ability to act rather than to take the system offline, because an AI asset that is merely unavailable stops delivering value while an asset whose tool access is revoked stops causing harm and may keep delivering. Name who is authorised to invoke each action at what hour, and verify the technical means exists and works. An action documented but never exercised is discovered to be unavailable at the worst moment.

Outcome metrics

Oversight metrics — Issues closed without a recorded remediation or acceptance (detects: closure as backlog hygiene rather than resolution, where an issue is marked done because it is old rather than because anything changed) · Severity downgrades applied after creation, with the downgrading authority recorded (detects: severity drift under service-level pressure, where reclassification is used to make a clock more comfortable rather than to correct an assessment)

Consumption metrics — Denial-of-wallet incidents detected, with time from onset to containment per per denial-of-wallet incident · Consumption anomaly cost accrued before containment per per active seat · AI incidents whose containment action was a consumption limit rather than a service withdrawal per per contained AI incident

Success criteria

• All AI issues are managed in the shared security backlog with the same ageing and escalation as other security issues.
• The severity taxonomy names the AI-specific classes including denial-of-wallet, with published criteria and worked examples.
• At least 95% of issues carry a severity, owner and service level at creation rather than at triage.
• At least 80% of issues arrive from a wired source rather than manual entry.
• Containment actions are documented per AI incident class and technically verified as available.

Assessment questions

1. Does the severity taxonomy extend the existing scheme with AI-specific classes including a vendor breach affecting organisational data and denial-of-wallet, with published criteria keyed to data exposed, reversibility, population affected and whether the asset acts or advises, plus worked examples per class?
2. Are all AI issues managed in the shared security backlog with at least 95% carrying severity, owner and service level at creation, and do at least 80% arrive from a wired source rather than manual entry? For Vendors, the assessor should be shown an incident record showing gateway suspension and fallback invocation.
3. Are containment actions documented for every AI incident class, covering suspending the vendor integration at the gateway and the others, with named invocation authority and technical availability verified rather than assumed?

Common pitfalls

• ❌ Keeping AI findings in a separate register, where they age quietly, escalate to nobody and are rediscovered at the next assessment.
• ❌ Using a severity scheme keyed to availability, which rates denial-of-wallet as nothing while the budget drains.
• ❌ Assigning severity at triage rather than creation, so the issue's clock does not start until someone looks at it.
• ❌ Reporting backlog size rather than service-level adherence, a number that improves when discovery stops.
• ❌ Documenting containment actions without verifying the technical means exists, which is discovered at the worst moment.

Level 2, Comprehensive

Calibrate vendors AI issue service levels to asset tier, run rehearsed AI incident response, and close the loop from root cause to the practice that owns it.

Activities

A) Calibrate service levels and authority to tier — Publish remediation service levels by severity and asset tier, tighter at Critical, and report adherence per tier rather than in aggregate, because an aggregate figure can be met while every Critical-tier breach is missed. Make deferral and risk-acceptance authority rise with tier and severity, so a deferred Critical finding requires executive acceptance with an expiry and quarterly review. Report deferrals to the sponsor alongside closures, since a practice reporting only what it closed describes a backlog that does not exist. Track findings deferred and later implicated in an incident as a distinct measure of whether deferral judgment is sound rather than merely authorised. Tier-proportionate response in this domain is constrained by dependency, since containment means losing a capability the business relies on and the fallback determines whether that is tolerable.

B) Rehearse AI incident response and instrument the notification clocks — Write and exercise incident playbooks for each AI-specific class, covering detection signals, containment using suspending the vendor integration at the gateway, revoking single sign-on access, invoking the fallback path, and triggering contractual notification obligations, evidence preservation, communication and recovery. Exercise at least semi-annually with the people who would actually respond, including out of hours, and measure the exercise by the gaps it finds rather than by its completion. Instrument the regulatory notification clocks explicitly: identify which incident classes trigger EU AI Act serious-incident reporting or GDPR breach notification, what starts the clock, who decides, and what evidence the notification requires. A team that discovers a 72-hour obligation 40 hours into an incident has lost most of the window to a question it could have answered in advance. Playbook exercises in this domain rehearse suspending an integration and running on the fallback, which is where organisations discover the fallback was never actually built.

C) Route root causes to the owning practice — Require every incident reaching a root cause to name the practice that owns the fix and to open a work item there, rather than closing the incident where it was found. An incident caused by a missing architectural control belongs to secure architecture, one caused by an untested threat belongs to security testing, one caused by a knowledge gap belongs to education. Closing it in the incident record alone guarantees recurrence, and recurrence is the single clearest signal that a response practice is functioning as a repair shop rather than as a feedback mechanism. Track the share of incidents whose root cause was already known and untreated, which separates a discovery problem from an action problem and is usually uncomfortable reading. Root causes in this domain most often belong to the contract rather than to any technical practice, which makes procurement a necessary participant in the review.

Outcome metrics

Oversight metrics — Incidents closed without a root cause recorded, by severity (detects: closure without diagnosis, which guarantees recurrence and is most common under load at exactly the times recurrence is least affordable) · Risk acceptances at Critical tier, with the accepting authority and expiry recorded (detects: acceptance authority drifting downward under pressure, so the decision to live with a Critical risk is taken by whoever is available) · Exercise findings closed without a change to the playbook or the environment (detects: rehearsal treated as a compliance activity rather than as a source of correction, which is visible only by tracking what the exercise changed)

Consumption metrics — Denial-of-wallet incidents contained within their tier service level per per denial-of-wallet incident · Cost accrued between detection and containment, by tier per per active seat · Emergency consumption limits applied during incidents and subsequently reverted per per applied emergency limit

Success criteria

• Service levels vary by tier and severity with adherence at or above 90% reported per tier rather than in aggregate.
• Every AI incident playbook has been exercised within 12 months, including an out-of-hours exercise, and measured by the gaps it found.
• Regulatory notification clocks are instrumented with owners and evidence requirements identified in advance.
• At least 90% of incidents reaching root cause open a work item in the practice that owns the fix.
• Incidents whose root cause was already known and untreated are tracked openly and falling.

Assessment questions

1. Do service levels vary by asset tier and severity with adherence reported per tier rather than in an aggregate that can be met while every Critical-tier breach is missed? Evidence this in the Vendors domain against an incident record showing gateway suspension and fallback invocation.
2. Have AI incident playbooks been exercised within 12 months including out of hours, measured by the gaps found, and are regulatory notification clocks instrumented in advance with owners and evidence requirements identified? For Vendors, the assessor should be shown an incident record showing gateway suspension and fallback invocation.
3. Do at least 90% of incidents reaching root cause open a work item in the owning practice rather than closing where they were found, and is the share whose root cause was already known and untreated tracked openly? In Vendors, ask to see an incident record showing gateway suspension and fallback invocation.

Common pitfalls

• ❌ Reporting service-level adherence in aggregate, which can be met while every Critical-tier breach is missed.
• ❌ Writing playbooks without exercising them, so their gaps are found during the incident rather than before it.
• ❌ Discovering a notification obligation mid-incident, by which point most of the window is gone.
• ❌ Closing incidents where they were found, which guarantees the same root cause returns.
• ❌ Reporting closures without deferrals, which describes a backlog that does not exist.

Level 3, Industry-Leading

Automate vendors AI issue triage and containment where it is safe, measure response by outcome rather than activity, and contribute incident learning back to the field.

Activities

A) Automate triage and bounded first-line containment — Automate severity assignment, ownership routing and duplicate detection from the detection signal, so an issue arrives with its clock started and its owner assigned. Enumerate explicitly the containment actions that may fire automatically, being those where the correct response is unambiguous and reversible, such as applying an emergency consumption limit, revoking a specific tool credential or blocking a destination. Everything else alerts. Log every automated action with what was done, on what evidence, and how to reverse it. Publish the human-judgment classes explicitly, because automated containment is safest when its boundary is stated rather than assumed, and the assumption is always that it covers more than it does. Automatic containment in this domain is limited by dependency: suspending an integration at the gateway is available, but the fallback must exist first or containment simply becomes an outage.

B) Measure response by outcome rather than activity — Shift the primary measures from throughput to outcome: recurrence rate by root-cause class, harm avoided estimated from the consumption ledger and the affected population, and time from detection to containment rather than to closure. Ticket throughput and backlog size remain as health indicators but are explicitly not the practice's outcome measures, because both are dominated by discovery rate and both improve when discovery degrades. Report recurrence to the sponsor as the headline, since a response practice that closes quickly and recurs constantly is failing at the thing it exists for while every throughput metric looks excellent. Recurrence in this domain clusters around vendor-initiated changes the organisation did not anticipate, which points remediation at contract terms rather than at any technical control.

C) Contribute anonymised incident learning back — Contribute at least four substantive artifacts per year: anonymised incident case studies, detection and containment patterns, playbooks others can adopt, and technique observations submitted to MITRE ATLAS where they fit. The public corpus of real AI security incidents is thin, disproportionately so relative to the deployment base, which means most organisations learn only from their own incidents, which is the slowest and most expensive way to learn anything. Contributing changes that for everyone including the contributor. Anonymise carefully, review legally, coordinate disclosure with any affected vendor or provider, and count only accepted artifacts. Prefer contributing the incidents that were most surprising internally, since those carry the most information for others. Vendor incident write-ups contribute unusually well because a term change or silent feature enablement at one customer is almost always happening at others simultaneously.

Outcome metrics

Oversight metrics — Automated containment actions reversed by a human within one hour (detects: automation acting on a misread signal, which indicates the enumerated class was wrongly scoped and is only visible if reversals are tracked separately from actions) · Incidents in human-judgment classes acknowledged but not actioned within the tier window (detects: alert fatigue concentrating precisely on the classes automation deliberately left to people because they need judgment) · Recurrence accepted without a corresponding work item in the owning practice (detects: recurrence normalised as a cost of doing business, which is how a response practice becomes a repair shop)

Consumption metrics — Automated emergency consumption limits applied within the detection window per per denial-of-wallet detection · Cost accrued between detection and automated containment per per active seat · Harm avoided from contained consumption incidents, computed from burn rate and time to contain per per contained consumption incident

Success criteria

• At least 85% of issues arrive auto-triaged with severity, owner and clock assigned.
• Automatic containment classes are enumerated explicitly, everything else alerts, and every automated action is logged with evidence and a reversal path.
• Recurrence rate by root-cause class is the practice's headline measure, reported to the sponsor, rather than throughput or backlog size.
• Harm avoided is estimated from the consumption ledger and affected population and reported annually.
• At least four accepted external incident-learning contributions per year, anonymised and disclosure-coordinated.

Assessment questions

1. Are at least 85% of issues auto-triaged with severity, owner and clock assigned, and are automatic containment classes enumerated explicitly with everything else alerting and every action logged with evidence and a reversal path? Evidence this in the Vendors domain against an incident record showing gateway suspension and fallback invocation.
2. Is recurrence rate by root-cause class the practice's headline outcome measure reported to the sponsor, rather than throughput and backlog size which are dominated by discovery rate and improve when discovery degrades? For Vendors, the assessor should be shown an incident record showing gateway suspension and fallback invocation.
3. Does the organisation contribute at least four accepted anonymised incident-learning artifacts per year with disclosure coordinated, prioritising the incidents that were most surprising internally? In Vendors, ask to see an incident record showing gateway suspension and fallback invocation.

Common pitfalls

• ❌ Automating containment without enumerating the safe classes, which eventually acts on a misread signal in a way that matters.
• ❌ Measuring the practice by throughput, which is dominated by discovery rate and improves when discovery degrades.
• ❌ Tracking automated actions without tracking reversals, which hides automation that is acting wrongly.
• ❌ Normalising recurrence as a cost of doing business, which is how a response practice becomes a repair shop.
• ❌ Withholding incident learning, which keeps the whole field, including the withholder, learning only from its own incidents.

Monitoring & Logging (ML)

Objective: Establish a logging baseline per vendors archetype and a detection set covering the top threats, with retention set by the longest applicable obligation.

A program at this level can see what its AI is doing. Each archetype has a logging baseline covering requests brokered to each vendor with the data classes involved, vendor administrative and configuration changes, single sign-on grants and revocations, vendor advisories and model-change notifications received, seat provisioning and activity, and consumption against commitment, and a detection set targeting the threats the threat library rates highest. Retention is set by the longest applicable regulatory obligation rather than by storage cost, because the retention decision is usually made on cost grounds by people who do not know the obligation and is discovered to be wrong during an investigation.

Level 1, Foundational

Establish a logging baseline per vendors archetype and a detection set covering the top threats, with retention set by the longest applicable obligation.

Activities

A) Publish and verify the logging baseline per archetype — Define a logging baseline per archetype covering requests brokered to each vendor with the data classes involved, vendor administrative and configuration changes, single sign-on grants and revocations, vendor advisories and model-change notifications received, seat provisioning and activity, and consumption against commitment. Verify that each stream is actually emitting rather than merely configured, by asserting the presence of recent events per stream per asset, because a logging configuration that exists and emits nothing passes every configuration check and fails the one investigation that needs it. Include the identity and context fields that make a log answerable: who initiated, what asset, which model and version, which session. A log that records that something happened without recording which model version did it cannot answer the question that follows any output-integrity incident. Publish the baseline alongside the hardening baseline so telemetry travels with configuration.

B) Build the detection set from the threat library — Build detections for the threats the TA library rates highest, covering in this domain a direct vendor integration bypassing the broker, a training opt-out flag reset by a vendor configuration change, a new data class appearing in a vendor payload, a vendor model change without notification, and consumption per seat departing from the tenant baseline. Map each detection to the threat identifiers it addresses and publish the threats with no detection, because coverage is the claim that matters and detection count is not a proxy for it. Route every detection to the shared issue backlog with a severity from the IM taxonomy rather than to a dashboard, since a detection whose only destination is a dashboard has no owner and no clock. Include at least one consumption-anomaly detection from the outset, expressed against a denominated metric, because it is the cheapest detection available and the only one that covers denial-of-wallet.

C) Set retention by obligation and make logs usable — Set retention per log class by the longest applicable regulatory, contractual or investigative requirement, and record which obligation drives each period so the decision is reviewable. Retention is routinely set on storage cost by people who do not hold the obligation, and the error surfaces during an investigation or a regulator request when it can no longer be corrected. Apply redaction at write rather than at read so that sensitive content is not retained in the clear pending a query-time control. Verify that logs are actually queryable at the volumes an investigation would need, because a retention period that cannot be searched within a working day is not meaningfully retained.

Outcome metrics

Oversight metrics — Detections acknowledged but closed without an investigation note (detects: alert closure as queue management rather than as analysis, which is invisible in any alert-volume metric and is the normal response to a noisy detection set) · Detections muted or suppressed for more than one week, with the suppressing authority recorded (detects: silent suppression, where a noisy detection is muted to clear a queue and never re-enabled, so coverage falls with no decision taken)

Consumption metrics — Consumption-anomaly detections active against a denominated metric per per archetype detection set · Detection threshold expressed against consumption per unit of work rather than absolute spend per per active seat · Logging and retention cost per retained record, by log class per per retained record

Success criteria

• A logging baseline is defined per archetype and verified as emitting for at least 85% of vendors archetypes.
• At least 70% of top-rated library threats are covered by a detection, with uncovered threats published as gaps.
• Every log class has a retention period traced to a named obligation rather than to storage cost.
• Every active detection routes to the shared issue backlog with an IM severity rather than to a dashboard.
• At least one consumption-anomaly detection operates against a denominated metric from the outset.

Assessment questions

1. Is a logging baseline defined per vendors archetype covering identity, asset, model and version context, and verified as actually emitting rather than merely configured, for at least 85% of archetypes?
2. Do detections cover at least 70% of the top-rated threats from the library with uncovered threats published as gaps, and does every detection route to the shared issue backlog with an IM severity rather than to a dashboard? For Vendors, the assessor should be shown the broker request log and the direct-integration bypass detection.
3. Is retention per log class traced to a named obligation rather than to storage cost, with redaction applied at write, and are log streams verified as queryable within a working day at investigation volumes? In Vendors, ask to see the broker request log and the direct-integration bypass detection.

Common pitfalls

• ❌ Verifying that logging is configured rather than that it is emitting, which passes a stream that has been silent for months.
• ❌ Omitting model and version from the log context, which makes every output-integrity investigation start from scratch.
• ❌ Setting retention on storage cost, an error discovered during an investigation when it can no longer be corrected.
• ❌ Routing detections to a dashboard, where they have no owner and no clock.
• ❌ Building a detection set with no consumption anomaly, leaving the one attack class that produces no availability signal entirely uncovered.

Level 2, Comprehensive

Calibrate vendors detection and logging depth to asset tier, tune for alert quality, and establish behavioural baselines so departure is detectable.

Activities

A) Calibrate logging and detection depth to tier — Publish what each tier receives: Critical gets full logging, the complete detection set tuned for the asset, behavioural baselining and the shortest alerting path; Low gets baseline logging and the shadow-AI detections. Derive the applicable set from the inventory tier automatically. Publish the retention and sampling position per tier explicitly, because sampling is where cost pressure lands first and a Critical-tier asset whose telemetry is sampled has monitoring that is statistical at exactly the tier where specific answers are needed. Treat a sampling or retention reduction at Critical tier as a material change requiring approval rather than an operational tuning decision. Behavioural baselining in this domain covers consumption per seat and payload composition, since the organisation sees only its own side of the interaction and must infer the rest.

B) Establish behavioural baselines per Critical-tier asset — Build a behavioural baseline per Critical-tier asset covering its normal operating envelope: request and tool-call volumes, the distribution of actions taken, typical session shapes, identity and access patterns, and consumption per unit of work. Alert on departure rather than only on signature, because the AI failure modes that matter most produce traffic that is individually well-formed and collectively abnormal, and a signature-based set cannot see an agent competently doing something nobody asked for. Refresh baselines on a defined cadence and after any material change to the asset, and record baseline shifts explicitly, since an unexplained baseline shift is itself the signal and is lost if baselines are refreshed silently. Detection-as-code in this domain is fragile to vendor API changes, which the organisation does not control and often learns about from the detection breaking.

C) Tune for alert quality and report precision with recall — Measure detection quality using analyst disposition labels, and report precision and recall together. Precision alone rewards a detector that fires twice a year and misses everything, which is exactly the detector a tuning programme under alert-fatigue pressure will produce. Set a false-positive budget per detection and tune or retire detections that exceed it, because analysts who learn that a detection is usually wrong stop reading it and the coverage is then notional. Assert continuously that each detection is still evaluating and each stream still emitting, alerting within hours, since a stopped detection and a clean environment are indistinguishable in every dashboard. Vendor detection content contributes well as broker-side rules, which peers using the same providers can deploy directly.

Outcome metrics

Oversight metrics — Alert disposition distribution per analyst, compared across analysts on comparable queues (detects: disposition drift, where the same alert would be escalated by one analyst and closed by another, making detection outcome depend on shift rather than on evidence) · Alerts per analyst per shift against the published ceiling (detects: analyst overload, where triage quality falls as volume rises while closure rates continue to look healthy) · Critical-tier detections closed as false positive and later implicated in an incident (detects: dismissal under load of exactly the alerts that mattered, visible only by tracing closures forward)

Consumption metrics — Consumption-anomaly detections operating against a per-asset behavioural baseline per per active seat · Time from a consumption anomaly beginning to its detection per per consumption anomaly · Telemetry cost per retained record at Critical tier, tracked against sampling decisions per per retained record

Success criteria

• Logging and detection depth are published per tier and derived automatically from the inventory.
• At least 90% of Critical-tier assets have a current behavioural baseline, refreshed on cadence and after material change, with shifts recorded.
• Every active detection reports precision and recall together, with a false-positive budget enforced by tuning or retirement.
• A silent log stream or stopped detection is detected within 4 hours.
• Sampling or retention reductions at Critical tier are approved as material changes rather than treated as operational tuning.

Assessment questions

1. Are logging and detection depth published per tier and derived automatically from the inventory, with sampling and retention positions stated per tier and reductions at Critical tier treated as material changes? Evidence this in the Vendors domain against the broker request log and the direct-integration bypass detection.
2. Do at least 90% of Critical-tier assets carry a current behavioural baseline alerting on departure rather than only on signature, with baseline shifts recorded explicitly rather than refreshed silently? For Vendors, the assessor should be shown the broker request log and the direct-integration bypass detection.
3. Is detection quality measured with precision and recall reported together using analyst disposition labels, with a false-positive budget enforced, and is a stopped detection or silent stream detected within 4 hours? In Vendors, ask to see the broker request log and the direct-integration bypass detection.

Common pitfalls

• ❌ Reporting precision without recall, which rewards a detector that fires twice a year and catches nothing.
• ❌ Relying on signature detections alone, which cannot see an agent competently doing something nobody asked for.
• ❌ Refreshing behavioural baselines silently, which discards the baseline shift that was itself the signal.
• ❌ Allowing sampling at Critical tier as an operational tuning decision, making monitoring statistical where specific answers are needed.
• ❌ Assuming a quiet detection is a clean environment, when a stopped detection looks identical.

Level 3, Industry-Leading

Engineer vendors detections as versioned, tested code, close the loop from incident to detection continuously, and contribute detection content back to the field.

Activities

A) Engineer detections as versioned, tested code — Move detection content into version control with the same discipline as application code: each detection carries an owner, a version, a documented intent, and test cases that assert it fires on known-positive events and does not fire on known-negative ones. Run the tests continuously against replayed or synthetic events, because the most common silent failure in monitoring is a detection that still exists, still evaluates and no longer matches anything due to a log-format change upstream, and nothing in a conventional monitoring stack reports that. Deploy through a pipeline with review, so a detection change is as traceable as a code change and can be reverted as easily. Behavioural baselining in this domain covers consumption per seat and payload composition, since the organisation sees only its own side of the interaction and must infer the rest. Detection tests in this domain replay broker-side request records, which is all the organisation has, and the unassertable list is correspondingly longer.

B) Close the incident-to-detection loop continuously — Require every incident and every adversarial finding to be assessed for detection coverage within days: did a detection fire, should one have, and if so why did it not. Where no detection existed, author one and deploy it within 10 business days, with its test cases. Where one existed and did not fire, treat that as a detection defect with an owner rather than as an incident detail. Track the share of incidents detected internally before external report as the practice's headline measure, since it is the one number that expresses whether monitoring is working, and report it alongside coverage rather than instead of it. Detection-as-code in this domain is fragile to vendor API changes, which the organisation does not control and often learns about from the detection breaking.

C) Contribute detection content back — Contribute at least four substantive artifacts per year in deployable form: detection rules, behavioural baselining approaches, log schemas for AI telemetry and the test cases that verify a detection works. Shared detection content is scarce in AI security while the failure modes are largely common across organisations, which makes this among the highest-leverage contributions available, since a peer can deploy a rule and cannot deploy a description of one. Work through OWASP projects, MITRE ATLAS, sector information-sharing bodies and detection-engineering communities. Anonymise, review legally, coordinate disclosure where content reveals a provider weakness, and prioritise contributing the detections that most often produce actioned findings internally. Vendor detection content contributes well as broker-side rules, which peers using the same providers can deploy directly.

Outcome metrics

Oversight metrics — Detection changes deployed without review, by severity of the detection (detects: detection content changed directly in the console, which is untraceable, unrevertable and the most common way coverage is lost without a decision) · Detections whose tests have been failing for more than one week without action (detects: test failure normalised, which restores exactly the silent-failure mode the tests were introduced to remove) · Incidents detected externally rather than internally, with a coverage assessment completed (detects: external detection accepted without asking why internal detection missed it, which forfeits the single most informative signal monitoring produces)

Consumption metrics — Consumption-anomaly detections held as tested code with known-positive cases per per consumption detection · Time from consumption anomaly onset to internal detection per per active seat · Telemetry and detection platform cost per monitored asset per per monitored asset

Success criteria

• At least 85% of active detections are held in version control with an owner, documented intent and test cases.
• Detection tests run continuously against replayed or synthetic events, with failures alerting within the day.
• Every incident and adversarial finding is assessed for detection coverage within 5 business days, and uncovered cases produce a deployed detection within 10.
• Internal detection share is reported as the practice's headline measure alongside coverage.
• At least four accepted external contributions per year in deployable form.

Assessment questions

1. Are at least 85% of active detections held in version control with an owner, documented intent and test cases running continuously against replayed or synthetic events, so a detection broken by an upstream log change is caught by its own test? Evidence this in the Vendors domain against the broker request log and the direct-integration bypass detection.
2. Is every incident and adversarial finding assessed for detection coverage within 5 business days, with uncovered cases producing a deployed detection and its tests within 10, and detections that existed but did not fire treated as defects with owners? For Vendors, the assessor should be shown the broker request log and the direct-integration bypass detection.
3. Is internal detection share reported as the headline outcome measure alongside coverage, and does the organisation contribute at least four accepted detection artifacts per year in deployable rather than descriptive form? In Vendors, ask to see the broker request log and the direct-integration bypass detection.

Common pitfalls

• ❌ Leaving detections in a console rather than version control, which makes changes untraceable and coverage loss undecided.
• ❌ Deploying detections without test cases, which leaves the silent-failure mode after an upstream log change entirely uncovered.
• ❌ Normalising failing detection tests, which restores exactly the problem the tests were introduced to remove.
• ❌ Accepting external detection of an incident without asking why internal detection missed it, forfeiting the most informative signal available.
• ❌ Contributing descriptions of detections rather than deployable rules, which peers read and cannot run.

Part V, Assessment workbook

The full questionnaires for this domain are in ../questionnaires/, one per practice, nine questions each. Work them in dependency order from Part III.

Scoring

Fully Mature 1.0 · Implemented 0.67 · Partial 0.33 · Not Implemented 0.0. A level is achieved when all three of its questions score 0.67 or better. Levels are derived, never self-assigned, and every claim is graded at its weakest evidence grade (E0–E3).

Generated by tools/render.py from the Vendors domain cell records.