HAIAMM v3.5, Data Domain Handbook

AI/HAI Data Assurance, the complete practice set for this domain

Generated document. Compiled from the canonical cell records in cells/. Do not edit by hand, edit the cell and re-run python3 tools/render.py domain.

Canonical model: HAIAMM-v3.5-Framing.md · Full model: HAIAMM-v3.5-Handbook.md

Part I, The domain

Subject of this domain: the training, retrieval, inference and evaluation data that AI/HAI systems consume and produce

Boundary rule

The asset is the data, not the system that reads it. A poisoned retrieval store is Data; the pipeline that queries it is Software.

Archetypes

Cells in this domain key their content to these archetypes. A generated cell must be usable against every archetype listed, or must say explicitly which archetypes it applies to and why.

1. Training corpus / training dataset
2. Inference input stream
3. Retrieval store
4. Prompt/completion log corpus
5. Embedding store
6. Fine-tuning dataset
7. Evaluation / test set

A single asset is often several archetypes at once. Generated content must accommodate that rather than assuming a single clean classification.

Shadow AI in this domain

DLP on AI egress — regulated or confidential data classes leaving for AI endpoints without a sanctioned path.

Every domain's SM L1 carries a domain-appropriate shadow-AI discovery mechanism. Downstream practices at L1 assume the inventory it produces exists.

Consumption unit and denominator

Unit: tokens per retrieval, embedding re-index cost, fine-tuning and training compute, prompt/completion log retention cost

Denominator: per retrieval, per indexed document, per training run

What it detects here: extraction reconnaissance against retrieval and embedding stores (AML.T0024) — systematic extraction is expensive, and the cost profile is its tell; also retrieval amplification and over-collection.

Meter: vector-store and retrieval telemetry, training-job cost records, storage cost-allocation tags.

Consumption is a governed resource in v3.5, not an incidental discovery signal. Cells for the consumption-sensitive practices (SM, SR, EH, ML, IM) divide this domain's unit by this domain's denominator; absolute spend is a ledger input and never a metric.

Primary stakeholders

Data Governance · Legal / Privacy Counsel · AI/ML Engineer · Compliance / Audit

HAI-TTP emphasis

RA dominates via training-data poisoning (AML.T0020); AGH via retrieval-store poisoning; extraction and inversion against embedding and log corpora.

Compliance overlay

GDPR Art. 5 / 6 / 17 / 22 and DSAR mechanics; EU AI Act Art. 10 data governance; sector retention rules.

Part II, Effort and ownership

What this domain costs to run, per cell, incremental over the level below. These are the BSAMM-style costs and personnel the v3.0 handbooks promised and did not ship.

Reaching L3 across all twelve practices in the Data domain is roughly 6,400 hours of stand-up and 4,500 hours per quarter of ongoing effort for a mid-size organization, incremental and cumulative across the three levels. L1 alone is a far smaller commitment, and is where the dependency graph says to start.

Tooling called for in this domain

• AI telemetry ingestion into the existing detection platform
• Adaptive limit management with governed ceilings
• Adversarial testing capability, internal or contracted
• Adversary emulation tooling and an isolated exercise environment
• Automated content classification on ingest and on schedule
• Automated design criteria evaluation in the delivery pipeline
• Automated remediation platform with scoped action classes
• Automated tier re-derivation triggered by catalogue field changes
• Automated triage and bounded containment orchestration
• Behavioural baselining across AI telemetry
• Canary record seeding and detection across governed stores
• Configuration conformance scanning against the baseline definitions
• Continuous adversarial testing capability
• Continuous configuration drift detection against declared control state
• Continuous conformance assertion evaluation against existing telemetry
• Continuous conformance verification against configuration and runtime state
• Continuous control assertion framework spanning pipeline and runtime
• Continuous drift detection across configuration, policy mode and runtime state
• DLP rules extended to AI-provider egress endpoints
• Detection quality measurement with disposition labelling
• Detection-as-code pipeline with continuous test execution
• Evidence collection automation against existing systems of record
• Evidence map collection wired to repository, CI and configuration systems
• Evolving adversarial harness with case generation
• Executable pattern artifacts in the platform scaffolding
• Existing learning management system, extended with applied-scenario assessment
• Extraction-pattern detection on retrieval stores
• Guidance search and question telemetry instrumentation
• Hands-on laboratory environment for Critical-tier curricula
• Lineage event emission across extract, transform, embed, train and evaluate steps
• Policy-as-code engine integrated into the delivery pipeline
• Regression corpus management with freshness tracking
• Test harness capable of exercising AI-specific controls in CI
• Threat model register linked to the asset inventory and change events

Part III, Build order

L1 dependencies within this domain, from the cell records. Start where nothing is required.

Part IV, The twelve practices

Business Function: Governance

Strategy & Metrics (SM)

Objective: Stand up AI data governance, inventory every corpus AI touches, and baseline the metrics that prove regulated data is not leaving governed stores for AI services.

A program at this level knows which data its AI systems consume and produce. One inventory covers all seven data archetypes, from training corpora through retrieval stores to the prompt and completion logs that are themselves a regulated corpus most organizations forget they created. Data-loss prevention on AI egress is the domain's shadow-AI mechanism, and canary-tagged datasets make unsanctioned flows detectable rather than theoretical. Every corpus has a named steward, a classification, and a lineage record, so the question of whether customer data reached a model has an answer that does not require an investigation.

Level 1, Foundational

Stand up AI data governance, inventory every corpus AI touches, and baseline the metrics that prove regulated data is not leaving governed stores for AI services.

Activities

A) Charter AI data governance and name the stewards — Publish an AI data governance charter within the first 30 days, sponsored jointly by the CISO and the Chief Data Officer or equivalent, and co-signed by Privacy counsel. The charter names the seven in-scope archetypes, being training corpus, fine-tuning dataset, inference input stream, retrieval store, embedding store, prompt and completion log corpus, and evaluation or test set; it assigns a named data steward per corpus rather than per system, because corpora outlive the systems that created them; it establishes three priority policies, being AI Data Use, Data Acceptable Use for AI, and the data intake and sanction gate; and it defines decision rights over which data classes may reach which model deployments, who approves an exception and who owns the revocation path when a flow must stop. The working group includes Data Governance, Privacy and Legal counsel, AI/ML engineering, Compliance and the platform team that operates the stores.

B) Inventory the corpora and instrument AI-egress detection — Build one AI data inventory recording, per corpus, the archetype, steward, classification against the enterprise scheme, volume and growth rate, lineage from originating source, the model deployments that consume it, whether it is used at training or inference or both, retention period and legal basis, cross-border flow and its transfer mechanism, and subject-access exposure. Seed it from the data catalogue, pipeline telemetry, training-run registries, cloud storage inventories and access-control groups. Then instrument the domain's shadow-AI mechanism: data-loss prevention rules on egress to model-provider endpoints, tuned to the classified data patterns that matter rather than to everything, and canary records seeded into governed stores so that a canary appearing in a model output or an external log is unambiguous evidence of an ungoverned flow. Treat prompt and completion logs as a first-class corpus in the inventory; they accumulate regulated data by default, they are frequently replicated to observability vendors, and they are the corpus most often missed at the first inventory pass.

C) Baseline the data scoreboard and open the retrieval ledger — Instrument and report quarterly: inventory coverage against discovered corpora, the share of corpora carrying a classification and a named steward, regulated-data flows to AI services that lack a sanctioned path, canary-trigger events, and the share of corpora with a defined retention period. Open the consumption side of the ledger for this domain, recording retrieval cost per query, embedding re-index cost per indexed document, and training-run cost per run, each attributed to the corpus and the owning team. Retention cost belongs in the ledger explicitly, because it is the mechanism by which log corpora get silently truncated to save money, taking with them the evidence that monitoring and deployer-duty obligations depend on. Record the denominator per archetype rather than reporting absolute storage or query spend, and where corpus-level cost allocation does not exist, record the ledger as blocked on allocation and raise the tagging requirement with FinOps.

Outcome metrics

Consumption metrics — Retrieval cost per query, by retrieval store per per retrieval query · Embedding re-index cost per indexed document per per indexed document · Log-retention cost per retained prompt-completion record per per retained record

Success criteria

• An AI data governance charter is published and jointly sponsored, with three priority policies and a named steward per corpus.
• One AI data inventory covers all seven archetypes at 85% or better coverage, including prompt and completion log corpora.
• DLP on AI egress and canary tagging are both operating, with zero unsanctioned regulated-data flows sustained.
• At least 90% of corpora carry a retention period and legal basis, and 100% of log corpora do.
• A retrieval and training consumption ledger is open with a declared denominator per archetype, or explicitly recorded as blocked on allocation.

Assessment questions

1. Is there a published AI data governance charter with a named steward per corpus, three priority policies covering AI data use, acceptable use and the intake gate, and explicit decision rights over which data classes may reach which model deployments?
2. Does one AI data inventory cover all seven archetypes including prompt and completion log corpora, recording classification, lineage, retention, legal basis, cross-border flow and subject-access exposure, at 85% coverage or better against discovered corpora?
3. Are DLP on AI-endpoint egress and canary tagging both operating as the domain's shadow-AI mechanism, with regulated-data flows lacking a sanctioned path standing at zero and every canary trigger investigated within five business days?

Common pitfalls

• ❌ Omitting prompt and completion logs from the inventory, which is the corpus most likely to accumulate regulated data and most likely to be replicated to an observability vendor.
• ❌ Classifying corpora once at creation and never revisiting, so a store that began with public documents and later ingested customer tickets retains its original classification.
• ❌ Tuning DLP for AI egress so broadly that it generates unactionable volume, after which the alerts are muted and the control exists only nominally.
• ❌ Assigning stewardship to systems rather than corpora, so a corpus is orphaned the moment its originating pipeline is decommissioned.
• ❌ Reporting storage and query spend in absolute terms, which grows with adoption and reveals nothing about whether the corpus is governed.

Level 2, Comprehensive

Tier the corpora by sensitivity and exposure, calibrate governance intensity per tier, and bound the consumption of high-tier retrieval and training workloads.

Activities

A) Publish the corpus risk-tier rubric — Define four tiers derived deterministically from six auditable dimensions. Data classification is the floor, where regulated personal, health, payment or customer-confidential content places a corpus at Critical or High. Training-versus-inference use elevates, because a corpus that shapes model weights carries a permanence that an inference input stream does not, and cannot be retracted by deleting a record. Decision-affecting use elevates to Critical where the corpus feeds a system making decisions with legal or significant effect under GDPR Art. 22 or falling within EU AI Act Annex III. Cross-border flow elevates where the transfer basis is anything other than adequacy. Subject-access exposure elevates where records are attributable to identifiable individuals and therefore carry erasure and portability obligations that must be executable against a model, not only against a store. Lineage quality lowers confidence and therefore elevates tier where provenance is incomplete, because a corpus of unknown origin cannot be defended. Publish the rubric with its derivation rule, permit overrides only with recorded rationale, and re-derive tiers automatically when a classification or exposure field changes.

B) Calibrate governance intensity and budgets per tier — Publish a tier-treatment matrix covering, per tier: access-control model from broad read to per-request authorization with purpose binding; lineage depth from source attribution to full transformation provenance; retention discipline and deletion-propagation obligations including whether erasure must propagate to embeddings and fine-tuned weights; encryption and residency requirements; monitoring detections including extraction-pattern detection on retrieval stores; testing obligations including poisoning-resistance and extraction-resistance batteries; and re-review triggers on material change. Add the consumption row: Critical- and High-tier retrieval stores and training workloads carry declared budgets with a defined behaviour at the cap, because systematic extraction against a retrieval store and an unbounded re-index loop both present first as cost. Treat a retention reduction, a sampling change or a cache extension on a Critical-tier corpus as a material change triggering re-review, since each alters the evidence base that monitoring and regulatory obligations rest on.

C) Report per tier and govern corpus tier movement — Report the scoreboard per tier and per archetype quarterly to the sponsor. Report inventory state, classification and steward coverage, unsanctioned-flow counts, canary triggers, retention conformance and per-tier service-level adherence, with every Critical-tier corpus as its own row. Report the Assured Spend Ratio for retrieval and training consumption per tier, targeting full attribution at Critical. Maintain a tier-movement log capturing every corpus that changed tier, with the triggering dimension recorded, and review it quarterly; corpus tier movement is more common than software tier movement because content drifts continuously, so a store that starts with public documentation and gradually accumulates customer tickets must surface as an upgrade rather than persist under its founding classification. Reconcile the steward roster against HR records each quarter so that departures produce a reassignment rather than an orphan.

Outcome metrics

Consumption metrics — Retrieval cost per query at Critical tier against the store's rolling median per per retrieval query · Training and fine-tuning compute cost per completed run, by corpus tier per per completed training run · Retention cost per retained record at Critical tier, tracked against retention conformance per per retained record

Success criteria

• A corpus tier rubric is published and applied to 100% of the inventory, with tiers re-derived automatically on classification or exposure change.
• A tier-treatment matrix is published covering access control, lineage depth, retention, monitoring and testing, and downstream Data-domain L2 practices are calibrated to it.
• Erasure requests propagate to embeddings and derived artifacts within the statutory window for all Critical- and High-tier corpora.
• Critical- and High-tier retrieval stores and training workloads operate inside declared budgets with a tested at-cap behaviour.
• A corpus tier-movement log is maintained with the triggering dimension recorded, and reviewed quarterly with the sponsor.

Assessment questions

1. Is every corpus assigned a tier from a published rubric covering classification, training-versus-inference use, decision-affecting use, cross-border flow, subject-access exposure and lineage quality, re-derived automatically when classification or exposure changes?
2. Is there a published tier-treatment matrix covering access control, lineage depth, retention and deletion propagation, encryption and residency, monitoring detections and testing obligations, with at least 95% of Critical-tier corpora having received full-scope treatment in the last 12 months?
3. Do erasure requests propagate to embeddings and derived artifacts within the statutory window for all Critical- and High-tier corpora, and do Critical-tier retrieval and training workloads operate inside declared budgets with a tested behaviour at the cap?

Common pitfalls

• ❌ Tiering corpora once and never re-deriving, so a store whose content drifted from public documentation to customer tickets keeps its founding classification indefinitely.
• ❌ Treating erasure as a store-level operation, so records are deleted from the source while their contribution persists in embeddings and fine-tuned weights.
• ❌ Assuming extraction against a retrieval store will look anomalous in access logs alone, when its most reliable early signal is the cost profile.
• ❌ Publishing a tier-treatment matrix that downstream practices treat as advisory, so detection and testing remain uniform and the tiering achieves nothing.
• ❌ Reducing log retention on a Critical-tier corpus as a cost measure without routing it through change governance.

Level 3, Industry-Leading

Drive corpus tiering and lineage from live pipeline signals, benchmark data-governance unit economics externally, and contribute data-attack intelligence back to the ecosystem.

Activities

A) Automate lineage, classification and tier from pipeline events — Derive the data picture from the pipelines themselves rather than from periodic curation. Emit lineage events from every extract, transform, embed, train and evaluate step, so a corpus record is assembled from what the pipeline actually did rather than from what a steward believed it would do. Run automated classification over corpus samples on a schedule and on ingest, so content drift triggers re-classification and therefore re-tiering without waiting for a human to notice that a documentation store has begun accumulating customer tickets. Wire tier changes to open the downstream obligations automatically across access control, monitoring, testing and retention. Publish a data-quality service-level objective covering lineage completeness, classification freshness and tier correctness within 48 hours of a material content change. Reserve human curation for three cases: corpora whose provenance cannot be established automatically, conflicting classification signals, and genuinely novel archetypes that the existing seven do not describe.

B) Benchmark data-governance unit economics and detection efficacy — Publish a semi-annual brief comparing the program against named peers on at least five metrics: corpus inventory coverage, erasure propagation time, extraction-detection precision and recall, retrieval cost per query by store archetype, and training cost per completed run. Unit economics matter here more than in any other domain, because retrieval and embedding costs are the ones most often optimised without a security review; a store whose cost per query is far below the peer median may be efficient, or it may be skipping the authorization and logging that peers pay for. Draw comparison data from FinOps Foundation AI-scope working groups for the economics, and from MITRE ATLAS practitioner exchanges, sector information-sharing bodies and CSA AI Safety Initiative groups for detection efficacy. Name the peer cohort and defend its selection; record where the program trails as explicitly as where it leads, and route the deltas into the next investment cycle.

C) Contribute observed data-attack intelligence and compute the domain return — Contribute at least four substantive artifacts per year drawn from the organization's own telemetry: poisoning and extraction techniques observed against first-party corpora submitted to MITRE ATLAS, retrieval-store and embedding-store attack patterns contributed to OWASP LLM working groups, data-governance reference patterns published through OpenSSF AI, responsibly disclosed findings submitted to the AI Vulnerability Database, and NIST AI RMF Playbook data-chapter contributions. Anonymize and legally vet every submission, and count only artifacts that external bodies accept, because talks and press releases do not move the ecosystem. Alongside this, compute the domain's return from the consumption ledger rather than asserting it: reclaimed re-indexing waste, avoided retention overspend where retention was right-sized to the longest applicable obligation rather than to habit, extraction attempts contained valued at burn rate multiplied by detection time, and storage reclaimed by retiring corpora that no model consumes. Reconcile the result with Finance annually.

Outcome metrics

Consumption metrics — Retrieval cost per query benchmarked against the external peer cohort median per per retrieval query · Reclaimed re-indexing and retention waste per per indexed document retained · Extraction-attempt containment value: burn rate multiplied by time to detect per per contained extraction attempt

Success criteria

• Lineage, classification and tier are event-derived at 90% or better, with a published latency objective met at the 99th percentile.
• Extraction and poisoning detections are tuned from first-party telemetry, with precision and recall both published rather than precision alone.
• A semi-annual benchmarking brief covers at least five peer-comparable metrics including retrieval and training unit economics, with the peer cohort named and defended.
• At least four data-attack intelligence artifacts are accepted by external bodies per year, anonymized and legally vetted.
• The domain return is computed from ledger arithmetic and reconciled with Finance annually.

Assessment questions

1. Are corpus lineage, classification and tier derived from pipeline events at 90% or better against a published data-quality service-level objective covering correctness within 48 hours of material content change, with human curation reserved for unestablishable provenance and conflicting signals?
2. Does the semi-annual benchmarking brief compare at least five peer-comparable metrics including retrieval cost per query and training cost per run, with a named peer cohort whose selection is defended and where the program trails recorded as explicitly as where it leads?
3. Does the program contribute at least four externally accepted data-attack intelligence artifacts per year from its own telemetry, and is the domain return computed from ledger arithmetic covering reclaimed waste and contained extraction rather than asserted?

Common pitfalls

• ❌ Automating lineage without a completeness objective, so gaps accumulate invisibly and stewards revert to trusting their own notes.
• ❌ Publishing detection precision while withholding recall, which makes a narrow detector look excellent and hides everything it never fires on.
• ❌ Benchmarking retrieval cost per query without checking detection coverage, so skipped controls read as efficiency.
• ❌ Treating automated classification as authoritative for corpora whose provenance was never establishable, rather than routing them to human curation.
• ❌ Counting submitted rather than accepted contributions, which converts a quality gate into an activity count.

Policy & Compliance (PC)

Objective: Publish the priority compliance map and the minimum policy stack governing the data AI/HAI systems consume and produce, so every obligation reaches a named policy and a named owner.

A program at this level has written down which rules apply to the data AI/HAI systems consume and produce and who answers for each. A one-page priority compliance map traces every in-scope obligation to a policy clause and an evidence artifact, so a regulator asking how a given article is addressed reaches one row, one policy and one piece of evidence rather than a research project. The AI Data Use Policy is published, acknowledged by the in-scope population, and specific enough to be violated, which is the test that separates a policy from a statement of values.

Level 1, Foundational

Publish the priority compliance map and the minimum policy stack governing the data AI/HAI systems consume and produce, so every obligation reaches a named policy and a named owner.

Activities

A) Publish the priority compliance map — Produce a one-page priority compliance map covering the obligations that actually apply to the data AI/HAI systems consume and produce: GDPR Art. 5 principles; GDPR Art. 6 lawful basis; GDPR Art. 17 erasure; GDPR Art. 30 records; EU AI Act Art. 10 data governance; ISO/IEC 42001 7.5, plus any sector-specific rules in scope. Each row names the obligation, the specific clause of the internal policy that implements it, the accountable owner, the evidence artifact that demonstrates conformance, and the practice that produces that evidence. The map is one page by design. A compliance register of two hundred rows is a document nobody reads and nobody maintains, whereas a one-page map that traces the obligations a regulator is actually likely to ask about is a document that survives contact with an audit. Review it quarterly and whenever a new obligation enters scope, and treat an obligation with no named evidence artifact as an open finding rather than as a row to be filled in later.

B) Publish the AI Data Use Policy and the supporting policy stack — Publish an AI Data Use Policy stating which data classes may reach which model deployments, a Data Acceptable Use standard for AI, and an intake and sanction gate that every new corpus passes before a model consumes it. Write each requirement so that conformance is observable: a rule stating that AI must be used responsibly cannot be assessed, whereas a rule naming the seven data archetypes, being training corpus, fine-tuning dataset, inference input stream, retrieval store, embedding store, prompt and completion log corpus, and evaluation set and stating what is permitted for each can be. Route the policy through the same approval and review cycle as the enterprise security policy stack rather than creating a parallel governance path, and set an explicit review interval of at most 12 months. Publish a short summary aimed at the population that must follow it, because a policy that requires reading twenty pages to answer a five-second question will be guessed at instead. Establish an exception process with a named approver, a required rationale, a mandatory expiry date and a register, since undocumented exceptions are how policy stacks quietly become fiction.

C) Attest the population and baseline conformance — Run an attestation campaign across the in-scope population, delivered through the existing human resources or learning system so that completion is recorded where compliance evidence already lives. Track coverage as a percentage of the in-scope population rather than as a count of completions, because the denominator is what an auditor asks for. Baseline three conformance measures in the first quarter: the share of obligations on the map carrying a named evidence artifact, the share of the population with a current acknowledgement, and the count of active exceptions with their age distribution. Reconcile the map against the data catalogue, lineage records and retention configuration to confirm that what the policy requires is what the environment actually does, and record every divergence as a finding with an owner rather than as a note. Report all three to the executive sponsor quarterly.

Outcome metrics

Success criteria

• A one-page priority compliance map is published, tracing every in-scope obligation to a policy clause, an owner and an evidence artifact.
• The AI Data Use Policy is published with an explicit review interval of at most 12 months and written so that violations are identifiable.
• At least 95% of the in-scope population holds a current acknowledgement.
• An exception process operates with named approvers, mandatory expiry dates and a reviewed register carrying no expired entries.
• Policy requirements are reconciled against the data catalogue, lineage records and retention configuration, with every divergence raised as a finding with an owner.

Assessment questions

1. Is there a published one-page priority compliance map tracing every in-scope obligation governing the data AI/HAI systems consume and produce to a specific policy clause, an accountable owner, an evidence artifact and the practice that produces it?
2. Is the AI Data Use Policy published with requirements written so conformance is observable, routed through the enterprise policy review cycle, carrying a review interval of at most 12 months, and acknowledged by at least 95% of the in-scope population?
3. Does an exception process operate with named approvers, mandatory rationale and expiry dates and a register carrying no expired entries, and are policy requirements reconciled against the data catalogue, lineage records and retention configuration with divergences raised as findings?

Common pitfalls

• ❌ Building a compliance register of hundreds of rows instead of a one-page map, which nobody reads, nobody maintains and no auditor is helped by.
• ❌ Writing policy requirements that cannot be violated, such as requiring AI to be used responsibly, which makes conformance unassessable.
• ❌ Creating a parallel AI policy governance path separate from the enterprise stack, which doubles the review burden and halves the authority.
• ❌ Granting exceptions without expiry dates, after which the exception register becomes the real policy.
• ❌ Publishing the policy without reconciling it against what the environment actually does, so policy and practice diverge from day one.

Level 2, Comprehensive

Calibrate policy depth and evidence requirements to the risk tier of each data asset, and assemble tier-specific evidence packs that can be produced on demand.

Activities

A) Calibrate policy requirements per tier — Extend the L1 policy stack with tier-differentiated requirements keyed to the SM L2 rubric. For each requirement, state what Critical, High, Medium and Low each owe: which controls are mandatory versus recommended, what review cadence applies, what approval authority an exception requires, and what evidence must exist. A Critical-tier the data AI/HAI systems consume and produce might owe a documented conformance assessment, an annual independent review and a named accountable executive, where a Low-tier one owes an intake record and an annual self-attestation. Publishing the difference is what prevents the two failure modes that follow from uniform policy: either every asset carries Critical-tier burden, which makes the policy unenforceable and drives teams around it, or every asset carries Low-tier burden, which leaves the Critical ones under-governed. Route tier-calibrated requirements through the same approval cycle as the base policy. In this domain tier calibration turns mainly on data class and lawful basis: a corpus containing regulated personal data carries obligations that a public-documentation store does not, and erasure and portability duties attach to the corpus rather than to the system that reads it.

B) Build bidirectional control mappings and tier-specific evidence packs — Map each obligation on the priority compliance map to the specific controls that satisfy it, and map each control back to the obligations that depend on it. The reverse direction is the one usually missing and the one that matters operationally: when a control fails or is deferred, the programme needs to state immediately which obligations are now unevidenced, rather than discovering it during the next audit. Assemble an evidence pack per tier specifying exactly which artifacts constitute conformance, drawn from the data catalogue, lineage records and retention configuration and from the outputs of TA, SR, DR, IR, ST and ML. Automate collection where the artifact already exists in a system of record, and set a service level for producing a complete pack on request, targeting 10 business days at Critical tier. An evidence pack assembled only when asked for is a pack that will be incomplete the one time it matters. The evidence pack for data draws on the catalogue, lineage records, retention configuration and the records of processing, which regulators ask for in almost exactly that order.

C) Operate regulatory change management and tiered exceptions — Establish a defined path from a regulatory change to the assets it affects. Assign a named owner to monitor the in-scope obligation sources, assess each change for applicability within 30 days of publication, identify the affected assets by tier, and open remediation work items with owners and dates. Without this, regulatory change arrives through whoever happens to read an update, which is neither reliable nor auditable. Calibrate the exception process by tier: a Critical-tier exception requires executive approval, a shorter maximum duration and quarterly review, where a Low-tier exception can be approved by the practice lead with an annual review. Report exception volume and age by tier to the sponsor, and treat a rising Critical-tier exception count as a signal that the policy is mis-calibrated rather than that teams are non-compliant. Regulatory change in this domain most often arrives as transfer-mechanism and retention guidance, which can invalidate a lawful basis for a corpus that has not itself changed at all.

Outcome metrics

Success criteria

• Tier-calibrated policy requirements are published, stating what each tier owes in controls, cadence, approval authority and evidence.
• Evidence packs exist per tier and are refreshed on cadence, with a Critical-tier production service level of 10 business days met.
• Bidirectional control mappings are complete, so a failing control names the obligations it jeopardises.
• A named owner assesses every regulatory change for applicability within 30 days and opens remediation work items by tier.
• Exceptions are tier-calibrated in approval authority, maximum duration and review cadence, with Critical-tier volume and age reported to the sponsor.

Assessment questions

1. Are policy requirements calibrated by the SM L2 risk tier, stating for each tier what controls are mandatory, what review cadence applies, what approval authority an exception requires and what evidence must exist? Evidence this in the Data domain against the AI Data Use Policy and the lawful-basis record for one regulated corpus.
2. Do bidirectional control mappings exist so that a failing or deferred control immediately names the obligations it leaves unevidenced, and do tier-specific evidence packs get refreshed on cadence rather than assembled on request? For Data, the assessor should be shown the AI Data Use Policy and the lawful-basis record for one regulated corpus.
3. Does a named owner assess every regulatory change for applicability within 30 days of publication and open tier-prioritised remediation work items, and are exceptions tier-calibrated in approval authority, duration and review cadence? In Data, ask to see the AI Data Use Policy and the lawful-basis record for one regulated corpus.

Common pitfalls

• ❌ Applying one policy depth to every asset, which either over-burdens Low-tier work until teams route around the policy, or under-governs the Critical-tier assets that justify the programme.
• ❌ Mapping obligations to controls in one direction only, so a deferred control's regulatory consequence is discovered at audit rather than at the deferral decision.
• ❌ Assembling evidence packs only when requested, which guarantees the pack is incomplete on the one occasion it is needed urgently.
• ❌ Leaving regulatory monitoring to whoever happens to read an update, which is neither reliable nor demonstrable to an auditor.
• ❌ Treating a rising Critical-tier exception count as non-compliance by teams rather than as evidence that the policy is mis-calibrated.

Level 3, Industry-Leading

Generate compliance evidence continuously from the operating environment, prove conformance by assertion rather than assembly, and engage in the regulatory process for the data AI/HAI systems consume and produce.

Activities

A) Generate conformance evidence continuously — Instrument the environment so that each policy requirement has a continuously evaluated conformance assertion rather than a periodically collected artifact. Draw the signals from the data catalogue, lineage records and retention configuration and from the telemetry the other practices already emit, and express each requirement as a check that returns conformant, non-conformant or not-evaluable. The third state matters and is usually omitted: a requirement that cannot be evaluated is a gap in the evidence chain, not a pass, and collapsing it into conformant is how continuous compliance dashboards become reassuring and wrong. Publish a service level covering detection of a conformance drift within 5 business days. Reserve manual attestation for requirements that are genuinely not machine-evaluable, list them explicitly, and review that list quarterly with the aim of shrinking it, since the list is the honest measure of how far the automation actually reaches.

B) Prove conformance on demand and shrink audit preparation — Convert the evidence pack from an assembled document into a generated view over the continuous assertions, so that producing it is a query rather than a project. Target production within one business day at every tier, and run unannounced internal exercises at least twice a year to verify the claim, because an evidence pipeline that has never been exercised under time pressure is an assumption. Track audit preparation effort explicitly in hours and drive it toward zero: the measure of this level is not that audits pass, which they did before, but that they cost almost nothing to prepare for. Where an external audit or regulatory enquiry does raise a finding, trace it back to the assertion that should have caught it and either fix the assertion or record why the requirement is not machine-evaluable.

C) Engage in the regulatory and standards process — Participate in the process that produces the obligations rather than only consuming its output. Submit implementation experience to consultations on GDPR implementing acts and successor guidance, contribute to the standards bodies developing conformance methods including ISO/IEC 42001 and the relevant national standards organisations, and share anonymised implementation patterns through sector bodies and practitioner communities. Implementation experience is the input regulators most consistently lack and most consistently ask for, which makes it the contribution with the highest leverage and the lowest cost to produce, since the programme generates it as a by-product of operating. Target at least four substantive submissions per year, count only those accepted or published, and have legal review every submission before it leaves the organisation.

Outcome metrics

Success criteria

• At least 85% of policy requirements carry a continuously evaluated conformance assertion, with the non-evaluable remainder listed explicitly rather than counted as conformant.
• Conformance drift is detected within 5 business days at the 90th percentile.
• A complete evidence pack is produced within one business day at any tier, verified by unannounced exercise twice a year.
• Audit preparation effort is tracked in hours and falling year over year.
• At least four accepted regulatory or standards contributions per year, each legally reviewed before submission.

Assessment questions

1. Do at least 85% of policy requirements carry a continuously evaluated conformance assertion returning conformant, non-conformant or not-evaluable, with the non-evaluable set listed explicitly and reviewed quarterly for reduction rather than collapsed into conformant? Evidence this in the Data domain against the AI Data Use Policy and the lawful-basis record for one regulated corpus.
2. Is a complete evidence pack produced within one business day at any tier, verified by unannounced exercises at least twice a year, and is audit preparation effort tracked in hours and falling year over year? For Data, the assessor should be shown the AI Data Use Policy and the lawful-basis record for one regulated corpus.
3. Does the organisation make at least four accepted regulatory or standards contributions per year drawn from its own implementation experience, each reviewed by legal counsel before submission? In Data, ask to see the AI Data Use Policy and the lawful-basis record for one regulated corpus.

Common pitfalls

• ❌ Collapsing the not-evaluable state into conformant, which produces a compliance dashboard that is reassuring and wrong.
• ❌ Claiming one-day evidence production without ever exercising it unannounced, which makes the claim an assumption rather than a capability.
• ❌ Automating the assertions that were easy and leaving the non-evaluable list unpublished, so the automation's real reach is unknown.
• ❌ Measuring audit success rather than audit preparation cost, when the latter is what this level actually changes.
• ❌ Submitting to consultations without legal review, which can commit the organisation to positions it has not agreed internally.

Education & Guidance (EG)

Objective: Give the people who build and steward the corpora AI consumes the role-based AI security literacy and the practical guidance they need to make safe decisions without asking, in the Data domain.

A program at this level has stopped relying on general security awareness to cover AI-specific risk. Role-based training reaches data engineers, ML engineers running training and fine-tuning, data stewards, and the privacy counsel who assess the corpora, and it teaches the failure modes those roles actually encounter rather than a survey of AI risk in general. A guidance library sits where the work happens, so the answer to a routine question is reachable in under a minute rather than requiring a security consultation. Competence is assessed rather than assumed, because completion rates measure attendance and say nothing about whether anyone can now do the thing.

Level 1, Foundational

Give the people who build and steward the corpora AI consumes the role-based AI security literacy and the practical guidance they need to make safe decisions without asking, in the Data domain.

Activities

A) Define roles and build role-based curricula — Identify the roles in scope, being data engineers, ML engineers running training and fine-tuning, data stewards, and the privacy counsel who assess the corpora, and build a short curriculum per role rather than one course for everyone. A single general course is the most common failure in this practice: it is long enough to be resented, general enough to be forgettable, and calibrated to no one. Each curriculum covers the AI failure modes that role actually meets, which in this domain means training-data poisoning, extraction through retrieval and embedding stores, regulated data entering a corpus without a lawful basis, and erasure that never propagates past the source store, together with the specific controls that role is expected to apply and the escalation path when something looks wrong. Keep each curriculum under 90 minutes, because attention is the binding constraint and a curriculum nobody finishes teaches nothing. Set a 90-day competence deadline for new joiners in scope, and deliver through the existing learning system so that completion and assessment records land where compliance evidence already lives.

B) Assess competence rather than completion — Build an assessment per role that tests whether the learner can apply the material, not whether they sat through it. Include a hands-on exercise where the learner traces a customer record from source through embedding into a model response, then executes an erasure and verifies what survives, because a scenario the learner works through is the only assessment format that distinguishes recognition from capability, and recognition is what multiple-choice tests measure. Set a pass threshold, require a retake on failure, and record the pass rate per role as the practice's headline outcome metric. Publish item-level results to the curriculum owner so that a question the whole cohort fails is treated as a teaching failure rather than a learner failure, which is the usual cause. Re-assess at least annually and whenever the threat picture or the sanctioned toolset changes materially.

C) Publish the guidance library where the work happens — Publish a guidance library covering patterns for corpus classification, lineage capture, canary seeding, retention right-sizing and deletion propagation. The binding constraint is reachability, not completeness: guidance that requires leaving the working context to find will be guessed around, so place it where the work is, in the repository for engineers, in the workflow tool for process owners, in the service catalogue for requesters. Every guidance item names an owner and a review date, and stale guidance is withdrawn rather than left standing, because out-of-date guidance is worse than none and is trusted for longer. Publish a named route for questions the library does not answer, staffed with a response service level, and mine those questions monthly as the highest-quality signal available about where the guidance is missing or wrong.

Outcome metrics

Success criteria

• Every in-scope role among data engineers, ML engineers running training and fine-tuning, data stewards, and the privacy counsel who assess the corpora has a curriculum under 90 minutes covering the failure modes that role actually encounters.
• Competence is assessed by applied scenario rather than recall, with an 80% or better pass rate per role and per-role results published.
• New joiners in scope reach baseline literacy within 90 days, enrolled automatically from the joiner process.
• A guidance library sits where the work happens, with every item carrying an owner and a review date and stale items withdrawn.
• A named question route operates with a published response service level and is mined monthly for guidance gaps.

Assessment questions

1. Does every in-scope role have its own curriculum under 90 minutes covering the AI failure modes that role actually encounters in the Data domain, delivered through the existing learning system with a 90-day deadline for new joiners?
2. Is competence assessed by applied scenario rather than recall, with a published pass threshold, an 80% or better pass rate per role, and item-level results reviewed by the curriculum owner so cohort-wide failures are treated as teaching defects? For Data, the assessor should be shown the data-steward curriculum and the erasure-propagation exercise.
3. Does a guidance library sit where the work happens with every item carrying an owner and a review date, are stale items withdrawn rather than retained, and is there a named question route with a response service level that is mined monthly for gaps? In Data, ask to see the data-steward curriculum and the erasure-propagation exercise.

Common pitfalls

• ❌ Building one general AI security course for everyone, which is too long to finish, too general to apply, and calibrated to no actual role.
• ❌ Measuring completion rather than competence, which tells the programme who attended and nothing about what anyone can now do.
• ❌ Publishing guidance in a wiki nobody opens, when reachability from the working context is the whole of the control.
• ❌ Leaving stale guidance standing, which is worse than no guidance because it is trusted for longer.
• ❌ Treating cohort-wide assessment failures as a learner problem rather than as evidence the material taught the wrong thing.

Level 2, Comprehensive

Scale Data AI security literacy through tier-calibrated depth, a champions network embedded in the teams, and exercises drawn from the organisation's own incidents.

Activities

A) Calibrate curriculum depth to asset tier — Extend the L1 curricula with tier-calibrated depth keyed to the SM L2 rubric. Practitioners working on Critical- and High-tier assets take an extended curriculum with hands-on laboratory work and a higher pass threshold; those on Medium and Low take the baseline. Publish the mapping from asset tier to required curriculum so that assignment is derived rather than negotiated, and wire enrolment to the asset inventory so that a practitioner joining a Critical-tier team is enrolled automatically rather than when someone remembers. Set a currency requirement: Critical-tier practitioners re-assess annually, others every two years. Report pass rates and currency by tier rather than in aggregate, because an organisation-wide 85% pass rate is consistent with the Critical-tier cohort being the weakest, and aggregate reporting actively conceals exactly the case that matters. In this domain the extended curriculum has learners trace a customer record from source through embedding into a model response, then execute an erasure and observe exactly what survives, because the gap between deleting a row and removing its influence is the single most misunderstood thing in AI data governance.

B) Build and resource a champions network — Establish a security champion inside each team working on Critical- and High-tier assets. The champion is not a security professional and should not be treated as one: the role is to be the person who recognises when something needs the central function, to answer the routine questions locally, and to carry context the central team cannot have. Resource the role explicitly with allocated time, typically half a day a week, agreed with the engineering or business management chain rather than assumed as goodwill, because an unresourced champion network decays within two quarters and its decay is invisible until it is total. Give champions a deeper curriculum, a private channel to the central team, early access to threat and guidance updates, and visible recognition in performance processes. Meet the network monthly and mine it for what the central function cannot see. Data champions sit with the pipeline and platform teams, and their distinguishing value is knowing which corpus just acquired a new source, which the catalogue learns only at the next reconciliation.

C) Run exercises from the organisation's own incidents — Convert incidents and near-misses from the issue-management practice into teaching material on a defined cadence, targeting one new exercise per quarter. An exercise built from something that actually happened in this organisation carries an authority that no generic example achieves, and it converts the incident from a cost into an asset. Anonymise where necessary but keep the specifics that make it recognisable, since the recognisable detail is the part that is remembered. Run the exercises with the champions network first, then let champions run them with their teams, which both scales delivery and tests whether the champion can teach the material rather than only pass an assessment. Track which failure modes recur across exercises and feed that back to the threat assessment and guidance practices as evidence of where understanding is genuinely weak. Data exercises are built from real canary triggers and erasure failures, which teach faster than any explanation of why lineage matters.

Outcome metrics

Success criteria

• Curriculum depth is calibrated to asset tier with enrolment derived from the inventory, and pass rates are reported per tier rather than in aggregate.
• Every team working on Critical-tier assets has a trained champion with management-agreed allocated time.
• At least four exercises per year are built from the organisation's own incidents and near-misses.
• Champions resolve at least 60% of routine questions without central escalation.
• Recurring failure modes across exercises are fed back to threat assessment and guidance as evidence of genuine weakness.

Assessment questions

1. Is curriculum depth calibrated to the SM L2 asset tier with enrolment derived automatically from the inventory, and are pass rates and currency reported per tier rather than in an aggregate that could conceal a weak Critical-tier cohort? Evidence this in the Data domain against the data-steward curriculum and the erasure-propagation exercise.
2. Does every team working on Critical-tier assets have a trained champion with explicitly allocated time agreed with the management chain, and do champions resolve at least 60% of routine questions without central escalation? For Data, the assessor should be shown the data-steward curriculum and the erasure-propagation exercise.
3. Are at least four exercises per year built from the organisation's own incidents and near-misses, run first with champions and then by champions with their teams, with recurring failure modes fed back to threat assessment and guidance? In Data, ask to see the data-steward curriculum and the erasure-propagation exercise.

Common pitfalls

• ❌ Reporting one organisation-wide pass rate, which is consistent with the Critical-tier cohort being the weakest and conceals precisely that case.
• ❌ Appointing champions without allocated time, after which the network decays within two quarters and its decay is invisible until total.
• ❌ Treating champions as junior security staff rather than as embedded context-holders, which drives away the people best suited to the role.
• ❌ Using generic exercise material when the organisation's own incidents are available, discarding the authority that recognisable detail carries.
• ❌ Anonymising exercises so thoroughly that the recognisable specifics are gone, which removes the part learners actually remember.

Level 3, Industry-Leading

Make Data AI security learning continuous and evidence-driven, measure whether it changes behaviour rather than knowledge, and contribute curricula back to the field.

Activities

A) Drive curriculum and guidance from observed signal — Close the loop between what the organisation observes and what it teaches. Feed incident root causes, design and implementation review findings, security test failures and monitoring detections into a quarterly curriculum review, and treat any failure mode appearing three times as a mandatory curriculum change rather than a discussion item. Instrument the guidance library so that search terms returning nothing, and the questions reaching the named route, are captured as gap signals, because what practitioners look for and fail to find is the highest-quality guidance-gap data available and it is free. Publish a service level covering the interval from an observed recurring failure mode to updated guidance, targeting 30 days. Retire material that the signal shows is no longer relevant, since a curriculum that only grows becomes one nobody completes. In this domain the extended curriculum has learners trace a customer record from source through embedding into a model response, then execute an erasure and observe exactly what survives, because the gap between deleting a row and removing its influence is the single most misunderstood thing in AI data governance.

B) Measure behaviour change rather than knowledge — Move the headline measure from assessment scores to behaviour in the environment. Compare teams and cohorts before and after training on measures the other practices already produce: the rate of findings by category in implementation review, the reuse rate of sanctioned patterns, the proportion of designs arriving at review already conformant, the rate of self-reported near-misses, which should rise as psychological safety improves and is one of the few metrics where an increase is good news. Accept that attribution is imperfect and say so, rather than claiming a causal link the data cannot carry; the honest claim is that behaviour moved in the trained cohort and did not in the untrained one, over a stated period. Report behaviour measures alongside assessment results so that a high pass rate with unchanged behaviour is visible as the failure it represents. Data champions sit with the pipeline and platform teams, and their distinguishing value is knowing which corpus just acquired a new source, which the catalogue learns only at the next reconciliation.

C) Contribute curricula and measure capability in progression — Contribute at least four substantive artifacts per year to the field, weighted toward teaching material: role-based AI security curricula, applied exercise and laboratory scenarios, anonymised incident case studies that other organisations can teach from, and guidance patterns. Good AI security teaching material is the field's scarcest resource, far scarcer than frameworks or tooling, which makes curriculum contribution unusually high-leverage and unusually likely to be adopted. Work through OWASP education projects, sector bodies, professional associations and academic partnerships. Separately, establish AI security capability as a recognised factor in performance and progression for in-scope roles, agreed with human resources, because a capability the organisation says matters but never recognises is one practitioners correctly deprioritise. Data exercises are built from real canary triggers and erasure failures, which teach faster than any explanation of why lineage matters.

Outcome metrics

Success criteria

• Curricula and guidance update from observed incidents, findings and telemetry, with recurring failure modes reaching updated guidance within 30 days.
• Guidance gaps are detected from search misses and unanswered questions rather than from periodic review alone.
• At least three behaviour measures are tracked pre and post training, reported alongside assessment results with attribution limits stated.
• At least four externally accepted curriculum or exercise contributions per year.
• AI security capability is a recognised factor in performance and progression for every in-scope role, agreed with human resources.

Assessment questions

1. Do curricula and guidance update from incident root causes, review findings, test failures and detections, with any failure mode appearing three times triggering a mandatory curriculum change within 30 days, and are guidance gaps detected from search misses and unanswered questions? Evidence this in the Data domain against the data-steward curriculum and the erasure-propagation exercise.
2. Does the programme measure behaviour change in the environment using at least three downstream measures tracked pre and post training, reported alongside assessment results with attribution limits stated rather than a causal claim the data cannot carry? For Data, the assessor should be shown the data-steward curriculum and the erasure-propagation exercise.
3. Does the organisation contribute at least four externally accepted curriculum or exercise artifacts per year, and is AI security capability a recognised factor in performance and progression for every in-scope role? In Data, ask to see the data-steward curriculum and the erasure-propagation exercise.

Common pitfalls

• ❌ Reviewing curricula annually while the threat picture moves quarterly, so the material lags by a planning cycle permanently.
• ❌ Adding material without ever retiring any, producing a curriculum too long to complete and therefore completed by nobody.
• ❌ Claiming training caused a behaviour improvement when the data supports only correlation, which discredits the honest measures alongside it.
• ❌ Ignoring search misses and unanswered questions, which are the cheapest and highest-quality guidance-gap signal available.
• ❌ Declaring AI security capability important while never recognising it in progression, which practitioners correctly read as a statement of priority.

Business Function: Building

Threat Assessment (TA)

Objective: Publish an archetype-level threat library for the Data domain, tagged to the HAI TTPs and MITRE ATLAS, so threat modelling starts from a reusable baseline rather than a blank page.

A program at this level has written down what can go wrong with each data archetype, once, so that no team has to rediscover it. The library covers training corpus, fine-tuning dataset, inference input stream, retrieval store, embedding store, prompt and completion log corpus, and evaluation set, and every entry carries a HAI TTP tag, an ATLAS technique where one applies, an OWASP LLM reference and the practice expected to address it. This is what makes threat modelling tractable at scale: a team adopting a known archetype inherits its threat model and spends its effort on what is genuinely novel about their case, rather than re-deriving prompt injection from first principles every time.

Level 1, Foundational

Publish an archetype-level threat library for the Data domain, tagged to the HAI TTPs and MITRE ATLAS, so threat modelling starts from a reusable baseline rather than a blank page.

Activities

A) Build the archetype threat library — Author one threat snapshot per archetype covering training corpus, fine-tuning dataset, inference input stream, retrieval store, embedding store, prompt and completion log corpus, and evaluation set. For each archetype, identify the entities and trust boundaries first, being the ingest path, the transformation pipeline, the store itself, the retrieval interface and the derived embeddings, because a threat list without a boundary model is a list of worries rather than an analysis. Then enumerate the threats that follow from those boundaries, which in this domain centre on training-data poisoning that survives into model behaviour, systematic extraction through retrieval and inference interfaces, regulated data entering a corpus without a lawful basis, and evaluation contamination that makes a model look safer than it is. Keep each snapshot to a length a team will actually read, targeting two pages, and write each threat as a concrete failure with a named precondition rather than as a risk category. Derive the snapshots from the data catalogue, lineage records and pipeline definitions and validate each with one practitioner who works on that archetype daily, because a threat library authored entirely by security will describe the system security imagines rather than the one that exists.

B) Tag every threat to the shared vocabulary — Tag each threat entry with the HAI TTP it expresses, being excessive agency, agent goal hijack, tool misuse or rogue agents; the MITRE ATLAS technique where a real one applies; the OWASP LLM 2025 reference; and the compliance obligation it bears on where relevant. Use only technique identifiers that genuinely exist, because a plausible but invented identifier is worse than none, it survives review and misleads everyone downstream. Where no ATLAS technique fits, record that explicitly rather than forcing the nearest match, and treat the gap as a candidate contribution at L3. The tagging is what makes the library composable with everything else: it lets monitoring ask which threats have detections, testing ask which have tests, and requirements ask which have controls, without any of them re-reading the prose.

C) Trace every threat to a treating practice and publish the gaps — For every threat in the library, name the practice expected to address it and the specific control, requirement, pattern, test or detection that does so. Then publish the gaps, which is the activity teams most often skip and the one that gives the library its value: a threat with no named treatment is an accepted risk whether or not anyone decided to accept it, and making that visible is the point of the exercise. Report the treatment-coverage rate to the sponsor quarterly and treat an untreated threat at a high-risk archetype as an open finding with an owner rather than as a line in a register. Establish a 30-day service level for producing a threat snapshot when a new archetype enters the inventory, and review the whole library at least quarterly against external threat intelligence.

Outcome metrics

Success criteria

• A threat snapshot exists for at least 90% of inventoried data archetypes, each opening with entities and trust boundaries rather than a threat list.
• Every threat entry carries a HAI TTP tag and, where applicable, a genuine ATLAS technique identifier, with no invented references.
• At least 85% of threats trace to a named treating practice and specific control, with the untreated remainder published as gaps rather than omitted.
• Untreated threats at high-risk archetypes carry an owner and a remediation date.
• A 30-day snapshot service level operates for new archetypes, triggered from the inventory.

Assessment questions

1. Does a published threat snapshot exist for at least 90% of inventoried data archetypes, each establishing entities and trust boundaries before enumerating threats, validated with a practitioner who works on that archetype?
2. Does every threat entry carry a HAI TTP tag, a genuine MITRE ATLAS technique where one applies with gaps recorded rather than forced, an OWASP LLM reference and the relevant compliance obligation? For Data, the assessor should be shown the threat snapshot for the retrieval store, with its poisoning and extraction paths.
3. Are at least 85% of threats traced to a named treating practice and specific control, with untreated threats published as gaps and those at high-risk archetypes carrying an owner and remediation date? In Data, ask to see the threat snapshot for the retrieval store, with its poisoning and extraction paths.

Common pitfalls

• ❌ Writing a threat list without first drawing entities and trust boundaries, which produces a list of worries rather than an analysis anyone can act on.
• ❌ Inventing plausible ATLAS technique identifiers where none fit, which survives review and misleads every downstream practice that trusts the tag.
• ❌ Authoring the library entirely within security, which describes the system security imagines rather than the one practitioners operate.
• ❌ Producing the library and never publishing the treatment gaps, which leaves threats accepted by default without anyone deciding to accept them.
• ❌ Writing snapshots so long that teams do not read them, after which the library is maintained for an audience of nobody.

Level 2, Comprehensive

Model threats per asset at a depth calibrated to its risk tier, refresh models on material change, and keep the Data threat picture current against real adversary behaviour.

Activities

A) Calibrate threat-model depth to asset tier — Publish what each tier receives. Critical: a per-asset threat model, an adversarial overlay walking the relevant ATLAS tactics end to end, an explicit abuse-case set, and a named architect reviewer. High: the archetype model plus documented per-asset deltas and an ATLAS tactic walk. Medium: the archetype model with deltas recorded. Low: the archetype model unchanged. Publishing this is what stops the two failure modes of uniform depth, namely modelling everything shallowly so the Critical assets are under-analysed, or modelling everything deeply so the practice becomes a bottleneck and teams route around it. Set turnaround service levels per tier and report adherence, because a threat-modelling practice that cannot say how long it takes will be worked around by teams under delivery pressure whatever its analytical quality. Per-asset modelling in this domain means walking the specific corpus: where it came from, who can write to it, what transformations touch it, which models consume it, and what an attacker who could influence any upstream source would achieve downstream, which is the poisoning path the archetype model can only gesture at.

B) Refresh on material change rather than on the calendar — Define material change per domain and wire the trigger to the signals that already exist. In this domain a material change includes a new capability or interface on the asset, a change to the data classes it handles, a model or provider change, a change in exposure or the population reached, and a tier increase from the SM rubric. Each triggers a re-model inside the tier's window, being 14 days at Critical and 30 at High. Annual review remains as a floor, not as the mechanism; the calendar is the least informative trigger available and relying on it means an asset that changed materially in month two carries a stale model for ten months. Record every refresh with what changed and what it changed in the model, so that a reviewer can see whether the re-model was substantive or a date stamp. In this domain the material changes that most often invalidate a model are a new source added to a pipeline and a retention or sampling change, both routinely made as data-engineering decisions with no security event attached.

C) Feed external intelligence in and push threat output downstream — Consume external threat intelligence on a defined cadence, drawing from MITRE ATLAS updates, OWASP LLM and Agentic project output, vendor and provider advisories, sector information-sharing bodies and published incident analyses. Assess each item for applicability to the archetypes in the library within 30 days, and record the assessment even when the answer is not applicable, because the record is what demonstrates the cadence is real. Push threat-model output downstream mechanically rather than by expectation: each Critical- and High-tier threat generates a requirement in the SR pack, a test case in the ST battery and a detection candidate in the ML set, tracked to closure. A threat model whose output is not visible in what gets tested and detected has produced a document, and the practice is then measured on documents. Data threat output lands most usefully in the poisoning-resistance and extraction-resistance batteries and in the canary seeding plan, which is where an abstract concern about provenance becomes a detectable condition.

Outcome metrics

Success criteria

• Threat-model depth per tier is published with turnaround service levels, and adherence is at or above 90% per tier.
• Every Critical-tier asset has a per-asset model with an adversarial ATLAS overlay and a named architect reviewer.
• Material change triggers a re-model inside the tier window, with refresh records stating what actually changed.
• External intelligence is assessed for applicability within 30 days, with negative assessments recorded.
• At least 90% of Critical- and High-tier threats generate a tracked requirement, test case and detection candidate.

Assessment questions

1. Is threat-model depth calibrated by asset tier with published turnaround service levels, such that Critical-tier assets receive a per-asset model with an adversarial ATLAS overlay and a named architect reviewer while Low-tier assets inherit the archetype snapshot? Evidence this in the Data domain against the threat snapshot for the retrieval store, with its poisoning and extraction paths.
2. Does material change, including a new capability, a data-class change, a model or provider change, an exposure change or a tier increase, trigger a re-model within 14 days at Critical and 30 at High, with refresh records stating what actually changed rather than only a date? For Data, the assessor should be shown the threat snapshot for the retrieval store, with its poisoning and extraction paths.
3. Is external threat intelligence assessed for applicability within 30 days with negative assessments recorded, and do at least 90% of Critical- and High-tier threats generate a tracked requirement, test case and detection candidate? In Data, ask to see the threat snapshot for the retrieval store, with its poisoning and extraction paths.

Common pitfalls

• ❌ Relying on annual review as the refresh mechanism, which leaves an asset that changed in month two carrying a stale model for ten months.
• ❌ Modelling every asset at the same depth, which either under-analyses the Critical assets or turns the practice into a bottleneck teams route around.
• ❌ Recording refreshes as date stamps without stating what changed, which makes it impossible to tell a substantive re-model from a compliance gesture.
• ❌ Consuming threat intelligence without recording negative assessments, so there is no evidence the cadence is actually operating.
• ❌ Ending the threat model at the document, so nothing in testing or detection changes and the practice is judged on paperwork.

Level 3, Industry-Leading

Drive Data threat modelling from live signal and adversary emulation, validate models against what actually happens, and contribute observed techniques back to ATLAS and the community.

Activities

A) Validate models through adversary emulation — Run adversary emulation against Critical-tier assets at least semi-annually, scoped from the threat model and executed by a team that did not author it. Measure the two directions of error explicitly: threats the model predicted that emulation could not demonstrate, and behaviours emulation found that the model did not anticipate. The second is the more valuable and the more uncomfortable, and a practice that only reports confirmations is not learning. Record for every threat in the library whether it has been demonstrated in this environment, demonstrated elsewhere, or remains theoretical, and report the distribution, since a library where most entries are theoretical is a library built from reading rather than from testing. Feed every unanticipated finding into the archetype library, not only into the asset's model, so that one exercise improves every asset sharing that archetype. Per-asset modelling in this domain means walking the specific corpus: where it came from, who can write to it, what transformations touch it, which models consume it, and what an attacker who could influence any upstream source would achieve downstream, which is the poisoning path the archetype model can only gesture at.

B) Update models from live signal continuously — Close the loop from operations back to the models. Feed detection telemetry, incident root causes, security test failures and near-misses into the library continuously rather than at review time, and treat any observed technique not represented in the library as a defect in the library with an owner and a date. Publish a service level covering the interval from an observed technique to an updated model, targeting 30 days. Track the share of incidents whose root cause was already present in the library as an untreated threat, which is the single most informative metric this practice produces: it separates a failure to foresee, which is hard, from a failure to act on what was already written down, which is not, and the second is far more common than most programmes admit. In this domain the material changes that most often invalidate a model are a new source added to a pipeline and a retention or sampling change, both routinely made as data-engineering decisions with no security event attached.

C) Contribute observed techniques back — Submit techniques observed in this environment to MITRE ATLAS, and contribute to OWASP LLM and Agentic project revisions, sector information-sharing bodies and published incident analyses. This matters more in AI security than in most fields because the public technique catalogue is young, and the gap between what practitioners observe and what is publicly documented is wide, which means a single well-documented submission can be genuinely novel rather than incremental. Target at least four substantive submissions per year, count only those accepted, anonymise and legally review every one, and coordinate disclosure with affected model providers or vendors where a technique touches their systems. Where the L1 activity recorded threats for which no ATLAS technique fit, revisit that list first: it is the highest-yield source of contribution candidates the programme has. Data threat output lands most usefully in the poisoning-resistance and extraction-resistance batteries and in the canary seeding plan, which is where an abstract concern about provenance becomes a detectable condition.

Outcome metrics

Success criteria

• Every Critical-tier asset is exercised by adversary emulation at least annually, by a team that did not author the model.
• Every threat in the library is classified as demonstrated here, demonstrated elsewhere or theoretical, with the theoretical share reported and falling.
• Emulation findings the model did not anticipate are fed back into the archetype library within 30 days, improving every asset sharing that archetype.
• Incidents whose root cause was an already-documented untreated threat are reported openly and falling year over year.
• At least four accepted external technique contributions per year, anonymised, legally reviewed and disclosure-coordinated.

Assessment questions

1. Is every Critical-tier asset exercised by adversary emulation at least annually by a team that did not author the model, with both error directions measured, including behaviours the model failed to anticipate? Evidence this in the Data domain against the threat snapshot for the retrieval store, with its poisoning and extraction paths.
2. Is every library threat classified as demonstrated here, demonstrated elsewhere or theoretical, and is the share of incidents whose root cause was an already-documented untreated threat reported openly and falling? For Data, the assessor should be shown the threat snapshot for the retrieval store, with its poisoning and extraction paths.
3. Does the organisation contribute at least four accepted techniques per year to ATLAS and the community, anonymised, legally reviewed and disclosure-coordinated, drawing first on the threats for which no existing technique fitted? In Data, ask to see the threat snapshot for the retrieval store, with its poisoning and extraction paths.

Common pitfalls

• ❌ Reporting only the threats emulation confirmed, which converts a learning exercise into a validation ritual.
• ❌ Feeding emulation findings into the single asset's model rather than the archetype library, so one exercise improves one asset instead of all of them.
• ❌ Not tracking incidents whose root cause was already in the library untreated, which conceals the difference between failing to foresee and failing to act.
• ❌ Leaving the unmapped-technique list from L1 unrevisited, discarding the best contribution candidates the programme has.
• ❌ Submitting techniques that touch a provider's systems without coordinating disclosure, which damages the relationships the programme depends on.

Security Requirements (SR)

Objective: Publish a base security requirements pack per data archetype, every requirement carrying a testable acceptance criterion, so teams inherit requirements rather than negotiating them.

A program at this level has written down what each data archetype must do, once, in a form a test can check. The pack covers training corpus, fine-tuning dataset, inference input stream, retrieval store, embedding store, prompt and completion log corpus, and evaluation set, and every requirement carries an acceptance criterion specific enough that two reviewers assessing the same asset reach the same verdict. This is the practice that converts threat-model output into something buildable: a threat without a requirement is an observation, and a requirement without a testable criterion is an aspiration. Requirements trace back to the threats they mitigate and forward to the tests that verify them.

Level 1, Foundational

Publish a base security requirements pack per data archetype, every requirement carrying a testable acceptance criterion, so teams inherit requirements rather than negotiating them.

Activities

A) Author the base requirements pack per archetype — Write one base pack per archetype covering training corpus, fine-tuning dataset, inference input stream, retrieval store, embedding store, prompt and completion log corpus, and evaluation set. The pack states what any asset of that archetype must do, drawn from the TA threat library so that every requirement answers a specific threat rather than a general anxiety. In this domain the base requirements centre on corpus classification before ingest, lawful basis recorded per corpus, lineage capture through every transformation, retention right-sized to the longest applicable obligation, erasure propagation to embeddings and derived artifacts, and provenance checks on externally sourced data. Keep the base pack deliberately small, targeting fifteen to twenty-five requirements: a pack of two hundred requirements is negotiated down in practice, and the negotiation happens invisibly at the team's discretion rather than visibly at the programme's. Write each requirement as an obligation on the system rather than on the team, because a requirement phrased as an activity is satisfied by doing the activity badly. Version the packs and publish the change history so teams can see what changed and when.

B) Attach a testable acceptance criterion to every requirement — Give every requirement an acceptance criterion that states how conformance is demonstrated and by what evidence. The criterion must be specific enough that two reviewers assessing the same asset independently reach the same verdict, which is the operational test of whether a criterion is real. Prefer criteria a test can evaluate over criteria a person must judge, and where judgment is unavoidable, state the evidence the judgment rests on. Requirements that cannot be given a testable criterion are either too vague to be useful or belong in guidance rather than requirements, and moving them is better than leaving them to be argued about at every review. Map each criterion to the practice that will verify it, being implementation review, security testing or environment hardening, so that the verification path exists before the requirement is issued.

C) Establish traceability and declare the consumption envelope — Build a requirements evidence map linking each requirement backward to the TA threat it mitigates and forward to the test case, review check or detection that verifies it. Publish the map and treat a requirement with no verifying mechanism as an open gap, because an unverified requirement is a statement of intent that the programme has no way to know is met. Separately, require every archetype to declare a consumption envelope: a spend or volume cap appropriate to its expected workload, and an explicit behaviour when that cap is reached, being degrade, queue or refuse, and never silently continue. Stating the cap as a requirement rather than an operational setting is what makes it testable at per retrieval query and reviewable at design time, and it is the point at which unbounded consumption stops being an operational surprise and becomes a design decision.

Outcome metrics

Consumption metrics — Archetypes whose requirements pack declares a consumption cap and an explicit behaviour at the cap per per archetype requirements pack · Declared consumption envelope per unit of work, stated per archetype per per retrieval query · Consumption-cap requirements carrying a testable acceptance criterion per per consumption requirement

Success criteria

• A base requirements pack exists for every inventoried data archetype, deliberately small at fifteen to twenty-five requirements.
• Every requirement carries an acceptance criterion specific enough that independent reviewers reach the same verdict, verified by double-review calibration at 85% or better.
• A requirements evidence map traces at least 90% of requirements backward to a threat and forward to a verifying mechanism, with gaps published.
• Every archetype declares a consumption envelope with an explicit behaviour at the cap, expressed per unit of work.
• Requirements are phrased as obligations on the system rather than activities for the team.

Assessment questions

1. Does a base security requirements pack exist for every inventoried data archetype, deliberately small, derived from the TA threat library, and phrased as obligations on the system rather than activities for the team?
2. Does every requirement carry an acceptance criterion specific enough that two reviewers independently reach the same verdict, verified by a periodic blind double-review calibration exercise at 85% agreement or better? For Data, the assessor should be shown the corpus requirements pack and its erasure-propagation acceptance criterion.
3. Does a requirements evidence map trace at least 90% of requirements backward to a threat and forward to a verifying mechanism with gaps published, and does every archetype declare a consumption envelope with an explicit, testable behaviour at the cap? In Data, ask to see the corpus requirements pack and its erasure-propagation acceptance criterion.

Common pitfalls

• ❌ Writing packs of two hundred requirements, which get negotiated down invisibly at each team's discretion rather than visibly at the programme's.
• ❌ Phrasing requirements as activities for the team rather than obligations on the system, which are satisfied by performing the activity badly.
• ❌ Accepting acceptance criteria that two reviewers would read differently, which turns every review into a negotiation about what the requirement meant.
• ❌ Issuing requirements with no verifying mechanism, so the programme has no way to know whether they are met.
• ❌ Leaving the consumption cap to operations, which makes unbounded spend an operational surprise rather than a design decision.

Level 2, Comprehensive

Calibrate data requirements packs to asset tier, maintain a per-asset requirements evidence map, and make conformance demonstrable rather than asserted.

Activities

A) Derive requirements scope from tier — Publish the mapping from asset tier to requirements scope: which requirements are mandatory at each tier, which are recommended, and which tier-specific additions apply above the base pack. Derive the scope from the inventory tier automatically rather than letting each team select, because scope selection under delivery pressure reliably produces the minimum defensible set and does so invisibly. Add tier-specific requirements where the tier warrants them, such as an independent conformance assessment and dual approval at Critical. Where a team believes a requirement does not apply, that is a deviation and goes through the deviation process rather than through scope interpretation, which is the distinction that keeps the tier mapping meaningful rather than advisory. Tier scope in this domain keys chiefly on data class and whether the corpus trains or only informs, since a corpus shaping model weights carries a permanence that inference inputs do not and cannot be retracted record by record.

B) Maintain a per-asset requirements evidence map — For every Critical- and High-tier asset, maintain a requirements evidence map recording per requirement its status, the evidence that demonstrates conformance, where that evidence lives, its evidence grade from E0 to E3, and its last verification date. The map is maintained continuously as the asset is built rather than assembled at review, which is what makes design review cheap and audit response fast. Wire evidence collection to the systems that already hold it, being the code repository, the CI pipeline, the test registry and the configuration store, so that the map reflects reality rather than the last time someone updated it. Report the distribution of evidence grades per asset, since an asset whose conformance rests entirely on E0 assertions is in a materially different position from one resting on E2 telemetry, and a single conformance percentage conceals that difference entirely. Evidence for data requirements lives in the catalogue, the lineage records and the retention configuration, and the erasure-propagation evidence is the one most organisations find they cannot produce.

C) Govern deviations and verify consumption caps per tier — Operate a deviation process: any requirement not met is recorded with a rationale, a compensating control, a named approver whose authority scales with tier, and a mandatory expiry date. Review the deviation register monthly, report Critical-tier deviations to the sponsor, and treat a deviation past expiry as an open finding. Deviations are healthy when visible and corrosive when absorbed, and the register is what determines which. Separately, make consumption caps tier-proportionate and verify them rather than declaring them: each Critical- and High-tier asset's cap is expressed relative to per retrieval query, and the at-cap behaviour is exercised at least annually so that what happens at the limit is known before it happens in production. Record the exercise result, because a cap whose behaviour has never been observed is a cap nobody should rely on. Cap exercises in this domain drive a retrieval store to its quota and confirm the declared behaviour, since unbounded retrieval is both a cost and an extraction exposure.

Outcome metrics

Consumption metrics — Critical- and High-tier assets whose consumption cap behaviour has been exercised and recorded per per Critical or High tier asset · Declared consumption envelope against observed consumption, per tier per per retrieval query · Deviations granted against a consumption requirement, with compensating control recorded per per active deviation

Success criteria

• Requirements scope is derived automatically from asset tier, and scope disagreements route through the deviation process rather than through interpretation.
• At least 95% of Critical-tier assets carry a current per-asset requirements evidence map maintained continuously rather than assembled at review.
• Evidence grades are reported per asset, with at least 60% of Critical-tier requirements resting on E2 evidence or better.
• Every deviation carries rationale, compensating control, a tier-appropriate approver and an expiry, with none past expiry.
• Every Critical-tier asset's consumption cap behaviour has been exercised within 12 months and the result recorded.

Assessment questions

1. Is requirements scope derived automatically from the SM L2 asset tier, with disagreements routed through a deviation process rather than resolved as scope interpretation under delivery pressure? Evidence this in the Data domain against the corpus requirements pack and its erasure-propagation acceptance criterion.
2. Does every Critical- and High-tier asset carry a per-asset requirements evidence map maintained continuously from repository, CI and configuration signals, reporting an evidence grade per requirement rather than a single conformance percentage? For Data, the assessor should be shown the corpus requirements pack and its erasure-propagation acceptance criterion.
3. Do all deviations carry rationale, compensating control, tier-appropriate approver and expiry with none past expiry, and has every Critical-tier asset's consumption cap behaviour been exercised and recorded within the last 12 months? In Data, ask to see the corpus requirements pack and its erasure-propagation acceptance criterion.

Common pitfalls

• ❌ Letting teams select their own requirements scope, which reliably produces the minimum defensible set and does so invisibly.
• ❌ Assembling evidence maps at review time, which makes every review expensive and every audit a project.
• ❌ Reporting a single conformance percentage, which conceals whether conformance rests on assertion or on telemetry.
• ❌ Granting deviations without expiry, after which the deviation register becomes the real requirements set.
• ❌ Declaring consumption caps without ever exercising them, so their behaviour at the limit is first learned during an incident.

Level 3, Industry-Leading

Express data requirements as executable policy evaluated continuously in the pipeline, so conformance is enforced at merge rather than assessed at review.

Activities

A) Express requirements as executable policy in the pipeline — Convert every machine-evaluable requirement into a policy check that runs in the delivery pipeline, evaluating the change rather than the documentation. Blocking severity follows asset tier: a Critical-tier breach fails the build, a Low-tier breach warns. Agree the blocking behaviour with engineering before enabling it, because a security check that blocks merges without an agreed escalation path is a check that gets disabled within a month and takes the practice's credibility with it. Publish the list of requirements that are not machine-evaluable and review it quarterly with the aim of shrinking it, since that list is the honest measure of how far enforcement actually reaches. Provide a documented break-glass path with automatic notification and mandatory post-hoc review, because emergencies happen and an unbreakable gate gets bypassed in ways nobody sees. Data requirements express as policy over pipeline definitions and catalogue state: a corpus without a classification, a training job referencing an unregistered dataset, a retention setting below the declared obligation, each checkable before the pipeline runs.

B) Evolve requirements from operational evidence — Close the loop from operations to requirements. Feed incident root causes, security test failures, implementation review findings and monitoring detections into a quarterly requirements review, and treat any failure mode appearing three times without a covering requirement as a mandatory pack change. Publish a service level covering the interval from a recurring failure mode to a new or amended requirement, targeting 60 days including the policy implementation. Retire requirements that operational evidence shows are no longer load-bearing, because a pack that only grows becomes one teams route around; the willingness to remove requirements is what keeps the pack small enough to be respected. Record for each requirement how many times its policy check has actually fired, which distinguishes requirements that are doing work from those that are decoration. The recurring failure modes that drive new data requirements are a source added without reclassification and an erasure that stopped at the store, the second of which surfaces only when someone checks the embeddings.

C) Contribute executable requirement packs back — Contribute at least four substantive artifacts per year, weighted toward executable form: requirement packs with their policy implementations, acceptance criteria that others can adopt directly, and conformance check libraries. Executable contributions are worth substantially more than prose in this practice, because a peer organisation can run a policy check and merely read a requirement, and the adoption difference between the two is large. Work through OWASP projects, OpenSSF AI, sector bodies and the relevant standards communities. Count only accepted or published artifacts, review each legally before release, and prefer contributing the checks that fire most often in this environment, since those are the ones carrying the most operational evidence behind them. Data requirement packs contribute well because erasure propagation and provenance checking are unsolved in common ways, and a working assertion is worth more than another description of the problem.

Outcome metrics

Consumption metrics — Consumption-cap requirements enforced as pipeline policy rather than asserted in a document per per consumption requirement · Changes blocked at merge for breaching a declared consumption envelope per per retrieval query · Consumption-cap policy checks that have never fired per per consumption policy check

Success criteria

• At least 80% of machine-evaluable requirements run as pipeline policy checks, with the non-evaluable remainder listed explicitly and shrinking.
• Critical-tier breaches are blocked at merge, with a documented break-glass path carrying automatic notification and mandatory post-hoc review within 5 business days.
• Recurring failure modes reach a new or amended requirement, including its policy implementation, within 60 days.
• Requirements are retired as well as added, with firing counts reviewed annually and never-firing requirements examined.
• At least four accepted external contributions per year, the majority in executable rather than prose form.

Assessment questions

1. Do at least 80% of machine-evaluable requirements run as pipeline policy checks with tier-proportionate blocking severity agreed with engineering, and is the non-machine-evaluable remainder listed explicitly and reviewed quarterly for reduction? Evidence this in the Data domain against the corpus requirements pack and its erasure-propagation acceptance criterion.
2. Do recurring failure modes from incidents, test failures and review findings reach a new or amended requirement including its policy implementation within 60 days, and are requirements retired as well as added based on firing counts? For Data, the assessor should be shown the corpus requirements pack and its erasure-propagation acceptance criterion.
3. Is there a documented break-glass path with automatic notification and mandatory post-hoc review within 5 business days, and does the organisation contribute at least four accepted artifacts per year, the majority executable rather than prose? In Data, ask to see the corpus requirements pack and its erasure-propagation acceptance criterion.

Common pitfalls

• ❌ Enabling blocking checks without an agreed escalation path, after which the checks are disabled within a month and take the practice's credibility with them.
• ❌ Automating the easy requirements and leaving the non-evaluable list unpublished, so nobody knows how far enforcement actually reaches.
• ❌ Adding requirements without ever retiring any, producing a pack large enough that teams route around it.
• ❌ Tolerating high false-positive rates on policy checks, which trains engineers to bypass rather than to fix.
• ❌ Contributing requirement prose rather than executable checks, which peers read and do not adopt.

Secure Architecture (SA)

Objective: Publish reference architecture patterns and an antipattern catalogue for each data archetype, so the secure design is the default path rather than an achievement.

A program at this level has drawn the secure shape of each data archetype once, so teams adopt rather than invent. The library covers the governed-ingest pattern with classification before storage, the retrieval pattern with per-request authorization and purpose binding, the training pattern isolating regulated data behind an approval gate, and the log pattern with redaction at write and retention right-sized at creation, and it is paired with an antipattern catalogue naming the shapes that recur and what to replace them with. An antipattern catalogue is the more useful half and the more commonly missing one: teams recognise what they are building far more readily than they recognise what they should have built. Every pattern states which threats it mitigates, so adoption is an argument rather than an instruction.

Level 1, Foundational

Publish reference architecture patterns and an antipattern catalogue for each data archetype, so the secure design is the default path rather than an achievement.

Activities

A) Publish reference patterns per archetype — Author one reference pattern per archetype, covering the governed-ingest pattern with classification before storage, the retrieval pattern with per-request authorization and purpose binding, the training pattern isolating regulated data behind an approval gate, and the log pattern with redaction at write and retention right-sized at creation. Each pattern states the components and their boundaries, the controls at each boundary, the threats it mitigates by identifier from the TA library, the requirements it satisfies from the SR pack, and the conditions under which it does not apply. Keep each pattern to a length a team reads before designing rather than after, targeting three pages with a diagram. Validate each with an engineer who has built that archetype, because a pattern authored entirely within security describes a system that is secure and unbuildable, and teams respond to that by ignoring it rather than by arguing. Publish patterns in the repository where the work happens rather than in a separate architecture space.

B) Publish the antipattern catalogue — Author an antipattern catalogue naming the insecure shapes that actually recur in this domain, being the retrieval store with a single service identity reading everything, the training pipeline that ingests first and classifies later, and the prompt log written unredacted and replicated to an observability vendor. For each antipattern, state why it is attractive, since it is always attractive for a reason and a catalogue that does not acknowledge that reads as naive; what specifically goes wrong; and which reference pattern replaces it. The antipattern half of the library is the more useful and the more commonly omitted, because teams recognise the thing they are building far more readily than they recognise the thing they should have built instead. Draw the catalogue from real designs seen in review rather than from literature, and add to it whenever review encounters a new recurring shape.

C) Establish the untrusted-content boundary and a design-time self-check — Document explicitly where untrusted content enters each archetype and how it is kept separate from instruction, control and identity. This is the single most consequential architectural decision in AI systems and the one most often left implicit, because the failure it prevents, content becoming instruction, does not look like a boundary violation in any conventional sense. State the pattern for isolation, for provenance marking, and for what the system must never do with content it did not author. Publish a short design-time self-check, at most ten questions, that a team can answer before requesting review, which both raises the quality of what arrives at review and tells the team what the review will ask, removing the asymmetry that makes design review feel arbitrary.

Outcome metrics

Oversight metrics — Share of designs where the reviewer recorded a required change before approval (detects: rubber-stamping at design review, where approval is recorded without the review having altered anything, which is the default state of a review with no capacity to say no) · Designs approved with a human gate declared on a training run against regulated data but no stated refusal authority (detects: a gate that exists structurally while the person holding it cannot actually refuse, which satisfies the control on paper and not in fact)

Success criteria

• A reference pattern exists for at least 85% of inventoried data archetypes, each validated with an engineer who has built that archetype.
• Every pattern states the threats it mitigates by TA identifier and the requirements it satisfies.
• An antipattern catalogue of at least eight entries is published, each drawn from a design actually seen in review and naming its replacement pattern.
• Every archetype ingesting content the organisation does not author has a documented untrusted-content boundary.
• A design-time self-check of at most ten questions is published and completed by at least 70% of designs arriving at review.

Assessment questions

1. Does a reference pattern exist for at least 85% of inventoried data archetypes, each stating components, boundaries, controls, the threats it mitigates by TA identifier, the requirements it satisfies and where it does not apply, validated with an engineer who has built that archetype?
2. Is there an antipattern catalogue of at least eight entries drawn from designs actually seen in review, each stating why the shape is attractive, what goes wrong, and which reference pattern replaces it? For Data, the assessor should be shown the governed-ingest pattern and the retrieval authorization pattern.
3. Does every archetype ingesting content the organisation does not author have a documented untrusted-content boundary, and is a design-time self-check published and completed by at least 70% of designs arriving at review? In Data, ask to see the governed-ingest pattern and the retrieval authorization pattern.

Common pitfalls

• ❌ Authoring patterns entirely within security, producing designs that are secure and unbuildable, which teams ignore rather than contest.
• ❌ Publishing reference patterns without an antipattern catalogue, omitting the half teams actually recognise themselves in.
• ❌ Writing an antipattern catalogue that does not acknowledge why each shape is attractive, which reads as naive and is dismissed.
• ❌ Leaving the untrusted-content boundary implicit, which is the most consequential architectural decision in AI systems and the least conventional-looking.
• ❌ Publishing patterns in a separate architecture space rather than where the work happens, after which they are found only by people already looking.

Level 2, Comprehensive

Calibrate the data reference-pattern set by risk tier and scale the autonomy and oversight controls in each pattern with the asset's capability tier.

Activities

A) Declare tier applicability across the pattern library — Annotate every pattern in the library with the tiers it applies to and the tiers at which it is mandatory rather than recommended. Publish the mapping so that a team's pattern set is derived from the asset's tier rather than assembled by preference. At Critical tier, pattern adherence is mandatory and any deviation requires a full-lane design review with a named architect; at Low tier the pattern is recommended and deviation needs only a recorded note. Stating this removes the ambiguity that otherwise resolves in favour of whatever is quickest to build. Re-derive the applicable pattern set automatically when an asset's tier changes, because a Medium-to-Critical promotion that does not change the pattern obligations has changed nothing that matters.

B) Scale autonomy and oversight controls with capability tier — Within each pattern, define how its controls tighten as the asset's capability tier rises. Bound blast radius more tightly, shorten the autonomy budget between human checkpoints, and require a human gate on a training run against regulated data at higher capability tiers where lower tiers need only logging. Specify each gate numerically rather than in principle: who holds the authority to refuse, what information they receive, and how long they have. A gate specified as a principle is compressed under delivery pressure until scrutiny is impossible, and because nothing in the specification objects, the erosion is invisible. Publish a reviewer-load ceiling per tier above which review capacity must be added rather than reviewers exhorted, since a gate held by an overloaded reviewer is a gate in name only.

C) Run the incident-to-pattern learning loop — Establish a standing review that takes every incident reaching a root cause and asks two questions: was there a pattern that should have prevented this, and if so why was it not followed, or if not what pattern should now exist. Record the answer for every incident, including the answer that no pattern change is warranted, because the discipline of recording the negative case is what keeps the loop honest rather than performative. Where a pattern existed and was not followed, the finding belongs to the review and adoption practices rather than to the library. Where no pattern existed, author one and publish it within 60 days. Track the share of incidents attributable to a pattern that existed and was not followed, which separates a library problem from an adoption problem and is the single most useful number this practice produces.

Outcome metrics

Oversight metrics — Reviewer override rate on pattern deviation requests, paired with sustained-override rate (detects: deviation approval becoming automatic under delivery pressure, where pairing with sustained rate counters the gaming that follows once the rate becomes a target) · Design reviews per reviewer per week against the published ceiling (detects: reviewer overload, where architectural scrutiny cannot survive the volume regardless of reviewer skill, which is a capacity defect rather than a personnel one) · Critical-tier designs where the human gate on a training run against regulated data specifies authority, information and a time budget numerically (detects: oversight specified as a principle rather than a number, which is compressed under delivery pressure until scrutiny is impossible and nothing objects)

Success criteria

• Every pattern declares its tier applicability and whether it is mandatory or recommended at each tier.
• Controls within each pattern scale with capability tier, with every human gate specified numerically in authority, information and time.
• At least 95% of Critical-tier assets either follow a reference pattern or carry a full-lane deviation review, with no third category.
• Every incident reaching root cause is checked against the library, with negative findings recorded.
• A per-tier reviewer-load ceiling is published and monitored, and breaching it adds capacity rather than exhortation.

Assessment questions

1. Does every pattern declare its tier applicability and mandatory-versus-recommended status, with the applicable set re-derived automatically when an asset's tier changes? Evidence this in the Data domain against the governed-ingest pattern and the retrieval authorization pattern.
2. Do the controls inside each pattern tighten as capability rises, with the human gate on a training run against regulated data specified numerically in authority, information and time budget rather than as a principle, and is a per-tier reviewer-load ceiling published?
3. Is every incident reaching a root cause checked against the library for a preventing pattern with negative findings recorded, and is the share attributable to an existing pattern not followed tracked separately from gaps in the library? In Data, ask to see the governed-ingest pattern and the retrieval authorization pattern.

Common pitfalls

• ❌ Publishing patterns without tier applicability, which leaves adoption scope to resolve in favour of whatever is quickest to build.
• ❌ Specifying human gates as principles rather than numbers, so they compress invisibly under delivery pressure until scrutiny is impossible.
• ❌ Recording only the incidents that produced a pattern change, which makes the learning loop performative rather than honest.
• ❌ Conflating a pattern that was missing with a pattern that was ignored, which sends improvement effort to the wrong practice.
• ❌ Responding to reviewer overload with exhortation rather than capacity, which converts the gate into a formality.

Level 3, Industry-Leading

Publish data patterns as executable artifacts adopted by default, verify conformance continuously, and contribute the pattern library back to the field.

Activities

A) Ship patterns as executable artifacts — Convert each reference pattern into something a team runs rather than reads: infrastructure modules, project scaffolds, libraries or configuration bundles that instantiate the pattern correctly by default. This is the level's central move and the one that changes adoption economics, because a pattern that is faster to adopt than to circumvent is adopted without enforcement, and no amount of mandate achieves what convenience achieves. Version the artifacts, publish their change history, and provide a migration path when a pattern changes so that existing adopters are not stranded on an old version, which is the most common reason executable pattern libraries decay. Measure adoption of the artifact rather than conformance to the document, since the artifact is now the pattern. In this domain the executable artifact is usually a pipeline template that performs classification at ingest, emits lineage at every step and configures retention from the declared obligation, so governed ingest is the default path rather than an added step.

B) Verify conformance continuously against the running system — Verify that what is running still matches the pattern that was approved, continuously rather than at review. Draw the signals from configuration state, runtime telemetry and the deployment pipeline, and evaluate each pattern's invariants as checks that can fail. Architecture drift is the defining failure of this practice at scale: designs are reviewed once and systems change continuously, so a library verified only at design time describes the past. Publish a service level covering detection of a Critical-tier conformance breach within 5 business days, and route breaches to the issue-management practice with tier-appropriate severity rather than to a report nobody owns. Where an invariant cannot be checked automatically, list it explicitly so the verification's real reach is known. Conformance verification reads catalogue state, pipeline definitions and retention configuration against the pattern, which catches the store that gained a source and never gained a reclassification.

C) Contribute the executable pattern library back — Contribute at least four substantive artifacts per year, weighted toward executable form: reference implementations, infrastructure modules, scaffolds and the conformance checks that verify them. Executable contributions carry disproportionate value in this practice, because a peer organisation can adopt a module and can only read a diagram, and the adoption gap between the two is the difference between influencing the field and describing it. Work through OpenSSF AI, CNCF where infrastructure applies, OWASP projects and sector bodies. Count only accepted artifacts, review each legally before release, and prefer contributing the patterns with the highest internal adoption, since those carry the most operational evidence and are the most likely to survive contact with another organisation's environment. Data patterns contribute well where erasure propagation and lineage emission are concerned, because these are commonly needed and rarely solved well.

Outcome metrics

Oversight metrics — Pattern deviations later implicated in an incident, as a share of deviations approved (detects: deviation approval that is procedurally correct and substantively wrong, which only post-hoc linkage to incidents can reveal) · Automated pattern conformance checks overridden by a human, with the override sustained (detects: automation override becoming the routine path around a check that is inconvenient rather than wrong, which is how enforced patterns quietly become advisory) · Reviewer disagreement rate on a blind double-review sample of Critical-tier designs (detects: reviewer drift, where the same design would be approved by one architect and refused by another, making the gate a lottery rather than a control)

Success criteria

• At least 80% of reference patterns ship as executable artifacts with versioning and a migration path on breaking change.
• At least 75% of new Critical- and High-tier assets adopt the executable artifact at creation.
• At least 85% of pattern invariants are verified continuously against the running system, with unverifiable invariants listed explicitly.
• Critical-tier conformance breaches are detected within 5 business days and routed to issue management with tier-appropriate severity.
• At least four accepted external contributions per year, the majority executable rather than documentary.

Assessment questions

1. Do at least 80% of reference patterns ship as executable artifacts with versioning and a migration path, and do at least 75% of new Critical- and High-tier assets adopt the artifact at creation rather than adopting the document? Evidence this in the Data domain against the governed-ingest pattern and the retrieval authorization pattern.
2. Are at least 85% of pattern invariants verified continuously against the running system with unverifiable invariants listed explicitly, and are Critical-tier breaches detected within 5 business days and routed to issue management? For Data, the assessor should be shown the governed-ingest pattern and the retrieval authorization pattern.
3. Does the organisation contribute at least four accepted artifacts per year in majority-executable form, prioritising the patterns with the highest internal adoption and therefore the most operational evidence? In Data, ask to see the governed-ingest pattern and the retrieval authorization pattern.

Common pitfalls

• ❌ Shipping executable patterns without a migration path, which strands existing adopters on old versions and is the usual cause of library decay.
• ❌ Verifying conformance only at design time, which describes the past while the running system changes continuously.
• ❌ Measuring conformance to the document after the artifact has become the real pattern, which measures the wrong thing.
• ❌ Leaving unverifiable invariants unlisted, so the verification's actual reach is unknown and overstated.
• ❌ Contributing diagrams rather than modules, which peers read and do not adopt.

Business Function: Verification

Design Review (DR)

Objective: Establish a two-lane design review for data AI changes with a published checklist and service level, so review is predictable enough that teams route through it.

A program at this level reviews data AI designs against a published checklist, in one of two lanes, with a stated turnaround. The fast lane clears low-risk changes against a standard checklist in days; the full lane handles novel or high-risk designs with a named architect. Publishing both the checklist and the service level is what makes the practice work: a review whose criteria and duration are unknown is a review teams plan around rather than through, and the routing-around is invisible until an incident reveals it.

Level 1, Foundational

Establish a two-lane design review for data AI changes with a published checklist and service level, so review is predictable enough that teams route through it.

Activities

A) Establish two review lanes with published service levels — Define a fast lane and a full lane, and publish the routing rule and the service level for each. The fast lane handles designs adopting a sanctioned reference pattern without material deviation, cleared against a standard checklist within 5 business days. The full lane handles novel architectures, pattern deviations and high-risk designs, with a named architect reviewer and a decision within 15 business days. Publishing both numbers is the practice's single most important act: a review of unknown duration is a review teams plan around, and they do so quietly. Measure and report adherence per lane, and treat a persistent breach as a capacity problem to be resourced rather than a communication problem to be explained.

B) Publish the checklist and make decisions written — Publish the review checklist so teams can self-assess before submitting. In this domain the checklist covers classification recorded before ingest, lawful basis stated, lineage capture through every transformation, retention right-sized to the longest applicable obligation, erasure propagation path to embeddings and derived artifacts, per-request authorization on retrieval, and redaction at log write. Every review produces a written decision of approve, approve with conditions, or refuse, naming the reviewer and stating the rationale. Conditions carry an owner and a date and are tracked to closure, because an approval with untracked conditions is an approval, and the conditions are the part that mattered. Record the checklist items that most often fail, and feed that back to the education and architecture practices as the clearest available evidence of where design understanding is weak.

C) Define review triggers and detect what bypassed review — Define what triggers a review rather than relying on teams to request one. In this domain the triggers are a new data class entering a corpus, a new consuming model, a cross-border flow change, or a retention or sampling change, plus any new asset entering the inventory. Wire the triggers to the signals that already exist, being inventory changes, deployment events and change records, so that a review obligation is raised by the system rather than remembered by a person. Then detect bypass: reconcile assets reaching production against review decisions, and treat a production asset with no review record as a finding with an owner. Bypass detection is what converts design review from a service teams may use into a gate whose absence is visible, and without it the practice measures only the designs it happened to see.

Outcome metrics

Success criteria

• Two lanes operate with published routing rules and service levels of 5 and 15 business days, with adherence at or above 90%.
• The review checklist is published and teams self-assess against it before submitting.
• Every decision is written with a named reviewer and rationale, and every condition carries an owner, a date and closure tracking.
• Review triggers fire from inventory, deployment and change signals rather than relying on team initiative.
• Bypass reconciliation runs monthly and every unreviewed production asset is raised as a finding.

Assessment questions

1. Do two published review lanes operate with stated routing rules and service levels of 5 and 15 business days, with adherence measured and reported per lane and persistent breaches treated as a capacity problem? Evidence this in the Data domain against a full-lane decision on a cross-border corpus flow.
2. Is the review checklist published so teams self-assess before submitting, covering classification recorded before ingest, lawful basis stated, lineage capture through every transformation, retention righ, and does every review produce a written decision with named reviewer, rationale and tracked conditions?
3. Do review triggers fire from inventory, deployment and change signals rather than team initiative, and does monthly bypass reconciliation raise every unreviewed production asset as a finding with an owner? In Data, ask to see a full-lane decision on a cross-border corpus flow.

Common pitfalls

• ❌ Running review without a published service level, which teams respond to by planning around review rather than through it, invisibly.
• ❌ Approving with conditions that are never tracked, which converts a conditional approval into an unconditional one.
• ❌ Relying on teams to request review, which means the practice only ever sees the designs of teams already inclined to ask.
• ❌ Never reconciling production against review records, so the practice measures the designs it saw and not the ones it missed.
• ❌ Treating persistent service-level breaches as a communication problem rather than as evidence that review capacity is short.

Level 2, Comprehensive

Calibrate data design review depth and reviewer authority to asset tier, and make review decisions traceable to the threats and requirements they turned on.

Activities

A) Derive review depth and reviewer authority from tier — Publish what each tier receives: at Critical, a full-lane review with a named architect, mandatory threat-model and requirements-evidence review, and a decision recorded where the sponsor can see it; at High, full lane where a deviation exists and fast lane otherwise; at Medium, fast lane; at Low, a checklist pass with sampling. Derive the lane from the inventory tier automatically rather than from reviewer judgment at intake, because intake routing under queue pressure drifts toward the fast lane and does so without anyone deciding to lower the bar. Publish a reviewer-load ceiling per tier, and when it is breached add reviewers rather than shortening reviews, since the alternative is a gate that approves at the rate the queue demands. In this domain the designs that genuinely need architect judgment are new corpus types and cross-border flows, while an additional consumer of an already-governed store usually does not.

B) Make decisions traceable to threats and requirements — Require every full-lane decision to cite the specific threat identifiers from the TA library and requirement identifiers from the SR pack that it assessed, and to state which ones drove each condition or refusal. This converts review from personal judgment into reconstructable reasoning, which matters for three reasons: a second reviewer can reach the same conclusion, an auditor can follow it, and the practice can tell which threats and requirements are actually doing work in review. Report the threats and requirements most frequently driving conditions, since that is direct evidence of where designs are systematically weak and belongs in the architecture and education practices rather than only in the review log. Automated design criteria read the catalogue entry, declared lawful basis, retention setting and lineage plan, which covers the conformance questions and leaves provenance judgment to a person.

C) Govern deviations and make Critical decisions visible — Operate deviation approval as a distinct decision with its own record: what pattern or requirement is not met, why, what compensating control applies, who approved it at an authority proportionate to tier, and when it expires. Review the deviation register monthly and report Critical-tier deviations to the sponsor alongside the approvals, because a review practice that reports only what it approved and not what it excused presents a misleading picture of the estate. Track deviations later implicated in incidents as a distinct measure, which is the only honest feedback on whether deviation judgment is sound rather than merely procedurally correct. Forward tracing in this domain most often implicates approvals that accepted a retention period without checking the longest applicable obligation.

Outcome metrics

Success criteria

• Review lane and reviewer seniority are derived from asset tier, with at least 95% of routing tier-derived rather than judged at intake.
• Every Critical-tier design receives a full-lane review with a named architect and a sponsor-visible decision.
• At least 90% of full-lane decisions cite the specific threat and requirement identifiers that drove each condition or refusal.
• Deviation approvals carry a compensating control, tier-appropriate authority and an expiry, with none past expiry.
• A per-tier reviewer-load ceiling is published and breaching it adds capacity rather than shortening reviews.

Assessment questions

1. Is review lane and reviewer seniority derived automatically from asset tier rather than judged at intake under queue pressure, with at least 95% tier-derived routing and a published per-tier reviewer-load ceiling? Evidence this in the Data domain against a full-lane decision on a cross-border corpus flow.
2. Do at least 90% of full-lane decisions cite the specific TA threat identifiers and SR requirement identifiers that drove each condition or refusal, making the reasoning reconstructable by a second reviewer and an auditor? For Data, the assessor should be shown a full-lane decision on a cross-border corpus flow.
3. Do deviation approvals carry a compensating control, tier-appropriate approval authority and an expiry with none past expiry, and are Critical-tier deviations reported to the sponsor alongside approvals? In Data, ask to see a full-lane decision on a cross-border corpus flow.

Common pitfalls

• ❌ Routing at intake by reviewer judgment, which under queue pressure drifts toward the fast lane without anyone deciding to lower the bar.
• ❌ Recording decisions without citing what drove them, which makes review personal judgment and leaves a second reviewer unable to reproduce it.
• ❌ Reporting approvals to the sponsor without reporting deviations, which presents a systematically flattering picture of the estate.
• ❌ Responding to reviewer overload by shortening reviews rather than adding reviewers, producing a gate that approves at queue rate.
• ❌ Never tracking whether deviations were later implicated in incidents, which leaves deviation judgment unevaluated.

Level 3, Industry-Leading

Shift data design review left into the pipeline, reserve human review for genuine novelty, and measure whether review decisions were right rather than only timely.

Activities

A) Automate the machine-checkable criteria — Convert every review criterion that can be evaluated from an artifact into an automated check running against the design representation, whether that is infrastructure-as-code, a configuration manifest, a service definition or a system card. A design meeting all automated criteria and adopting a sanctioned pattern clears without human review at Medium and Low tier, and arrives at Critical and High review pre-verified so the reviewer spends attention on what is novel rather than on confirming what is conformant. Publish the list of criteria that cannot be automated and review it quarterly for reduction, because that list is the honest boundary of the automation. Keep a human veto and sampling at every tier, since a wholly automated gate encodes yesterday's criteria permanently and nothing in the system notices. In this domain the designs that genuinely need architect judgment are new corpus types and cross-border flows, while an additional consumer of an already-governed store usually does not. Novelty in this domain means a new corpus type or an unfamiliar provenance situation, not another consumer of a governed store.

B) Measure decision quality, not throughput — Trace review decisions forward. For every approval, record whether the asset subsequently generated an incident, an implementation-review finding or a security-test failure attributable to something the review should have caught. Report false-negative rate, being approvals later implicated, and investigate false-positive pressure, being conditions imposed that evidence later showed unnecessary, since a gate that over-conditions trains teams to argue rather than comply. Throughput and service-level adherence remain as health metrics but are explicitly not the practice's outcome measure, because a review function optimising for turnaround will achieve it by approving, and the metric will look excellent throughout. Publish decision-quality measures to the sponsor alongside throughput. Automated design criteria read the catalogue entry, declared lawful basis, retention setting and lineage plan, which covers the conformance questions and leaves provenance judgment to a person.

C) Calibrate reviewers and concentrate attention on novelty — Run blind double-review on a sample of Critical-tier designs at least quarterly and report the disagreement rate, because a gate whose outcome depends on which architect drew the review is a lottery rather than a control, and disagreement is invisible without deliberately measuring it. Use the disagreement cases to refine criteria rather than to rank reviewers. Reserve human review explicitly for novelty: designs using an unestablished pattern, a new archetype, a new model capability class or an unfamiliar threat surface, and route everything else to automated verification with sampling. Track the share of reviewer hours spent on novel designs, which should rise steadily and is the clearest measure that the automation is doing its job. Forward tracing in this domain most often implicates approvals that accepted a retention period without checking the longest applicable obligation.

Outcome metrics

Success criteria

• At least 75% of machine-checkable criteria run automatically, with the non-automatable remainder listed and shrinking.
• Medium- and Low-tier conformant designs clear without human review while sampling continues, and a human veto is retained at every tier.
• Decision quality is measured by forward tracing to incidents and findings, and reported to the sponsor alongside throughput.
• Blind double-review runs quarterly on Critical-tier designs with a disagreement rate at or below 15%, used to refine criteria rather than rank reviewers.
• At least 60% of reviewer hours are spent on genuinely novel designs, and rising.

Assessment questions

1. Are at least 75% of machine-checkable review criteria evaluated automatically with the non-automatable remainder listed explicitly, and do conformant Medium- and Low-tier designs clear without human review while sampling and a human veto are retained? Evidence this in the Data domain against a full-lane decision on a cross-border corpus flow.
2. Is decision quality measured by tracing approvals forward to incidents, implementation findings and test failures, reported to the sponsor alongside throughput and explicitly rather than throughput as the outcome measure? For Data, the assessor should be shown a full-lane decision on a cross-border corpus flow.
3. Does blind double-review run at least quarterly on Critical-tier designs with disagreement at or below 15% used to refine criteria rather than rank reviewers, and is at least 60% of reviewer time spent on genuinely novel designs? In Data, ask to see a full-lane decision on a cross-border corpus flow.

Common pitfalls

• ❌ Optimising the practice for turnaround, which a review function achieves by approving while the metric looks excellent throughout.
• ❌ Removing the human veto and sampling once automation is in place, which encodes yesterday's criteria permanently with nothing to notice.
• ❌ Never measuring reviewer disagreement, leaving a gate whose outcome depends on which architect drew the review.
• ❌ Using double-review disagreement to rank reviewers, which produces defensive convergence rather than better criteria.
• ❌ Leaving the non-automatable criteria list unpublished, so the boundary of the automation is unknown and assumed wider than it is.

Implementation Review (IR)

Objective: Review what was actually built against what was designed and required for data AI assets, at go-live, with findings tracked to closure.

A program at this level checks the built system rather than the intended one. Every data AI asset gets an implementation review before go-live, working from a published checklist that examines classification applied at ingest rather than asserted, lineage actually emitted by each pipeline step, retention configured to the declared period, erasure propagation reaching embeddings and derived artifacts, per-request authorization enforced on retrieval, and redaction applied at log write rather than at read. The distinction from design review is the entire point: design review assesses what a team intends, implementation review assesses what exists, and the gap between the two is where most AI security failures actually live. Findings carry an owner, a severity and a date, and go-live is conditional on the blocking ones being closed.

Level 1, Foundational

Review what was actually built against what was designed and required for data AI assets, at go-live, with findings tracked to closure.

Activities

A) Publish the implementation review checklist and run it at go-live — Publish a checklist per archetype that examines the built artifact rather than its documentation. In this domain the review looks at classification applied at ingest rather than asserted, lineage actually emitted by each pipeline step, retention configured to the declared period, erasure propagation reaching embeddings and derived artifacts, per-request authorization enforced on retrieval, and redaction applied at log write rather than at read. The discipline that makes this practice worth running is verifying presence and effect rather than presence alone: a logging configuration that exists but emits nothing, an allowlist enforced in audit mode, a kill switch wired to nothing, all pass a documentation check and fail a real one. Run the review before go-live for every asset, and make go-live conditional on blocking findings being closed. Where the review cannot verify a control because the environment does not expose it, record that explicitly as unverifiable rather than as passed, since silence is not a pass.

B) Compare the built system against the approved design — Reconcile what was built against the design review decision and the requirements pack: every requirement in scope for the asset's archetype gets a verified, not-verified or unverifiable disposition, and every condition attached to the design approval gets confirmed as met. Record the design-to-implementation gap as a measure in its own right, because a gap that is never measured is assumed to be zero and never is. Where implementation diverges from the approved design, treat it as a finding against the change rather than as a re-opening of the design decision, and route material divergence back to design review rather than absorbing it in the implementation record.

C) Track findings to closure with severity and ownership — Give every finding a severity, an owner, a remediation date and a disposition of remediated, accepted or deferred. An accepted finding requires a named acceptor with authority proportionate to the severity, because a finding that is neither fixed nor formally accepted leaves residual risk that nobody owns, which is the most common way implementation review findings evaporate. Feed all findings into the issue-management backlog rather than maintaining a separate review register, so that AI findings age and escalate on the same rails as everything else. Report finding density by archetype and by checklist item quarterly, which tells the architecture and education practices where implementations systematically diverge rather than where reviewers happened to look.

Outcome metrics

Oversight metrics — Share of implementation reviews recording at least one finding before sign-off (detects: rubber-stamping at implementation review, where sign-off is recorded without the review having examined anything, which is the default outcome when review is a release-checklist item) · Findings accepted as risk at sign-off rather than remediated, with a named acceptor (detects: silent acceptance, where a finding is noted and neither fixed nor formally accepted, so nobody owns the residual risk)

Success criteria

• Every AI asset reaching production has an implementation review against the published per-archetype checklist.
• Controls are verified as present and effective, with controls the environment cannot expose recorded as unverifiable rather than passed.
• Every in-scope requirement receives a verified, not-verified or unverifiable disposition at review.
• Blocking findings are closed before go-live, and every accepted finding has a named acceptor with proportionate authority.
• Findings flow into the shared issue backlog rather than a separate review register.

Assessment questions

1. Does every data AI asset receive an implementation review before go-live against a published per-archetype checklist that verifies controls are present and effective rather than documented, with controls the environment cannot expose recorded as unverifiable rather than passed?
2. Is the built system reconciled against the approved design and requirements pack, with every in-scope requirement given a verified, not-verified or unverifiable disposition and the design-to-implementation gap measured as a metric in its own right? For Data, the assessor should be shown the go-live review record showing classification applied at ingest rather than asserted.
3. Do all findings carry an owner, severity, date and disposition in the shared issue backlog, with blocking findings closed before go-live and every accepted finding carrying a named acceptor with proportionate authority? In Data, ask to see the go-live review record showing classification applied at ingest rather than asserted.

Common pitfalls

• ❌ Verifying that a control is configured rather than that it works, which passes logging that emits nothing and allowlists running in audit mode.
• ❌ Recording a control the environment cannot expose as passed, which converts an observability gap into a false assurance.
• ❌ Keeping findings in a separate review register, where they age without the escalation the shared backlog would give them.
• ❌ Allowing findings to be neither fixed nor formally accepted, which leaves residual risk owned by nobody.
• ❌ Treating implementation divergence as a chance to re-open the design decision rather than as a finding against the change.

Level 2, Comprehensive

Calibrate data implementation review cadence to asset tier and detect configuration drift continuously, so controls verified at go-live stay verified.

Activities

A) Set re-review cadence by tier and material change — Publish the re-review cadence per tier: Critical at go-live, semi-annually and on every material change; High at go-live, annually and on material change; Medium at go-live and annually; Low at go-live. Define material change concretely for this domain, being a new source added to a pipeline without reclassification, retention shortened for cost, redaction disabled for debugging and left off, and a retrieval store granted a broader service identity, and wire the triggers to change records and deployment events so the obligation is raised by the system rather than remembered. The calendar is the weakest trigger available and should be the floor rather than the mechanism, because an asset that changed materially in month two carries a verified-once record for ten months and the record looks identical to an accurate one.

B) Detect configuration drift continuously on high-tier assets — Stand up continuous drift detection for Critical- and High-tier assets covering the controls whose silent disablement would matter most. In this domain the drift that recurs is a new source added to a pipeline without reclassification, retention shortened for cost, redaction disabled for debugging and left off, and a retrieval store granted a broader service identity. Detect it from configuration state, infrastructure-as-code divergence, policy enforcement mode and runtime telemetry, and alert on change rather than on schedule. Drift detection is what converts implementation review from a point-in-time assertion into a continuing one, and it is the single highest-value activity at this level, because implementations rarely fail review, they pass and then change. Publish a detection service level of 5 business days at Critical tier and route detections into the issue backlog with tier-appropriate severity rather than into a report.

C) Apply tier-proportionate service levels and deferral authority — Set remediation service levels by finding severity and asset tier, tighter at Critical, and report adherence rather than only backlog size. Require deferral authority to rise with tier and severity, so a deferred Critical-tier finding needs an executive acceptance with an expiry and a quarterly review, while a Low-tier one needs the practice lead. Report deferred Critical-tier findings to the sponsor alongside remediated ones, because a practice that reports only closures presents a picture that flatters it. Track findings that were deferred and later implicated in an incident as a distinct measure, since that is the only feedback available on whether deferral judgment is sound rather than merely authorised.

Outcome metrics

Oversight metrics — Reviewer finding rate per review, compared across reviewers on comparable assets (detects: reviewer drift, where the same implementation would pass with one reviewer and fail with another, making sign-off depend on assignment rather than on the code) · Implementation reviews per reviewer per week against the published ceiling (detects: reviewer overload, where depth falls as volume rises and the review continues to be recorded as complete) · Critical-tier findings deferred past their remediation window with a recorded acceptance authority (detects: deferral becoming the default disposition under release pressure, which converts a blocking finding into an advisory one without a decision)

Success criteria

• Re-review cadence is published per tier and driven by material-change triggers wired to change and deployment events.
• At least 90% of Critical-tier assets are under continuous drift detection, with detection within 5 business days.
• Findings carry tier and severity service levels with adherence at or above 90% per tier.
• Deferral authority rises with tier and severity, and deferred Critical findings carry executive acceptance with expiry.
• Deferred findings later implicated in incidents are tracked as a distinct measure of deferral judgment.

Assessment questions

1. Is re-review cadence published per tier and driven by material-change triggers wired to change records and deployment events, with the calendar serving as a floor rather than the mechanism? Evidence this in the Data domain against the go-live review record showing classification applied at ingest rather than asserted.
2. Are at least 90% of Critical-tier assets under continuous configuration drift detection with a 5 business day detection service level, and are detections routed into the shared backlog with tier-appropriate severity? For Data, the assessor should be shown the go-live review record showing classification applied at ingest rather than asserted.
3. Do remediation service levels vary by tier and severity with adherence at or above 90% per tier, does deferral authority rise with tier, and are deferred findings later implicated in incidents tracked as a distinct measure? In Data, ask to see the go-live review record showing classification applied at ingest rather than asserted.

Common pitfalls

• ❌ Relying on annual re-review, which leaves an asset that changed in month two carrying a record indistinguishable from an accurate one for ten months.
• ❌ Detecting drift on a schedule rather than on change, which reintroduces exactly the point-in-time weakness the activity exists to remove.
• ❌ Routing drift detections into a report rather than the backlog, where they age without escalation and without an owner.
• ❌ Reporting closures to the sponsor without deferrals, which systematically flatters the practice.
• ❌ Authorising deferral without ever checking whether deferred findings later caused incidents, leaving the judgment unevaluated.

Level 3, Industry-Leading

Verify data implementation continuously in the pipeline and at runtime, and measure whether review judgment was right rather than only whether review happened.

Activities

A) Assert controls continuously in pipeline and runtime — Express each verifiable control as an assertion that runs both in the delivery pipeline against the change and against the running system on a continuous schedule. Pipeline assertions prevent a regression from merging; runtime assertions catch what was changed outside the pipeline, and both are needed because the second is where drift actually originates. Give each assertion a tier-proportionate severity, block merges on Critical-tier failures, and route runtime failures to the backlog automatically. Publish the list of controls that cannot be asserted and review it quarterly with a reduction target, because that list is the true boundary of the verification and is otherwise assumed to be empty. Assertions in this domain read catalogue classification, pipeline lineage emission, retention configuration and the erasure propagation path, which is the set that conventional data-quality checks omit entirely.

B) Concentrate human review on the unassertable — Reserve human implementation review for the controls automation cannot assert and for the judgment calls automation cannot make, such as whether a control's implementation is adequate to its intent rather than merely present. Retain sampling on automatically verified assets so that the assertions themselves are checked, because an assertion that has silently stopped evaluating reports success indistinguishably from one that is passing. Track the share of reviewer hours spent on unassertable controls, which should rise as automation extends, and use it as the measure that the automation is releasing capacity rather than merely adding a second process alongside the first. What automation cannot assert here is whether a corpus's provenance is genuinely establishable, which is a judgment about evidence quality rather than a property of configuration.

C) Measure review judgment by forward tracing — Trace every sign-off forward. Where a production incident is attributable to something implementation review examined and passed, record it as a review false negative and investigate whether the checklist, the assertion or the judgment was at fault. Report the false-negative rate to the sponsor alongside coverage and throughput, and treat it as the practice's primary quality measure, because coverage and timeliness are both achievable by a review function that examines nothing carefully. Run blind double-review on a sample of Critical-tier implementations quarterly and report the disagreement rate, using disagreements to refine checklists and assertions rather than to rank reviewers. Forward tracing in this domain most often implicates sign-offs that accepted redaction at read rather than at write, which passes inspection and retains the data in the clear.

Outcome metrics

Oversight metrics — Automated verification failures overridden by a human, with the override sustained across builds (detects: override becoming the routine path around a check that is inconvenient rather than wrong, which is how automated verification quietly becomes advisory) · Sign-offs later contradicted by a production incident attributable to what the review examined (detects: review that is procedurally complete and substantively wrong, which only forward tracing to incidents can reveal) · Blind double-review disagreement rate on Critical-tier implementations (detects: reviewer drift at the tier where it matters most, invisible unless deliberately sampled)

Success criteria

• At least 80% of verifiable controls carry continuous assertions in both pipeline and runtime, with unassertable controls listed and shrinking.
• Critical-tier assertion failures block merges or reach an owner within the day.
• Review false-negative rate is measured by forward tracing to incidents and reported to the sponsor as the practice's primary quality measure.
• At least 60% of reviewer hours go to controls automation cannot assert, and rising.
• Sampling continues on automatically verified assets so that a silently stopped assertion is detectable.

Assessment questions

1. Do at least 80% of verifiable controls carry continuous assertions running both in the pipeline and against the running system, with the unassertable remainder listed explicitly and reviewed quarterly for reduction? Evidence this in the Data domain against the go-live review record showing classification applied at ingest rather than asserted.
2. Is review false-negative rate measured by tracing sign-offs forward to incidents and reported to the sponsor as the primary quality measure, rather than coverage and timeliness which a review examining nothing carefully can also achieve? For Data, the assessor should be shown the go-live review record showing classification applied at ingest rather than asserted.
3. Is sampling maintained on automatically verified assets so that an assertion which has silently stopped evaluating is detectable, and does at least 60% of reviewer time go to controls automation cannot assert? In Data, ask to see the go-live review record showing classification applied at ingest rather than asserted.

Common pitfalls

• ❌ Asserting only in the pipeline, which misses the drift that originates outside it and is where most silent disablement happens.
• ❌ Assuming a passing assertion is evaluating, when a stopped assertion reports success indistinguishably.
• ❌ Dropping sampling once automation is in place, removing the only check on the checks.
• ❌ Reporting coverage and timeliness as quality, both of which a review function that examines nothing carefully achieves easily.
• ❌ Leaving the unassertable control list unpublished, so the verification boundary is assumed empty and is not.

Security Testing (ST)

Objective: Baseline a foundational security test battery per data archetype that asserts the controls actually hold, and run it before every release.

A program at this level tests the AI-specific controls rather than assuming them. Each archetype has a foundational battery, and every test asserts a control holds rather than that a feature works. The battery covers a data-egress canary test asserting that seeded canary records do not appear in model outputs or external logs, a no-train assertion test verifying that data sent to a provider is not retained for training where that is contractually required, a retrieval authorization test asserting that a request cannot retrieve beyond its purpose binding, an erasure-propagation test asserting a deleted record does not survive in embeddings or derived artifacts, a poisoning-resistance test against the ingest path, and a redaction test asserting logs are redacted at write. The discipline that distinguishes this practice from general quality assurance is that each test must be able to fail for the right reason: a test that passes because the condition it checks never arises provides assurance that is indistinguishable from real assurance on every dashboard.

Level 1, Foundational

Baseline a foundational security test battery per data archetype that asserts the controls actually hold, and run it before every release.

Activities

A) Build the foundational battery per archetype — Author a foundational test battery per archetype covering a data-egress canary test asserting that seeded canary records do not appear in model outputs or external logs, a no-train assertion test verifying that data sent to a provider is not retained for training where that is contractually required, a retrieval authorization test asserting that a request cannot retrieve beyond its purpose binding, an erasure-propagation test asserting a deleted record does not survive in embeddings or derived artifacts, a poisoning-resistance test against the ingest path, and a redaction test asserting logs are redacted at write. Write each test as an assertion about a control rather than as a check on behaviour, and verify that each test can fail: introduce the condition deliberately once and confirm the test catches it, which is the only way to distinguish a test that asserts something from one that passes because its condition never arises. Keep the battery small enough to run on every build at the tiers that need it, because a battery too slow to run continuously will be moved to a nightly job and then to a weekly one and then to nowhere. Version the battery alongside the reference patterns so that a pattern change and its test change travel together.

B) Map tests to threats and publish the coverage gaps — Map each test to the threat identifiers from the TA library that it exercises, and publish the threats that have no test. The coverage map is what converts a test suite into an assurance argument: without it the battery asserts that the tests pass, which is not the same claim as that the threats are addressed. Report threat coverage rather than test count, since test count rises with effort while coverage rises only with effect. Treat an untested threat at a high-risk archetype as an open finding with an owner rather than as a backlog item, and feed coverage gaps back to the threat assessment practice so that the library and the battery evolve together.

C) Run before release, with a governed waiver path — Run the battery before every release, and trigger it on the changes that actually invalidate prior results, being a new source in a pipeline, a retention or sampling change, an embedding model change, and a retrieval authorization change. Failures block release for Critical- and High-tier assets and warn for the rest. Provide a governed waiver path, because an unwaivable gate is bypassed in ways nobody sees: a waiver needs a named authority, a stated reason, an expiry and automatic notification. Review waivers weekly and treat a test waived across more than one release cycle as a finding, since the common failure here is not a single waiver but a test quietly disabled to unblock a release and never re-enabled, which reduces coverage without anyone deciding to reduce it.

Outcome metrics

Oversight metrics — Failing tests waived to permit a release, with a named waiver authority (detects: waiver becoming the routine release path, where a failing security test is treated as a scheduling obstacle rather than a result) · Tests marked as skipped or quarantined for more than one release cycle (detects: silent test decay, where a test is disabled to unblock a release and never re-enabled, so coverage falls without any decision being taken)

Success criteria

• A foundational battery exists for at least 85% of inventoried data archetypes and runs before every release.
• Every test has been verified as able to fail by deliberately introducing its condition once.
• At least 75% of threats from the library are exercised by a test, with untested threats published as gaps carrying owners.
• Failures block release for Critical- and High-tier assets, with a governed waiver path carrying authority, reason and expiry.
• A test waived across more than one release cycle is raised as a finding rather than tolerated.

Assessment questions

1. Does a foundational security test battery exist for at least 85% of inventoried data archetypes, with each test written as an assertion about a control and verified as able to fail by deliberately introducing its condition once?
2. Is every test mapped to the TA threat identifiers it exercises, with threats having no test published as gaps and threat coverage reported rather than test count? For Data, the assessor should be shown the canary-egress test and the erasure-propagation test result.
3. Does the battery run before every release and on the change types that invalidate prior results, with failures blocking Critical- and High-tier releases and a governed waiver path carrying named authority, reason and expiry? In Data, ask to see the canary-egress test and the erasure-propagation test result.

Common pitfalls

• ❌ Writing tests that pass because their condition never arises, which are indistinguishable from real assurance on every dashboard.
• ❌ Reporting test count rather than threat coverage, since count rises with effort while coverage rises only with effect.
• ❌ Building a battery too slow to run on every build, which migrates to nightly, then weekly, then nowhere.
• ❌ Providing no waiver path, which produces bypasses nobody sees rather than compliance.
• ❌ Tolerating tests quietly disabled to unblock a release, which reduces coverage without any decision being taken.

Level 2, Comprehensive

Calibrate the data test battery to asset tier, run it continuously in CI, and add adversarial testing where the automated battery cannot reach.

Activities

A) Calibrate battery depth and gating to tier — Publish what each tier receives: Critical takes the full battery on every build with failures blocking, plus quarterly adversarial testing; High takes the full battery in CI with failures blocking; Medium takes a defined subset; Low takes spot checks. Derive the applicable battery from the inventory tier automatically rather than letting teams select, because selection under release pressure converges on the fastest subset. Publish gating behaviour per tier so that nobody negotiates it at release time. Where battery runtime makes per-build execution impractical at Critical tier, treat that as a test-performance problem to be solved rather than a reason to reduce frequency, since reduced frequency is how continuous testing becomes periodic testing. Adversarial work in this domain concentrates on extraction through legitimate-looking retrieval and on poisoning through upstream sources the organisation does not control, neither of which an automated battery finds without being told where to look.

B) Add adversarial testing where automation cannot reach — Schedule adversarial testing by people against Critical-tier assets at least quarterly, scoped from the threat model and executed by testers who did not build the asset. The purpose is specifically to find what the automated battery cannot: an automated suite tests the techniques it encodes and therefore converges, passing continuously while the attack surface moves underneath it. Measure the adversarial exercise by what it found that the battery missed, and feed every such finding back into the battery as a new test, so that the exercise permanently raises the automated floor rather than producing a report. Scope must be set by the testing party rather than the tested team, because a team that steers scope away from what it knows is weak produces a clean result that means nothing. The corpus refresh that matters most here is extraction and membership-inference technique, which advances mainly through published research rather than through incidents.

C) Keep corpora fresh and remove tests that assert nothing — Refresh regression corpora on a defined cadence, targeting 90 days, drawing new cases from published research, incidents, adversarial findings and threat-intelligence updates. A prompt-injection or adversarial-input corpus that has not changed in a year tests the techniques of a year ago and passes with increasing confidence as the real surface moves. Separately, audit the battery for tests that have never failed across a full quarter and examine each: some are genuinely stable controls and some assert nothing at all, and the two are indistinguishable from a dashboard. Remove or strengthen the latter. Trigger regression runs on a new source in a pipeline, a retention or sampling change, an embedding model change, and a retrieval authorization change in addition to the scheduled cadence. Data corpora contribute well as poisoning and extraction test sets, which are expensive to build and directly reusable.

Outcome metrics

Oversight metrics — Critical-tier test waivers, with the waiver authority and expiry recorded (detects: waiver authority drifting downward under release pressure, so the decision to ship with a failing security test is taken by whoever is available) · Waived tests later implicated in a production incident (detects: waiver judgment that is procedurally correct and substantively wrong, which only forward tracing can reveal) · Test suites whose pass rate is 100% across every run for a full quarter (detects: tests that assert nothing, which are indistinguishable from passing tests on every dashboard and provide false assurance at exactly the tiers that need real assurance)

Success criteria

• Battery depth and gating behaviour are published per tier and derived automatically from the inventory.
• Every Critical-tier asset is exercised by adversarial testing at least annually, scoped by the testing party rather than the tested team.
• Every adversarial finding becomes a new automated test within 30 days, raising the automated floor permanently.
• Regression corpora are refreshed within 90 days, and triggered by the change types that invalidate prior results.
• Tests that have never failed across a quarter are audited for assertion strength and removed or strengthened.

Assessment questions

1. Is battery depth and gating behaviour published per tier and derived automatically from the inventory rather than selected by teams under release pressure? Evidence this in the Data domain against the canary-egress test and the erasure-propagation test result.
2. Is every Critical-tier asset exercised by adversarial testing at least annually with scope set by the testing party, and does every adversarial finding become a new automated test within 30 days so the automated floor rises permanently? For Data, the assessor should be shown the canary-egress test and the erasure-propagation test result.
3. Are regression corpora refreshed within 90 days and triggered by invalidating change types, and are tests that have never failed across a quarter audited for assertion strength rather than assumed to be stable controls? In Data, ask to see the canary-egress test and the erasure-propagation test result.

Common pitfalls

• ❌ Letting the tested team scope the adversarial exercise, which produces a clean result that means nothing.
• ❌ Treating adversarial findings as a report rather than as new automated tests, so the same class of finding recurs next cycle.
• ❌ Leaving regression corpora static, which tests last year's techniques with rising confidence as the real surface moves.
• ❌ Reducing test frequency when the battery gets slow, rather than treating runtime as a performance problem to solve.
• ❌ Assuming a test that never fails is a stable control, when it is indistinguishable from one that asserts nothing.

Level 3, Industry-Leading

Run continuous adversarial testing against data AI assets, measure the battery against what people can still find, and contribute test corpora back to the field.

Activities

A) Run adversarial testing continuously — Move from scheduled engagements to continuous adversarial operation against Critical-tier assets, whether by a standing internal team, a contracted continuous service or an automated adversarial harness that evolves its own cases. Measure the practice by time-to-find rather than by engagement count, and track the surprise rate, being the share of findings the automated battery should have caught but did not. A falling surprise rate indicates the battery is converging on reality; a flat one at high volume indicates the battery has converged on itself. Feed every finding into the battery within days rather than within the 30-day window that L2 allowed, since continuous operation makes the slower loop the binding constraint. Adversarial work in this domain concentrates on extraction through legitimate-looking retrieval and on poisoning through upstream sources the organisation does not control, neither of which an automated battery finds without being told where to look.

B) Close the automation gap deliberately — Treat the gap between what automation catches and what humans still find as the practice's primary work item rather than as an inevitable residue. For each human finding, ask specifically why the battery missed it: a missing case, a missing technique class, an assertion too weak, or a surface the harness cannot reach. Record the category, because the distribution tells you whether to extend the corpus, strengthen assertions or rebuild the harness, and these are very different investments. Publish the gap measure alongside coverage so that a rising coverage number with a static gap is visible as the illusion it represents. The corpus refresh that matters most here is extraction and membership-inference technique, which advances mainly through published research rather than through incidents.

C) Contribute corpora and harnesses back — Contribute at least four substantive artifacts per year, weighted toward runnable form: adversarial corpora, test harnesses, assertion libraries and reproducible technique demonstrations. Shared adversarial corpora are the scarcest resource in AI security and the one where contribution most directly raises the whole field's floor, because a corpus a peer can run is worth far more than a paper describing what it contains. Work through OWASP projects, MITRE ATLAS, OpenSSF AI and sector bodies. Anonymise, legally review and coordinate disclosure with affected providers before release. Count only accepted artifacts, and prioritise contributing the cases that most often catch real defects internally, since those carry the most evidence. Data corpora contribute well as poisoning and extraction test sets, which are expensive to build and directly reusable.

Outcome metrics

Oversight metrics — Adversarial findings that the automated battery should have caught but did not (detects: a battery that has converged on what it already tests, which passes continuously while the real attack surface moves) · Automated test results overridden at merge, with the override sustained across builds (detects: override becoming the normal path around an inconvenient check, which converts a gate into advice without any decision) · Share of red-team scope determined by the team being tested (detects: scope capture, where the tested team steers the exercise away from what it knows is weak, which makes a clean result meaningless)

Success criteria

• Critical-tier assets are under continuous adversarial testing, measured by time-to-find rather than engagement count.
• The surprise rate is tracked and falling, and reported alongside coverage so that rising coverage with a static gap is visible.
• Human findings reach the automated battery within 5 business days.
• New public techniques reach the corpus within 14 days at the 90th percentile.
• At least four accepted external contributions per year, the majority runnable corpora or harnesses, disclosure-coordinated.

Assessment questions

1. Is adversarial testing continuous against Critical-tier assets rather than scheduled, measured by time-to-find and surprise rate rather than by engagement count? Evidence this in the Data domain against the canary-egress test and the erasure-propagation test result.
2. Is the automation gap treated as the primary work item, with each human finding categorised as a missing case, missing technique class, weak assertion or unreachable surface so that investment goes to the right remedy? For Data, the assessor should be shown the canary-egress test and the erasure-propagation test result.
3. Do new public techniques reach the corpus within 14 days at the 90th percentile, and does the organisation contribute at least four accepted artifacts per year in majority-runnable form with disclosure coordinated? In Data, ask to see the canary-egress test and the erasure-propagation test result.

Common pitfalls

• ❌ Measuring adversarial testing by engagement count, which rewards activity and says nothing about whether defects are found faster.
• ❌ Treating the automation gap as an inevitable residue rather than as the work, which leaves the battery converged on itself.
• ❌ Reporting coverage without the surprise rate, so a rising coverage number conceals a static real-world gap.
• ❌ Contributing papers rather than runnable corpora, which peers read and cannot execute.
• ❌ Releasing technique demonstrations touching a provider's systems without coordinating disclosure first.

Business Function: Operations

Environment Hardening (EH)

Objective: Publish and apply a hardening baseline for each data AI archetype, so the operating environment enforces the controls rather than documenting them.

A program at this level has a published hardening baseline per archetype and applies it. The baseline covers per-request authorization on retrieval stores, encryption at rest and in transit with managed keys, redaction applied at log write, retention configured to the declared period, network isolation of training environments from production data paths, and immutable audit logging on corpus access. The distinction that matters is enforcement: a control in enforcing mode changes what can happen, and a control in audit mode changes only what is recorded, and the two are frequently confused in reporting. Consumption limits are part of the baseline rather than an operational afterthought, because an environment with no limit has no defence against the one attack that leaves every availability signal green.

Level 1, Foundational

Publish and apply a hardening baseline for each data AI archetype, so the operating environment enforces the controls rather than documenting them.

Activities

A) Publish the hardening baseline per archetype — Author a hardening baseline per archetype covering per-request authorization on retrieval stores, encryption at rest and in transit with managed keys, redaction applied at log write, retention configured to the declared period, network isolation of training environments from production data paths, and immutable audit logging on corpus access. Express the baseline as configuration that can be applied and checked rather than as prose, and publish it alongside the reference architecture patterns so that the secure configuration travels with the secure design. State for every control whether it is enforcing or advisory, and default to enforcing, because a baseline whose enforcement status is unstated is assumed to be enforcing by everyone reporting on it and is frequently not. Where a control genuinely cannot be enforced in the current environment, record it as a known limitation rather than listing it in the baseline as though it were active.

B) Apply the baseline and measure conformance against the running environment — Apply the baseline to every asset of the archetype and measure conformance from the environment rather than from a checklist. Read configuration state, policy enforcement mode and runtime behaviour, and report conformance per control rather than as a single score, because one aggregate number lets a fully conformant set of easy controls conceal a missing hard one. Report separately the count of controls running in audit rather than enforcing mode, which is the most common way a baseline is nominally met and practically absent. Reconcile monthly and treat any asset with no baseline applied as a finding with an owner rather than as an onboarding backlog item.

C) Configure consumption limits and govern exceptions — Configure per-consumer retrieval quotas with a declared behaviour at the quota as part of the baseline for every asset, expressed relative to per retrieval query so that the limit bears a defensible relationship to the work the asset actually does. Declare the behaviour at the limit explicitly, being degrade, queue or refuse, and never silently continue, because a limit that is exceeded without effect is a reporting artefact rather than a control. Operate a baseline exception process: any control not applied requires an owner, a compensating control, an approver and an expiry, reviewed monthly, with expired exceptions treated as findings. Exceptions are healthy when visible and corrosive when absorbed into the environment as the new normal.

Outcome metrics

Oversight metrics — Baseline exceptions approved without a compensating control recorded (detects: exception approval becoming a formality, where the control is waived and nothing replaces it, which is indistinguishable from not having the control at all) · Controls switched from enforcing to audit mode, with the change approved and time-bounded (detects: silent downgrade, where a control is moved to audit mode to resolve an operational problem and left there, so the baseline reports as met while enforcing nothing)

Consumption metrics — Assets with a consumption limit configured and a declared at-limit behaviour per per production asset · Configured limit against observed consumption per per retrieval query · Limit breaches where the declared at-limit behaviour did not occur per per limit breach event

Success criteria

• A hardening baseline is published per archetype as applicable configuration rather than prose, and applied to at least 90% of production data AI assets.
• At least 90% of baseline controls run in enforcing mode, with every audit-mode control recorded as an exception with an owner.
• Every production asset has a consumption limit configured with an explicit, declared at-limit behaviour.
• Conformance is reported per control rather than as an aggregate score that can conceal a missing hard control.
• Every baseline exception carries an owner, compensating control, approver and expiry, with none past expiry.

Assessment questions

1. Is a hardening baseline published per data archetype as applicable configuration rather than prose, applied to at least 90% of production assets, and measured against the running environment rather than a checklist?
2. Are at least 90% of baseline controls in enforcing rather than audit mode, with audit-mode controls recorded as exceptions, and is conformance reported per control rather than as an aggregate that can conceal a missing hard control? For Data, the assessor should be shown the retrieval authorization and log-redaction configuration as applied.
3. Does every production asset have per-consumer retrieval quotas with a declared behaviour at the quota configured with an explicit declared at-limit behaviour, and does every baseline exception carry an owner, compensating control, approver and expiry?

Common pitfalls

• ❌ Leaving enforcement status unstated, so controls in audit mode are reported and believed to be enforcing.
• ❌ Reporting a single aggregate conformance score, which lets a set of easy controls conceal a missing hard one.
• ❌ Measuring conformance from a checklist rather than from the running environment, which records intent rather than state.
• ❌ Configuring a limit without declaring the behaviour at the limit, producing a threshold that is crossed without effect.
• ❌ Allowing exceptions without compensating controls, which is indistinguishable from not having the control.

Level 2, Comprehensive

Calibrate data hardening intensity to asset tier, detect and remediate baseline drift continuously, and make consumption limits tier-proportionate and tested.

Activities

A) Publish tier-calibrated baselines — Publish a baseline variant per tier rather than one baseline with exceptions. Critical adds per-tenant isolation, no standing human access with time-bound elevation, shorter secret rotation intervals, stricter egress allowlisting and full logging; Low takes single sign-on and the essential controls. Deriving the applicable variant from the inventory tier removes the negotiation that otherwise happens per asset and resolves toward the lighter variant. Publish the reasoning alongside the variants so that a team subject to the Critical baseline understands why, because a baseline that arrives without justification is complied with minimally and circumvented where possible. Drift in this domain most often appears as redaction disabled for debugging and a retrieval store granted a broader service identity, the second of which looks like a routine access change.

B) Detect and remediate drift continuously — Stand up continuous drift detection across the baseline controls for Critical- and High-tier assets, reading configuration state, policy enforcement mode and runtime behaviour, and alerting on change rather than on schedule. Publish remediation windows by tier and severity, tighter at Critical, and route detections into the shared issue backlog with tier-appropriate severity rather than into a dashboard. Drift is the defining failure of hardening at scale because environments are hardened once and then changed for operational reasons that are individually reasonable: a rule widened for a debugging session, a quota raised under load, an enforcement mode relaxed to unblock a deployment. Each is defensible and none is reverted unless something notices. Limit exercises in this domain drive a retrieval store to its quota and confirm the declared behaviour, which matters because extraction and legitimate heavy use are indistinguishable without it.

C) Make limits tier-proportionate and exercise them — Set consumption limits proportionate to tier and to the asset's observed workload, expressed relative to per retrieval query. Exercise the at-limit behaviour at least annually for Critical- and High-tier assets by driving the asset to its limit in a controlled test and confirming that the declared behaviour actually occurs, because a limit whose behaviour has never been observed is an assumption. Record the exercise result. Treat a cost-driven change to a limit, a logging level, a retention period or an enforcement mode on a Critical-tier asset as a material change requiring approval, since these are precisely the controls that cost pressure erodes and the erosion is invisible on any security dashboard. Cost pressure in this domain lands first on retention, which is the control regulators and investigators both depend on and the one whose absence is discovered latest.

Outcome metrics

Oversight metrics — Time-bound elevations to Critical-tier environments that expired without being reviewed (detects: elevation becoming effectively standing access, where a time bound exists on paper and is renewed automatically without anyone assessing whether it is still needed) · Drift remediation deferrals at Critical tier, with a named acceptance authority (detects: deferral becoming the default response to drift under operational pressure, which converts continuous detection into continuous reporting) · Enforcement-mode relaxations approved without an expiry (detects: a temporary relaxation becoming permanent because nothing in the approval obliges anyone to revisit it)

Consumption metrics — Critical- and High-tier assets whose at-limit behaviour has been exercised and recorded per per Critical or High tier asset · Limit headroom: configured limit against observed peak consumption, by tier per per retrieval query · Cost-driven relaxations of limits, logging or retention on Critical-tier assets per per approved change

Success criteria

• A baseline variant per tier is published with its reasoning and derived automatically from the inventory.
• At least 95% of Critical-tier assets are under continuous drift detection, with detection within 3 business days.
• Standing human access to Critical-tier AI environments is zero, with time-bound elevation only.
• Every Critical-tier asset's at-limit behaviour has been exercised within 12 months and the result recorded.
• Cost-driven changes to limits, logging, retention or enforcement mode at Critical tier require approval as material changes.

Assessment questions

1. Is a baseline variant published per tier with its reasoning and derived automatically from the inventory, rather than one baseline with per-asset exceptions that resolve toward the lighter option? Evidence this in the Data domain against the retrieval authorization and log-redaction configuration as applied.
2. Are at least 95% of Critical-tier assets under continuous drift detection with a 3 business day detection window, with detections routed into the shared backlog rather than a dashboard, and is standing human access to Critical-tier environments zero? For Data, the assessor should be shown the retrieval authorization and log-redaction configuration as applied.
3. Has every Critical-tier asset's at-limit behaviour been exercised within 12 months, and do cost-driven changes to limits, logging, retention or enforcement mode require approval as material changes? In Data, ask to see the retrieval authorization and log-redaction configuration as applied.

Common pitfalls

• ❌ Running one baseline with per-asset exceptions rather than tier variants, which turns every asset into a negotiation resolving toward the lighter option.
• ❌ Detecting drift on a schedule rather than on change, which reintroduces the point-in-time weakness the activity exists to remove.
• ❌ Renewing time-bound elevations automatically, which produces standing access with a time bound written on it.
• ❌ Declaring limits without exercising them, so their behaviour is first learned during the event they exist for.
• ❌ Exempting cost-driven changes from approval, which is how hardened environments degrade with everyone's agreement.

Level 3, Industry-Leading

Make data AI environments self-correcting, with baselines applied at provisioning and drift remediated automatically where it is safe to do so.

Activities

A) Apply baselines at provisioning — Move baseline application to provisioning time so that an asset is conformant from creation rather than hardened afterwards. Implement the baseline as infrastructure-as-code modules, admission policies and platform defaults, so the conformant configuration is the default path and a non-conformant asset requires deliberate effort to create. Block provisioning of Critical-tier assets that would not meet baseline. Measure the time an asset spends non-conformant after creation and drive it to zero, since that window is the one period in an asset's life when it is both unhardened and unmonitored, and it is the window most exploitation of new infrastructure actually uses. Provisioning-time conformance in this domain means a new corpus cannot be created without a classification and a retention period, which removes the most common origin of ungoverned data.

B) Automate remediation within enumerated bounds — Enumerate explicitly the drift classes that may be remediated automatically, being those where the correct state is unambiguous and reversion carries no operational risk, such as re-enabling logging, restoring an enforcement mode, or closing an egress rule not in the allowlist. Everything outside that list alerts to a human. Log every automated action with what changed, what it was reverted to and why, and make the log reviewable, because self-correction that is silent can mask a genuine operational need indefinitely and the team whose change was reverted will simply make it again. Track repeated remediation of the same control on the same asset as a signal that the baseline is wrong or the operational need is real, and route it to review rather than continuing to revert. Safe automatic remediation in this domain covers restoring redaction at log write and reverting an over-broad retrieval identity, while anything touching a live pipeline alerts instead.

C) Govern adaptive limits and publish the automation boundary — Allow consumption limits to adapt to observed workload within governed bounds, so that a limit tracks the asset's real demand rather than remaining static and becoming either irrelevant or obstructive. Set a hard ceiling that adaptation cannot cross without human approval, because an adaptive limit with no ceiling accommodates an attack as readily as it accommodates growth, and that is precisely the failure mode denial-of-wallet exploits. Express bounds relative to per retrieval query. Publish the list of controls that cannot be automatically applied or remediated and review it quarterly for reduction, so that the boundary of the automation is known rather than assumed to be complete. Adaptive limits in this domain track retrieval cost per query, where a rising per-query cost is a stronger extraction signal than rising volume alone.

Outcome metrics

Oversight metrics — Automated remediations reverted by a human within 24 hours (detects: automation acting against a genuine operational need, which indicates the enumerated safe class was wrongly scoped and is invisible unless reversions are tracked) · Adaptive limit ceiling increases approved, with the approver and justification recorded (detects: ceiling creep, where each individual increase is justified and the cumulative effect is an unbounded limit nobody decided to create) · Alerts for non-automatable drift acknowledged but not remediated within the tier window (detects: alert fatigue on the drift classes that specifically require human judgment, which are the ones automation deliberately left to people)

Consumption metrics — Adaptive limits operating within a governed hard ceiling per per asset with an adaptive limit · Cumulative ceiling increase over the trailing year, per asset per per retrieval query · Drift events on consumption-limit configuration, and time spent drifted per per limit configuration drift event

Success criteria

• At least 95% of new assets are provisioned already conformant, with non-conformance windows at or under one hour.
• The drift classes eligible for automatic remediation are enumerated explicitly, and everything outside alerts to a human.
• Every automated remediation is logged with change, reversion and reason, and is reviewable.
• Repeated remediation of the same control is treated as a signal and routed to review rather than continuing to revert.
• Adaptive limits operate within a governed hard ceiling that requires human approval to cross.

Assessment questions

1. Are at least 95% of new assets provisioned already conformant with non-conformance windows at or under one hour, and is provisioning of non-conformant Critical-tier assets blocked? Evidence this in the Data domain against the retrieval authorization and log-redaction configuration as applied.
2. Are the drift classes eligible for automatic remediation enumerated explicitly with everything else alerting to a human, and is every automated action logged with change, reversion and reason and reviewable rather than silent? For Data, the assessor should be shown the retrieval authorization and log-redaction configuration as applied.
3. Is repeated automatic remediation of the same control treated as a signal routed to baseline or operational review, and do adaptive limits operate within a governed hard ceiling requiring human approval to cross? In Data, ask to see the retrieval authorization and log-redaction configuration as applied.

Common pitfalls

• ❌ Hardening after provisioning rather than at it, leaving a window where the asset is both unhardened and unmonitored.
• ❌ Automating remediation without enumerating the safe classes, which eventually reverts something that mattered operationally.
• ❌ Making self-correction silent, so the team whose change was reverted simply makes it again and nobody learns.
• ❌ Treating repeated automatic remediation as a success metric rather than as evidence the baseline or the operational need is wrong.
• ❌ Allowing adaptive limits without a hard ceiling, which accommodates an attack as readily as growth.

Issue Management (IM)

Objective: Run one backlog for data AI issues with an AI-aware severity taxonomy and published service levels, so AI findings age and escalate like everything else.

A program at this level handles AI issues on the same rails as every other security issue, with a severity taxonomy extended to cover the failure modes AI actually produces. The taxonomy names regulated data reaching an ungoverned corpus, a canary appearing outside its governed store, systematic extraction against a retrieval or embedding store, poisoning detected in a training corpus, and an erasure that failed to propagate, because a severity scheme built for classical vulnerabilities rates these either as nothing or as whatever the assessor improvises. One backlog matters more than it sounds: AI findings kept in a separate register age quietly, escalate to nobody, and are rediscovered at the next assessment.

Level 1, Foundational

Run one backlog for data AI issues with an AI-aware severity taxonomy and published service levels, so AI findings age and escalate like everything else.

Activities

A) Extend the severity taxonomy to AI failure modes — Extend the existing severity taxonomy rather than writing a new one, adding the AI-specific classes that a classical scheme cannot rate: regulated data reaching an ungoverned corpus, a canary appearing outside its governed store, systematic extraction against a retrieval or embedding store, poisoning detected in a training corpus, and an erasure that failed to propagate. For each class, state what makes it Critical, High, Medium or Low, using the dimensions that actually vary, being the data exposed, the reversibility of the effect, the population affected and whether the asset acts or advises. Include denial-of-wallet as a named class with its own criteria, because it degrades no availability signal and a scheme keyed to availability rates it as nothing while the budget drains. Publish worked examples per class, since a taxonomy without examples is applied inconsistently and its inconsistency is invisible in the aggregate.

B) Run one backlog with wired-in sources and service levels — Route every AI finding into the shared security backlog rather than a separate AI register. Wire the sources so findings arrive automatically: implementation review, security testing, monitoring detections, adversarial exercises, vendor advisories and self-reports. Assign a severity, an owner and a service level at creation rather than at triage, because an issue that waits for triage to acquire a service level is an issue whose clock has not started. Publish service levels by severity and report adherence rather than backlog size, since backlog size is a function of discovery rate as much as of remediation and improves when discovery stops.

C) Document containment for AI-specific incidents — Write containment actions for each AI-specific incident class before they are needed, covering suspending the affected pipeline, revoking retrieval credentials, quarantining the corpus, and halting training runs consuming it. AI containment differs from classical containment in a way that catches teams out: the fastest effective action is usually to remove the system's ability to act rather than to take the system offline, because an AI asset that is merely unavailable stops delivering value while an asset whose tool access is revoked stops causing harm and may keep delivering. Name who is authorised to invoke each action at what hour, and verify the technical means exists and works. An action documented but never exercised is discovered to be unavailable at the worst moment.

Outcome metrics

Oversight metrics — Issues closed without a recorded remediation or acceptance (detects: closure as backlog hygiene rather than resolution, where an issue is marked done because it is old rather than because anything changed) · Severity downgrades applied after creation, with the downgrading authority recorded (detects: severity drift under service-level pressure, where reclassification is used to make a clock more comfortable rather than to correct an assessment)

Consumption metrics — Denial-of-wallet incidents detected, with time from onset to containment per per denial-of-wallet incident · Consumption anomaly cost accrued before containment per per retrieval query · AI incidents whose containment action was a consumption limit rather than a service withdrawal per per contained AI incident

Success criteria

• All AI issues are managed in the shared security backlog with the same ageing and escalation as other security issues.
• The severity taxonomy names the AI-specific classes including denial-of-wallet, with published criteria and worked examples.
• At least 95% of issues carry a severity, owner and service level at creation rather than at triage.
• At least 80% of issues arrive from a wired source rather than manual entry.
• Containment actions are documented per AI incident class and technically verified as available.

Assessment questions

1. Does the severity taxonomy extend the existing scheme with AI-specific classes including regulated data reaching an ungoverned corpus and denial-of-wallet, with published criteria keyed to data exposed, reversibility, population affected and whether the asset acts or advises, plus worked examples per class?
2. Are all AI issues managed in the shared security backlog with at least 95% carrying severity, owner and service level at creation, and do at least 80% arrive from a wired source rather than manual entry? For Data, the assessor should be shown an incident record showing corpus quarantine and dependent training runs halted.
3. Are containment actions documented for every AI incident class, covering suspending the affected pipeline and the others, with named invocation authority and technical availability verified rather than assumed?

Common pitfalls

• ❌ Keeping AI findings in a separate register, where they age quietly, escalate to nobody and are rediscovered at the next assessment.
• ❌ Using a severity scheme keyed to availability, which rates denial-of-wallet as nothing while the budget drains.
• ❌ Assigning severity at triage rather than creation, so the issue's clock does not start until someone looks at it.
• ❌ Reporting backlog size rather than service-level adherence, a number that improves when discovery stops.
• ❌ Documenting containment actions without verifying the technical means exists, which is discovered at the worst moment.

Level 2, Comprehensive

Calibrate data AI issue service levels to asset tier, run rehearsed AI incident response, and close the loop from root cause to the practice that owns it.

Activities

A) Calibrate service levels and authority to tier — Publish remediation service levels by severity and asset tier, tighter at Critical, and report adherence per tier rather than in aggregate, because an aggregate figure can be met while every Critical-tier breach is missed. Make deferral and risk-acceptance authority rise with tier and severity, so a deferred Critical finding requires executive acceptance with an expiry and quarterly review. Report deferrals to the sponsor alongside closures, since a practice reporting only what it closed describes a backlog that does not exist. Track findings deferred and later implicated in an incident as a distinct measure of whether deferral judgment is sound rather than merely authorised. Tier-proportionate response in this domain is shaped by whether data has already moved, since containment after egress is a notification problem rather than a technical one.

B) Rehearse AI incident response and instrument the notification clocks — Write and exercise incident playbooks for each AI-specific class, covering detection signals, containment using suspending the affected pipeline, revoking retrieval credentials, quarantining the corpus, and halting training runs consuming it, evidence preservation, communication and recovery. Exercise at least semi-annually with the people who would actually respond, including out of hours, and measure the exercise by the gaps it finds rather than by its completion. Instrument the regulatory notification clocks explicitly: identify which incident classes trigger EU AI Act serious-incident reporting or GDPR breach notification, what starts the clock, who decides, and what evidence the notification requires. A team that discovers a 72-hour obligation 40 hours into an incident has lost most of the window to a question it could have answered in advance. Playbook exercises in this domain rehearse quarantining a corpus mid-pipeline and halting dependent training runs, which surfaces dependencies nobody had mapped.

C) Route root causes to the owning practice — Require every incident reaching a root cause to name the practice that owns the fix and to open a work item there, rather than closing the incident where it was found. An incident caused by a missing architectural control belongs to secure architecture, one caused by an untested threat belongs to security testing, one caused by a knowledge gap belongs to education. Closing it in the incident record alone guarantees recurrence, and recurrence is the single clearest signal that a response practice is functioning as a repair shop rather than as a feedback mechanism. Track the share of incidents whose root cause was already known and untreated, which separates a discovery problem from an action problem and is usually uncomfortable reading. Root causes in this domain most often belong to security requirements and secure architecture, particularly classification at ingest and erasure propagation.

Outcome metrics

Oversight metrics — Incidents closed without a root cause recorded, by severity (detects: closure without diagnosis, which guarantees recurrence and is most common under load at exactly the times recurrence is least affordable) · Risk acceptances at Critical tier, with the accepting authority and expiry recorded (detects: acceptance authority drifting downward under pressure, so the decision to live with a Critical risk is taken by whoever is available) · Exercise findings closed without a change to the playbook or the environment (detects: rehearsal treated as a compliance activity rather than as a source of correction, which is visible only by tracking what the exercise changed)

Consumption metrics — Denial-of-wallet incidents contained within their tier service level per per denial-of-wallet incident · Cost accrued between detection and containment, by tier per per retrieval query · Emergency consumption limits applied during incidents and subsequently reverted per per applied emergency limit

Success criteria

• Service levels vary by tier and severity with adherence at or above 90% reported per tier rather than in aggregate.
• Every AI incident playbook has been exercised within 12 months, including an out-of-hours exercise, and measured by the gaps it found.
• Regulatory notification clocks are instrumented with owners and evidence requirements identified in advance.
• At least 90% of incidents reaching root cause open a work item in the practice that owns the fix.
• Incidents whose root cause was already known and untreated are tracked openly and falling.

Assessment questions

1. Do service levels vary by asset tier and severity with adherence reported per tier rather than in an aggregate that can be met while every Critical-tier breach is missed? Evidence this in the Data domain against an incident record showing corpus quarantine and dependent training runs halted.
2. Have AI incident playbooks been exercised within 12 months including out of hours, measured by the gaps found, and are regulatory notification clocks instrumented in advance with owners and evidence requirements identified? For Data, the assessor should be shown an incident record showing corpus quarantine and dependent training runs halted.
3. Do at least 90% of incidents reaching root cause open a work item in the owning practice rather than closing where they were found, and is the share whose root cause was already known and untreated tracked openly? In Data, ask to see an incident record showing corpus quarantine and dependent training runs halted.

Common pitfalls

• ❌ Reporting service-level adherence in aggregate, which can be met while every Critical-tier breach is missed.
• ❌ Writing playbooks without exercising them, so their gaps are found during the incident rather than before it.
• ❌ Discovering a notification obligation mid-incident, by which point most of the window is gone.
• ❌ Closing incidents where they were found, which guarantees the same root cause returns.
• ❌ Reporting closures without deferrals, which describes a backlog that does not exist.

Level 3, Industry-Leading

Automate data AI issue triage and containment where it is safe, measure response by outcome rather than activity, and contribute incident learning back to the field.

Activities

A) Automate triage and bounded first-line containment — Automate severity assignment, ownership routing and duplicate detection from the detection signal, so an issue arrives with its clock started and its owner assigned. Enumerate explicitly the containment actions that may fire automatically, being those where the correct response is unambiguous and reversible, such as applying an emergency consumption limit, revoking a specific tool credential or blocking a destination. Everything else alerts. Log every automated action with what was done, on what evidence, and how to reverse it. Publish the human-judgment classes explicitly, because automated containment is safest when its boundary is stated rather than assumed, and the assumption is always that it covers more than it does. Automatic containment in this domain is narrower, since suspending a pipeline has downstream consequences; revoking retrieval credentials and quarantining a corpus are the safely automatable actions.

B) Measure response by outcome rather than activity — Shift the primary measures from throughput to outcome: recurrence rate by root-cause class, harm avoided estimated from the consumption ledger and the affected population, and time from detection to containment rather than to closure. Ticket throughput and backlog size remain as health indicators but are explicitly not the practice's outcome measures, because both are dominated by discovery rate and both improve when discovery degrades. Report recurrence to the sponsor as the headline, since a response practice that closes quickly and recurs constantly is failing at the thing it exists for while every throughput metric looks excellent. Recurrence in this domain clusters around classification and erasure propagation, which points remediation at security requirements and secure architecture.

C) Contribute anonymised incident learning back — Contribute at least four substantive artifacts per year: anonymised incident case studies, detection and containment patterns, playbooks others can adopt, and technique observations submitted to MITRE ATLAS where they fit. The public corpus of real AI security incidents is thin, disproportionately so relative to the deployment base, which means most organisations learn only from their own incidents, which is the slowest and most expensive way to learn anything. Contributing changes that for everyone including the contributor. Anonymise carefully, review legally, coordinate disclosure with any affected vendor or provider, and count only accepted artifacts. Prefer contributing the incidents that were most surprising internally, since those carry the most information for others. Data incident write-ups contribute well, particularly erasure-propagation failures, which most organisations have and few discuss.

Outcome metrics

Oversight metrics — Automated containment actions reversed by a human within one hour (detects: automation acting on a misread signal, which indicates the enumerated class was wrongly scoped and is only visible if reversals are tracked separately from actions) · Incidents in human-judgment classes acknowledged but not actioned within the tier window (detects: alert fatigue concentrating precisely on the classes automation deliberately left to people because they need judgment) · Recurrence accepted without a corresponding work item in the owning practice (detects: recurrence normalised as a cost of doing business, which is how a response practice becomes a repair shop)

Consumption metrics — Automated emergency consumption limits applied within the detection window per per denial-of-wallet detection · Cost accrued between detection and automated containment per per retrieval query · Harm avoided from contained consumption incidents, computed from burn rate and time to contain per per contained consumption incident

Success criteria

• At least 85% of issues arrive auto-triaged with severity, owner and clock assigned.
• Automatic containment classes are enumerated explicitly, everything else alerts, and every automated action is logged with evidence and a reversal path.
• Recurrence rate by root-cause class is the practice's headline measure, reported to the sponsor, rather than throughput or backlog size.
• Harm avoided is estimated from the consumption ledger and affected population and reported annually.
• At least four accepted external incident-learning contributions per year, anonymised and disclosure-coordinated.

Assessment questions

1. Are at least 85% of issues auto-triaged with severity, owner and clock assigned, and are automatic containment classes enumerated explicitly with everything else alerting and every action logged with evidence and a reversal path? Evidence this in the Data domain against an incident record showing corpus quarantine and dependent training runs halted.
2. Is recurrence rate by root-cause class the practice's headline outcome measure reported to the sponsor, rather than throughput and backlog size which are dominated by discovery rate and improve when discovery degrades? For Data, the assessor should be shown an incident record showing corpus quarantine and dependent training runs halted.
3. Does the organisation contribute at least four accepted anonymised incident-learning artifacts per year with disclosure coordinated, prioritising the incidents that were most surprising internally? In Data, ask to see an incident record showing corpus quarantine and dependent training runs halted.

Common pitfalls

• ❌ Automating containment without enumerating the safe classes, which eventually acts on a misread signal in a way that matters.
• ❌ Measuring the practice by throughput, which is dominated by discovery rate and improves when discovery degrades.
• ❌ Tracking automated actions without tracking reversals, which hides automation that is acting wrongly.
• ❌ Normalising recurrence as a cost of doing business, which is how a response practice becomes a repair shop.
• ❌ Withholding incident learning, which keeps the whole field, including the withholder, learning only from its own incidents.

Monitoring & Logging (ML)

Objective: Establish a logging baseline per data archetype and a detection set covering the top threats, with retention set by the longest applicable obligation.

A program at this level can see what its AI is doing. Each archetype has a logging baseline covering corpus access with requester identity and purpose, retrieval queries and the documents returned, training and fine-tuning run records with their input corpora, embedding generation and re-index events, erasure operations and their propagation, and canary access events, and a detection set targeting the threats the threat library rates highest. Retention is set by the longest applicable regulatory obligation rather than by storage cost, because the retention decision is usually made on cost grounds by people who do not know the obligation and is discovered to be wrong during an investigation.

Level 1, Foundational

Establish a logging baseline per data archetype and a detection set covering the top threats, with retention set by the longest applicable obligation.

Activities

A) Publish and verify the logging baseline per archetype — Define a logging baseline per archetype covering corpus access with requester identity and purpose, retrieval queries and the documents returned, training and fine-tuning run records with their input corpora, embedding generation and re-index events, erasure operations and their propagation, and canary access events. Verify that each stream is actually emitting rather than merely configured, by asserting the presence of recent events per stream per asset, because a logging configuration that exists and emits nothing passes every configuration check and fails the one investigation that needs it. Include the identity and context fields that make a log answerable: who initiated, what asset, which model and version, which session. A log that records that something happened without recording which model version did it cannot answer the question that follows any output-integrity incident. Publish the baseline alongside the hardening baseline so telemetry travels with configuration.

B) Build the detection set from the threat library — Build detections for the threats the TA library rates highest, covering in this domain a canary appearing outside its governed store, retrieval volume or breadth departing from a requester's baseline, an unregistered corpus entering a training run, erasure that did not propagate within its window, and retrieval cost per query departing from the store median. Map each detection to the threat identifiers it addresses and publish the threats with no detection, because coverage is the claim that matters and detection count is not a proxy for it. Route every detection to the shared issue backlog with a severity from the IM taxonomy rather than to a dashboard, since a detection whose only destination is a dashboard has no owner and no clock. Include at least one consumption-anomaly detection from the outset, expressed against a denominated metric, because it is the cheapest detection available and the only one that covers denial-of-wallet.

C) Set retention by obligation and make logs usable — Set retention per log class by the longest applicable regulatory, contractual or investigative requirement, and record which obligation drives each period so the decision is reviewable. Retention is routinely set on storage cost by people who do not hold the obligation, and the error surfaces during an investigation or a regulator request when it can no longer be corrected. Apply redaction at write rather than at read so that sensitive content is not retained in the clear pending a query-time control. Verify that logs are actually queryable at the volumes an investigation would need, because a retention period that cannot be searched within a working day is not meaningfully retained.

Outcome metrics

Oversight metrics — Detections acknowledged but closed without an investigation note (detects: alert closure as queue management rather than as analysis, which is invisible in any alert-volume metric and is the normal response to a noisy detection set) · Detections muted or suppressed for more than one week, with the suppressing authority recorded (detects: silent suppression, where a noisy detection is muted to clear a queue and never re-enabled, so coverage falls with no decision taken)

Consumption metrics — Consumption-anomaly detections active against a denominated metric per per archetype detection set · Detection threshold expressed against consumption per unit of work rather than absolute spend per per retrieval query · Logging and retention cost per retained record, by log class per per retained record

Success criteria

• A logging baseline is defined per archetype and verified as emitting for at least 85% of data archetypes.
• At least 70% of top-rated library threats are covered by a detection, with uncovered threats published as gaps.
• Every log class has a retention period traced to a named obligation rather than to storage cost.
• Every active detection routes to the shared issue backlog with an IM severity rather than to a dashboard.
• At least one consumption-anomaly detection operates against a denominated metric from the outset.

Assessment questions

1. Is a logging baseline defined per data archetype covering identity, asset, model and version context, and verified as actually emitting rather than merely configured, for at least 85% of archetypes?
2. Do detections cover at least 70% of the top-rated threats from the library with uncovered threats published as gaps, and does every detection route to the shared issue backlog with an IM severity rather than to a dashboard? For Data, the assessor should be shown the corpus access log and the canary-trigger detection.
3. Is retention per log class traced to a named obligation rather than to storage cost, with redaction applied at write, and are log streams verified as queryable within a working day at investigation volumes? In Data, ask to see the corpus access log and the canary-trigger detection.

Common pitfalls

• ❌ Verifying that logging is configured rather than that it is emitting, which passes a stream that has been silent for months.
• ❌ Omitting model and version from the log context, which makes every output-integrity investigation start from scratch.
• ❌ Setting retention on storage cost, an error discovered during an investigation when it can no longer be corrected.
• ❌ Routing detections to a dashboard, where they have no owner and no clock.
• ❌ Building a detection set with no consumption anomaly, leaving the one attack class that produces no availability signal entirely uncovered.

Level 2, Comprehensive

Calibrate data detection and logging depth to asset tier, tune for alert quality, and establish behavioural baselines so departure is detectable.

Activities

A) Calibrate logging and detection depth to tier — Publish what each tier receives: Critical gets full logging, the complete detection set tuned for the asset, behavioural baselining and the shortest alerting path; Low gets baseline logging and the shadow-AI detections. Derive the applicable set from the inventory tier automatically. Publish the retention and sampling position per tier explicitly, because sampling is where cost pressure lands first and a Critical-tier asset whose telemetry is sampled has monitoring that is statistical at exactly the tier where specific answers are needed. Treat a sampling or retention reduction at Critical tier as a material change requiring approval rather than an operational tuning decision. Behavioural baselining in this domain covers retrieval breadth and volume per requester, because extraction looks like enthusiastic legitimate use until compared against a baseline.

B) Establish behavioural baselines per Critical-tier asset — Build a behavioural baseline per Critical-tier asset covering its normal operating envelope: request and tool-call volumes, the distribution of actions taken, typical session shapes, identity and access patterns, and consumption per unit of work. Alert on departure rather than only on signature, because the AI failure modes that matter most produce traffic that is individually well-formed and collectively abnormal, and a signature-based set cannot see an agent competently doing something nobody asked for. Refresh baselines on a defined cadence and after any material change to the asset, and record baseline shifts explicitly, since an unexplained baseline shift is itself the signal and is lost if baselines are refreshed silently. Detection-as-code in this domain is fragile to retrieval and embedding schema changes, which silently break a rule that continues to report healthy.

C) Tune for alert quality and report precision with recall — Measure detection quality using analyst disposition labels, and report precision and recall together. Precision alone rewards a detector that fires twice a year and misses everything, which is exactly the detector a tuning programme under alert-fatigue pressure will produce. Set a false-positive budget per detection and tune or retire detections that exceed it, because analysts who learn that a detection is usually wrong stop reading it and the coverage is then notional. Assert continuously that each detection is still evaluating and each stream still emitting, alerting within hours, since a stopped detection and a clean environment are indistinguishable in every dashboard. Data detection content contributes well as extraction and canary-trigger rules, which are expensive to develop and widely applicable.

Outcome metrics

Oversight metrics — Alert disposition distribution per analyst, compared across analysts on comparable queues (detects: disposition drift, where the same alert would be escalated by one analyst and closed by another, making detection outcome depend on shift rather than on evidence) · Alerts per analyst per shift against the published ceiling (detects: analyst overload, where triage quality falls as volume rises while closure rates continue to look healthy) · Critical-tier detections closed as false positive and later implicated in an incident (detects: dismissal under load of exactly the alerts that mattered, visible only by tracing closures forward)

Consumption metrics — Consumption-anomaly detections operating against a per-asset behavioural baseline per per retrieval query · Time from a consumption anomaly beginning to its detection per per consumption anomaly · Telemetry cost per retained record at Critical tier, tracked against sampling decisions per per retained record

Success criteria

• Logging and detection depth are published per tier and derived automatically from the inventory.
• At least 90% of Critical-tier assets have a current behavioural baseline, refreshed on cadence and after material change, with shifts recorded.
• Every active detection reports precision and recall together, with a false-positive budget enforced by tuning or retirement.
• A silent log stream or stopped detection is detected within 4 hours.
• Sampling or retention reductions at Critical tier are approved as material changes rather than treated as operational tuning.

Assessment questions

1. Are logging and detection depth published per tier and derived automatically from the inventory, with sampling and retention positions stated per tier and reductions at Critical tier treated as material changes? Evidence this in the Data domain against the corpus access log and the canary-trigger detection.
2. Do at least 90% of Critical-tier assets carry a current behavioural baseline alerting on departure rather than only on signature, with baseline shifts recorded explicitly rather than refreshed silently? For Data, the assessor should be shown the corpus access log and the canary-trigger detection.
3. Is detection quality measured with precision and recall reported together using analyst disposition labels, with a false-positive budget enforced, and is a stopped detection or silent stream detected within 4 hours? In Data, ask to see the corpus access log and the canary-trigger detection.

Common pitfalls

• ❌ Reporting precision without recall, which rewards a detector that fires twice a year and catches nothing.
• ❌ Relying on signature detections alone, which cannot see an agent competently doing something nobody asked for.
• ❌ Refreshing behavioural baselines silently, which discards the baseline shift that was itself the signal.
• ❌ Allowing sampling at Critical tier as an operational tuning decision, making monitoring statistical where specific answers are needed.
• ❌ Assuming a quiet detection is a clean environment, when a stopped detection looks identical.

Level 3, Industry-Leading

Engineer data detections as versioned, tested code, close the loop from incident to detection continuously, and contribute detection content back to the field.

Activities

A) Engineer detections as versioned, tested code — Move detection content into version control with the same discipline as application code: each detection carries an owner, a version, a documented intent, and test cases that assert it fires on known-positive events and does not fire on known-negative ones. Run the tests continuously against replayed or synthetic events, because the most common silent failure in monitoring is a detection that still exists, still evaluates and no longer matches anything due to a log-format change upstream, and nothing in a conventional monitoring stack reports that. Deploy through a pipeline with review, so a detection change is as traceable as a code change and can be reverted as easily. Behavioural baselining in this domain covers retrieval breadth and volume per requester, because extraction looks like enthusiastic legitimate use until compared against a baseline. Detection tests in this domain replay retrieval and canary-access events, and the known-negative cases matter most because legitimate heavy retrieval is common.

B) Close the incident-to-detection loop continuously — Require every incident and every adversarial finding to be assessed for detection coverage within days: did a detection fire, should one have, and if so why did it not. Where no detection existed, author one and deploy it within 10 business days, with its test cases. Where one existed and did not fire, treat that as a detection defect with an owner rather than as an incident detail. Track the share of incidents detected internally before external report as the practice's headline measure, since it is the one number that expresses whether monitoring is working, and report it alongside coverage rather than instead of it. Detection-as-code in this domain is fragile to retrieval and embedding schema changes, which silently break a rule that continues to report healthy.

C) Contribute detection content back — Contribute at least four substantive artifacts per year in deployable form: detection rules, behavioural baselining approaches, log schemas for AI telemetry and the test cases that verify a detection works. Shared detection content is scarce in AI security while the failure modes are largely common across organisations, which makes this among the highest-leverage contributions available, since a peer can deploy a rule and cannot deploy a description of one. Work through OWASP projects, MITRE ATLAS, sector information-sharing bodies and detection-engineering communities. Anonymise, review legally, coordinate disclosure where content reveals a provider weakness, and prioritise contributing the detections that most often produce actioned findings internally. Data detection content contributes well as extraction and canary-trigger rules, which are expensive to develop and widely applicable.

Outcome metrics

Oversight metrics — Detection changes deployed without review, by severity of the detection (detects: detection content changed directly in the console, which is untraceable, unrevertable and the most common way coverage is lost without a decision) · Detections whose tests have been failing for more than one week without action (detects: test failure normalised, which restores exactly the silent-failure mode the tests were introduced to remove) · Incidents detected externally rather than internally, with a coverage assessment completed (detects: external detection accepted without asking why internal detection missed it, which forfeits the single most informative signal monitoring produces)

Consumption metrics — Consumption-anomaly detections held as tested code with known-positive cases per per consumption detection · Time from consumption anomaly onset to internal detection per per retrieval query · Telemetry and detection platform cost per monitored asset per per monitored asset

Success criteria

• At least 85% of active detections are held in version control with an owner, documented intent and test cases.
• Detection tests run continuously against replayed or synthetic events, with failures alerting within the day.
• Every incident and adversarial finding is assessed for detection coverage within 5 business days, and uncovered cases produce a deployed detection within 10.
• Internal detection share is reported as the practice's headline measure alongside coverage.
• At least four accepted external contributions per year in deployable form.

Assessment questions

1. Are at least 85% of active detections held in version control with an owner, documented intent and test cases running continuously against replayed or synthetic events, so a detection broken by an upstream log change is caught by its own test? Evidence this in the Data domain against the corpus access log and the canary-trigger detection.
2. Is every incident and adversarial finding assessed for detection coverage within 5 business days, with uncovered cases producing a deployed detection and its tests within 10, and detections that existed but did not fire treated as defects with owners? For Data, the assessor should be shown the corpus access log and the canary-trigger detection.
3. Is internal detection share reported as the headline outcome measure alongside coverage, and does the organisation contribute at least four accepted detection artifacts per year in deployable rather than descriptive form? In Data, ask to see the corpus access log and the canary-trigger detection.

Common pitfalls

• ❌ Leaving detections in a console rather than version control, which makes changes untraceable and coverage loss undecided.
• ❌ Deploying detections without test cases, which leaves the silent-failure mode after an upstream log change entirely uncovered.
• ❌ Normalising failing detection tests, which restores exactly the problem the tests were introduced to remove.
• ❌ Accepting external detection of an incident without asking why internal detection missed it, forfeiting the most informative signal available.
• ❌ Contributing descriptions of detections rather than deployable rules, which peers read and cannot run.

Part V, Assessment workbook

The full questionnaires for this domain are in ../questionnaires/, one per practice, nine questions each. Work them in dependency order from Part III.

Scoring

Fully Mature 1.0 · Implemented 0.67 · Partial 0.33 · Not Implemented 0.0. A level is achieved when all three of its questions score 0.67 or better. Levels are derived, never self-assigned, and every claim is graded at its weakest evidence grade (E0–E3).

Generated by tools/render.py from the Data domain cell records.