Strategy & Metrics (SM)

Data Domain - HAIAMM v3.0

Practice Overview

Objective: Stand up an AI/HAI Data Assurance program that discovers, inventories, and strategically governs all data flowing into and out of AI/HAI systems, with shadow-data-in-AI prevention as the primary L1 outcome and a defensible risk-tier rubric as the primary L2 deliverable.

Description: The Data domain governs the data that AI/HAI systems consume and produce, training corpora, inference input streams, retrieval stores, prompt/completion log corpora, embedding stores, fine-tuning datasets, and evaluation/test sets. SM-Data establishes the program charter, an authoritative inventory of these data assets, and the practice-maturity metrics that prove the program is working. SM-Data L2 produces the risk-tier rubric every other Data-domain L2 practice depends on, and every downstream Data-domain practice (PC, TA, SR, SA, DR, IR, ST, EH, ML, IM) inherits its tier-calibration from the rubric authored here.

Context: Data moves into AI systems through many paths, a researcher uploads a fine-tuning dataset from a personal drive, an engineer wires a new retrieval store to a production RAG pipeline, prompt/completion logs from a customer-facing feature accumulate in an S3 bucket with no retention policy, and an ML team trains on a corpus that includes PII without privacy-officer sign-off. None of this is necessarily malicious, it is the normal pace of AI-enabled development operating faster than data governance. But it creates data-class exposure at inference, training-data leakage risk (TM, AGH), retrieval-source poisoning vectors (AGH), embedding-store inversion risk, and unmet EU AI Act Art. 10 data governance obligations. The AI/HAI Data Assurance program makes the data surface visible, attaches accountable ownership per asset, and ensures that data flowing to AI systems is known, classified, governed, and compliant.

Maturity Level 1

Objective: Stand up the AI/HAI Data Assurance program, build an inventory of data assets serving AI/HAI systems, and establish baseline metrics that prove shadow data in AI is decreasing

At this level, the organization makes its AI/HAI data surface visible, assigns accountability for each data asset archetype, and begins measuring the reduction of shadow data in AI, data flowing to AI systems without a known owner, classification label, or governance record.

Dependencies

• None, entry-point practice for the Data domain. SM-Data L1 precedes all other Data-domain L1s.
• Alignment (not a hard dependency): enterprise-wide data governance program (if one exists), DPO/CPO function, existing SM program at enterprise level or in an adjacent domain (Vendors, Software), so the AI/HAI data program connects to existing data catalogs and risk committees rather than forming a parallel stack.
• Supports / unblocks: PC-Data L1 (policies need the inventory and archetypes), TA-Data L1 (threat modeling needs the asset list), SR-Data L1 (requirements packs key on archetype), SA-Data L1 (reference patterns need the archetype list), IM-Data L1 (incident routing needs the owner and sponsor structure), ML-Data L1 (logging baseline needs the inventory).

Desired Outcomes

• Shadow data in AI is visible, attributed to a named owning team, and trending down quarter-over-quarter.
• A single AI/HAI data inventory is the authoritative source of truth across Security, Engineering, Data/ML, Privacy/Legal, and Platform.
• An accountable executive owns AI/HAI data risk; decision rights for classification, retention enforcement, cross-border flow approval, and intake of new data sources are unambiguous.
• Practice maturity is measurable from a small, automatable metric set rather than from activity counts.
• The program is positioned as an enabler, fast-track governance for Low-tier public data, full treatment for Critical-tier regulated training corpora, so teams work through it rather than around it.

Activities

A) Charter the AI/HAI Data Assurance program

Publish a short program charter that names the problem (shadow data in AI, ungoverned training corpora, inference inputs containing PII without consent basis, retrieval stores populated without classification review), defines scope, and assigns accountable ownership. The program does not require a new team, it requires a named owner, a cross-functional working group, and a clear intake gate for new data sources feeding AI systems.

Charter elements: - Problem statement, why data flowing into AI is a distinct governance category: data becomes training signal (poisoning, leakage), retrieval context (retrieval-poisoning, AGH), prompt input (injection vectors), or logged exhaust (privacy obligation); EU AI Act Art. 10 places data-governance duties on deployers; GDPR Arts. 5/6/9 require lawful basis before personal data reaches an inference endpoint. - In-scope data archetypes, the seven canonical Data-domain asset types: 1. Training corpus / training dataset, data used to pretrain or fine-tune models the org builds. 2. Inference input stream, prompts, queries, user inputs flowing into models at inference. 3. Retrieval store, vector databases, knowledge bases, RAG indexes. 4. Prompt/completion log corpus, logged interactions retained for evaluation, audit, or monitoring. 5. Embedding store, vector representations of text, images, or other content (may be invertible). 6. Fine-tuning dataset, curated data specifically prepared for fine-tuning, often proprietary or customer-origin. 7. Evaluation / test set, golden datasets used for regression, eval, or red-team exercises. - Executive sponsor, typically the CISO co-sponsored by the DPO / CPO and the Head of Data or Head of Engineering; co-signed by Privacy/Legal. - Working group, Security, Engineering/ML Platform, Data/Analytics, Privacy/Legal, Product, one application-architect reviewer. - Decision rights, who can approve a new data source feeding AI; who can block one; who handles exceptions; who approves cross-border data flows. - Success definition for year one, a numerical target for the L1 outcome metrics below.

B) Build the AI/HAI data inventory and discover shadow data in AI

Establish a single AI/HAI data inventory as the program's source of truth. Seed it from authoritative data signals, then actively discover shadow data using signals already available to platform and security teams.

Inventory fields (minimum): - Asset name, owning team, archetype (training corpus, inference input stream, retrieval store, prompt/completion log corpus, embedding store, fine-tuning dataset, evaluation/test set). - Data classification: regulated PII / PHI / PCI / source code / customer confidential (Critical); org confidential (High); internal (Medium); public (Low). - Lineage and provenance: known consented source, licensed corpus, scraped / unknown, or internally generated. - Volume and criticality: total size (tokens, records, bytes), production-load-bearing (yes/no), system-of-record or derivative. - Cross-border flows: countries of origin, countries where data transits or is processed; EU AI Act / GDPR Art. 44–49 transfer flag. - Use context: training vs. inference vs. eval (training elevates posture). - Decision-affecting use: data feeds an Art. 22 / EU AI Act Annex III system (yes/no). - Subject-access-rights exposure: data subjects whose data is present may invoke GDPR Arts. 15–21 (yes/no). - Retention policy: defined / enforced / undefined. - Approval status: Sanctioned / Provisional / Under review / Prohibited / Awaiting Intake. - Risk tier assignment (populated at L2, see SM-Data L2 Activity A). - Linked artifacts: TA threat snapshot, SR requirements-evidence map (REM), latest IR finding, ML logging-baseline status.

Discovery sources (use what platform and data teams already have): - Data catalogs, Atlan, Collibra, DataHub, Unity Catalog, AWS Glue Data Catalog: search for datasets tagged or named with "training," "embedding," "eval," "fine-tune," "inference," "rag," "vector." - Model registry lineage, MLflow, Weights & Biases, SageMaker Model Registry, Vertex AI: training-data lineage fields reveal which corpora were consumed. - ETL/ELT pipeline metadata, Airflow DAGs, dbt models, Fivetran connectors: identify pipelines whose destination is a training or retrieval store. - Object-store inventories, S3 / GCS / Azure Blob bucket listings filtered for buckets/paths matching training, embeddings, eval, logs, prompts. - Vector-store listings, Pinecone, Weaviate, Qdrant, Chroma, pgvector: namespaces and collections reveal retrieval stores in use. - Prompt/completion log volumes, CloudWatch, Datadog, BigQuery: services emitting prompt/completion events reveal active inference input streams and log corpora. - Classification scanners, Amazon Macie, BigID, Microsoft Purview: run against object stores and databases to surface regulated data flowing to AI. - Self-attestation, a short intake form publicized to engineering and data science teams; amnesty window for disclosing assets already in use.

C) Establish foundational metrics that measure practice maturity and shadow-data-in-AI reduction

Baseline and track a small set of outcome, process, and effectiveness metrics. Keep L1 metrics simple, automatable, and tied to the L1 outcome (shadow data reduction and inventory coverage).

Shadow-data-in-AI scoreboard (published quarterly to the executive sponsor): 1. AI/HAI data assets in inventory (total / sanctioned / provisional / prohibited / awaiting intake), broken out by archetype. 2. New data assets discovered this quarter and their intake status. 3. Shadow-data-in-AI ratio trend (last 4 quarters): data assets flowing to AI without a known owner or classification label. 4. AI Data Acceptable Use Policy (AUP) attestation coverage across engineers and data scientists handling AI data. 5. Top 5 unmitigated data risks (TA-flagged, classification-scanner-flagged, or external-advisory-flagged) with owners and remediation status.

Outcome Metrics (L1)

Process Metrics (leading)

• Discovery cadence, shadow-data-in-AI discovery sweeps (catalog + object-store + vector-store + pipeline + classification-scanner) run at least monthly.
• Intake SLA, new data-source intake triaged within 5 business days; provisional approval within 10 BD for Low-tier (public data, no regulated content).
• Inventory freshness, ≥80% of inventory records reviewed or updated in the last 90 days, tied to last-pipeline-run or last-catalog-update timestamp.
• Working-group cadence, at least monthly; minutes published.

Effectiveness Metrics (business value)

• Data-team cycle-time impact, time from "team requests a new data source for AI" to "provisional approval issued" should decrease as the program matures; the program is not a bottleneck.
• Reuse rate, % of AI/HAI data assets using a sanctioned retrieval or training pattern vs. a bespoke one; rising reuse indicates the program scales.
• Avoided-incident stories, documented cases where early discovery caught a regulated-data-in-AI exposure before production (PII in a fine-tuning dataset, customer-data in an unclassified retrieval store, prompt logs retained past policy limit).

Success Criteria

• Program charter published and sponsored by an accountable executive (CISO + DPO/CPO + Head of Data or Engineering).
• AI/HAI data inventory exists as a single source of truth with ≥90% coverage of discovered data assets within 12 months, broken out by archetype.
• Shadow-data-in-AI ratio baselined and trending down for two consecutive quarters.
• ≥95% of engineers and data scientists handling AI data have acknowledged the AI Data AUP.
• Quarterly shadow-data-in-AI scoreboard delivered to the executive sponsor with archetype-level breakdown.

Maturity Level 2

Objective: Risk-tier every AI/HAI data asset using the canonical rubric, calibrate the program's intensity per tier, and measure practice maturity and shadow-data reduction per tier, establishing the rubric every other Data-domain L2 practice depends on

At this level, the program stops treating every data asset the same. Risk tiers drive how deep intake goes, how often reviews happen, what classification controls are enforced, and what the sponsor sees on the scoreboard. A Critical-tier regulated fine-tuning dataset is not equivalent to a Low-tier public-domain evaluation corpus. Per §9.3 of the v3.0 framing, the rubric established here is the prerequisite for L2 at PC, TA, SR, SA, DR, IR, ST, EH, ML, and IM in the Data domain.

Dependencies

• SM-Data L1 (required): inventory, charter, working group, and baseline metrics are the substrate L2 tiers and calibrates.
• PC-Data L1 (required): the priority compliance map provides several tier dimensions (EU AI Act Art. 10 / Annex III scope, GDPR special-category data, sector-specific restrictions).
• TA-Data L1 (required): the threat library provides threat dimensions (retrieval-poisoning risk, embedding-inversion exposure, training-data-leakage surface, AGH/TM vectors through data).
• Supports / unblocks: PC-Data L2 (tier-driven policy depth), TA-Data L2 (per-asset deep threat models), SR-Data L2 (per-tier requirements packs), SA-Data L2 (tier-conditional reference patterns), DR/IR/ST/EH/ML/IM-Data L2 (all per-tier calibrated).

Desired Outcomes

• Every AI/HAI data asset in the inventory carries a risk-tier assignment tied to explicit, auditable dimensions, not reviewer judgment.
• Program intensity is visibly differentiated: Critical training corpora with regulated PII get encryption with HSM-rooted keys, full lineage, retention enforcement, DPIA coverage, and EU AI Act Art. 10 evidence; Low public eval sets get baseline documentation.
• The quarterly scoreboard splits by tier; the sponsor can see which tiers are healthy and which are drifting.
• Tier movements (an asset upgraded when it gains regulated content, cross-border flow, or training use; downgraded when scope shrinks) are tracked, rationale-logged, and sponsor-visible.
• Practice maturity is defensible per tier: "we are mature at Critical and still building at Medium" is a real, evidenced statement.

Activities

A) Define the AI/HAI data risk-tier rubric

Four tiers, Critical / High / Medium / Low, assigned from a small set of auditable dimensions specific to AI/HAI data assets. This rubric is the canonical source of truth; all downstream Data-domain L2 practices inherit it.

Data classification: - Regulated PII / PHI / PCI / source code / customer confidential → Critical or High (depending on volume and use). - Org confidential → High. - Internal → Medium. - Public → Low.

Lineage and provenance: - Unknown origin, scraped without explicit license or consent verification → elevate tier. - Known consented source with documented legal basis → no elevation. - Licensed corpus without data-subject consent (common in pre-training) → treat per applicable sector rules and GDPR Art. 5(1)(b) compatibility.

Volume and criticality: - System-of-record data or data production-load-bearing for a revenue-critical AI system → elevate. - Derivative or experimental data → neutral.

Cross-border flows: - Transfer of personal data to a third country triggers GDPR Art. 44–49 assessment → elevate to at least High; Critical if no adequacy decision or SCC in place. - Sector-specific cross-border restrictions (e.g., HIPAA PHI outside US, ITAR, financial sector localization) → Critical.

Use in training vs. inference vs. eval: - Data used in training or fine-tuning elevates posture vs. the same data used only at inference → training use is a Critical dimension. - Eval/test sets used in red-team exercises for a Critical-tier model → at least High.

Decision-affecting use: - Data feeds an EU AI Act Annex III high-risk system or a GDPR Art. 22 automated-decisioning system → Critical.

Subject-access-rights exposure: - Data subjects covered by GDPR Arts. 15–21 (right of access, erasure, portability, objection) are present → requires clear retention boundaries and deletion capability → elevate if retention control is absent.

Tier derivation is deterministic from the rubric inputs; human overrides are allowed but recorded with rationale and reviewed by the working group.

B) Calibrate program intensity per tier

Publish a tier-treatment matrix, what each tier gets from the Data-domain program. Every downstream practice inherits this calibration.

C) Per-tier scoreboard and governance

The L1 shadow-data-in-AI scoreboard becomes tier-aware at L2: - Inventory state reported by tier and by archetype. - Shadow-data-in-AI ratio per tier, a Critical-tier unclassified training corpus is a headline; a Low-tier one is a line item. - SLA adherence per tier (intake, IR, ML, IM) reported monthly. - Tier-movement log, assets that moved up (tighter treatment now applies) or down (with rationale) this quarter. - Quarterly executive review discusses tier-balance: is the program's effort matching the risk profile?

Outcome Metrics (L2)

Process Metrics (leading)

• Tier-rubric review cadence, reviewed every 2 quarters; changes change-logged.
• DPIA gate exercised for all Critical and applicable High assets; no Critical asset reaches production without a closed or accepted DPIA.
• Per-tier queue depth monitored; no tier's backlog exceeds a published threshold.
• Classification-scanner runs against Critical/High inventory assets at least monthly.

Effectiveness Metrics (business value)

• Effort allocation match, % of reviewer hours on Critical+High tiers vs. Medium+Low; should rise relative to L1.
• Compliance cost reduction, DPIA and regulatory-evidence preparation time per asset for auditors decreases as the program matures.
• Avoided-incident stories where tier-differentiation caught risk earlier (Critical-tier fine-tuning dataset caught with unclassified PII before training run; retrieval store classified as High flagged a cross-border transfer trigger before go-live).

Success Criteria

• Risk-tier rubric published and applied; tier assigned to 100% of inventory.
• Tier-treatment matrix published; downstream practices (PC, TA, SR, SA, DR, IR, ST, EH, ML, IM) calibrated to it.
• Per-tier shadow-data-in-AI ratio reported quarterly; Critical-tier unclassified or ungoverned data assets in production = 0.
• Per-tier SLA adherence ≥90% across practices.
• Tier-movement governance active, changes logged with rationale, reviewed by the sponsor.

Maturity Level 3

Objective: Automate inventory and tier maintenance from catalog, lineage, classifier, and pipeline telemetry; benchmark against external data-governance peers; and contribute to industry data-governance and AI-risk-management standards

At this level, the program is signal-driven rather than ticket-driven. Inventory and tiering update from data catalogs, lineage APIs, classification-scanner events, and pipeline-run telemetry. Human review is exception-based. The program benchmarks against CDMC, EDM Council maturity assessments, and sector ISACs with AI data-governance working groups, and contributes to DAMA, ISO/IEC 23894 AI risk management, NIST AI RMF Data chapter, and CSA AI Safety Initiative.

Dependencies

• SM-Data L2 (required): tiering and calibration must be settled before automation is trustworthy.
• ML-Data L2+ (required): telemetry signals (pipeline runs, classification-scanner events, catalog-metadata updates) need the monitoring pipeline behind them.
• EG-Data L2+ (required): data-handler literacy enables teams to self-attest inventory accurately.
• Supports / unblocks: the other 11 Data-domain practices can move to L3 automation patterns because SM now supplies automated inventory and tier data.

Desired Outcomes

• Inventory accuracy is measured in hours of latency, not months.
• Tier assignments adjust automatically when dimensional inputs change (new regulated data class flows into a retrieval store, a training corpus gains a cross-border transfer, a fine-tuning dataset gains an Annex III use context); humans intervene only on exceptions.
• External benchmarking is routine, the program sponsor can answer "how do we compare on data-governance maturity?" with specific deltas.
• The organization is a net contributor to AI data-governance standards, DAMA, EDM Council, ISO/IEC 23894, NIST AI RMF Data, and sector ISACs reference program outputs.

Activities

A) Continuous inventory and tier automation from catalog, lineage, classifier, and pipeline signals

• Inventory auto-updates from: data-catalog metadata events (new dataset registered, schema change, classification-label update), model-registry lineage events (new training-data source linked to a model version), ETL/ELT pipeline runs (new destination is a training or retrieval store), classification-scanner findings (new regulated data class detected in an existing asset), object-store inventory diffs (new buckets or paths matching AI data patterns), vector-store collection changes, prompt/completion log volume spikes (new asset emitting logs is a discovery signal), self-attestation and intake.
• Tier assignments are rule-based on the L2 rubric inputs; rule changes are versioned and replayable; tier changes auto-trigger downstream practice obligations (e.g., a Medium→Critical upgrade triggers DPIA gate, encryption upgrade, IR reconfiguration).
• Human curation handles: new archetypes, ambiguous classification-scanner findings, dimensional-input conflicts.
• A data-quality SLO is published: ≥99% of active AI/HAI data assets correctly tiered within 48 hours of a material change; ≥95% inventory completeness against discovery-source reconciliation.

B) External benchmarking

• Program metrics compared against peer benchmarks via:
• CDMC (Cloud Data Management Capabilities) maturity assessments.
• EDM Council Data Management Capability Assessment Model (DCAM) for AI data-governance components.
• DAMA DMBOK practitioner communities.
• ISO/IEC 23894 AI risk management working groups.
• Sector ISACs with AI data-governance tracks (FS-ISAC, H-ISAC, IT-ISAC).
• Formal peer roundtables (CISO/DPO communities, AI safety practitioner circles).
• A published "how we compare" brief refreshed semi-annually covers: inventory coverage, shadow-data-in-AI ratio, per-tier SLA adherence, automation level, classification accuracy, retention-enforcement rate, time from "new data source proposed" to "provisional approval issued."
• Benchmark deltas inform program investment and the next year's L2/L3 priorities.

C) Contribute to industry data-governance and AI-risk-management standards

• Contribute to:
• DAMA DMBOK and AI data-management community working groups.
• EDM Council AI Risk and Data Governance Principles.
• ISO/IEC 23894 AI risk-management standard (data-domain implementation guidance).
• NIST AI RMF Playbook Data chapter and successor editions.
• CSA AI Safety Initiative (AI data-governance controls matrix).
• OpenSSF AI (training-data supply-chain advisories, embedding-store security guidance).
• Sector ISACs where AI data-governance working groups accept practitioner input.
• Target: minimum 4 substantive contributions per year; quality over volume; every contribution anonymized and legally vetted.

Outcome Metrics (L3)

Process Metrics (leading)

• Automation health, signal-feed freshness and error rate monitored; on-call paged when a feed staleness threshold is exceeded.
• Benchmarking cadence honored (semi-annual brief published on schedule).
• Contribution pipeline always has ≥2 items in-flight (draft, in-review, or being prepared).
• Tier-rule change-log healthy, rule changes versioned, replayable, reviewed quarterly.

Effectiveness Metrics (business value)

• Sponsor decisions (budget, headcount, scope) citing benchmark data and tier-level metrics.
• Industry recognition, citations of the program's contributions, invitations to working groups, peer adoption of published data-governance reference patterns.
• DPO/privacy-team audit-preparation overhead trending down as the automated inventory supplies evidence on demand.
• Faster sanctioned data-source onboarding, time from "data team proposes a new training corpus" to "provisional approval issued" is industry-leading.

Success Criteria

• Inventory auto-update SLO published and met.
• Tier-assignment automation operational with published rules, replayable change-log, and exception-based human review.
• Semi-annual external-benchmarking brief published to the sponsor with ≥5 peer-comparable metrics.
• ≥4 substantive industry contributions per year, anonymized and cited.
• ROI narrative including external benchmarks delivered to exec/board at least annually.

Key Success Indicators

Level 1: - AI/HAI Data Assurance program charter published and sponsored by an accountable executive (CISO + DPO/CPO + Head of Data or Engineering), with a cross-functional working group (Security, Engineering/ML Platform, Data/Analytics, Privacy/Legal, Product). - AI/HAI data inventory exists as a single source of truth, covering all seven in-scope archetypes (training corpus, inference input stream, retrieval store, prompt/completion log corpus, embedding store, fine-tuning dataset, evaluation/test set). - Shadow data in AI actively discovered each month from data catalogs, model-registry lineage, ETL/ELT pipeline metadata, object-store inventories, vector-store listings, classification-scanner findings, and prompt/completion log volumes. - AI Data Acceptable Use Policy acknowledged by ≥95% of engineers and data scientists handling AI data. - Foundational metrics baselined: inventory coverage, shadow-data-in-AI ratio, AUP attestation, intake SLA; quarterly shadow-data-in-AI scoreboard delivered to the exec sponsor.

Level 2: - Risk-tier rubric published and applied, 100% of inventory carries a current tier from the seven auditable dimensions (data classification, lineage and provenance, volume and criticality, cross-border flows, use in training vs. inference vs. eval, decision-affecting use, subject-access-rights exposure). - Tier-treatment matrix published; downstream practices (PC, TA, SR, SA, DR, IR, ST, EH, ML, IM) calibrated to it. - Quarterly shadow-data-in-AI scoreboard reports per tier and per archetype; Critical-tier unclassified or ungoverned data assets in production = 0. - Per-tier SLA adherence ≥90% across program activities. - Tier-movement governance operating with logged rationale and sponsor review.

Level 3: - Inventory auto-update latency ≤48 hours for material changes; ≥80% of curation automated; ≥99% inventory completeness against discovery-source reconciliation. - Tier-assignment automation operates on a published, versioned rule set with exception-based human review; tier changes auto-trigger downstream practice obligations within 24 hours. - Semi-annual external-benchmarking brief published, citing ≥5 peer-comparable metrics from CDMC / EDM Council / DAMA / ISAC / ISO/IEC 23894. - ≥4 substantive anonymized industry contributions per year (DAMA, EDM Council, ISO/IEC 23894, NIST AI RMF, CSA AI Safety Initiative, OpenSSF AI, sector ISACs). - Executive/board ROI narrative refreshed at least annually with external benchmarks and documented avoided-loss examples.

Common Pitfalls

Level 1: - ❌ Inventory is seeded only from "data the ML team told us about", misses retrieval stores added by engineering, prompt/completion log corpora accumulating in object storage, and fine-tuning datasets uploaded by researchers from personal drives. - ❌ Treating data flowing through org-built AI services as a Software-domain concern only, data classification, lineage, and retention at the corpus/log/embedding level are Data-domain responsibilities; the two cross-reference but do not duplicate. - ❌ Program positioned as a blocker, intake SLA unpublished, data teams route around the program by using unreviewed data sources. - ❌ Executive sponsor is security-only; DPO/CPO and Head of Data are not co-owners, so the program lacks data-governance authority. - ❌ Metrics count activity (scans run, datasets tagged) instead of outcomes (shadow-data-in-AI ratio down, regulated-data exposure events trending down). - ❌ No amnesty window, data scientists hide ungoverned training datasets rather than surface them. - ❌ Archetype list too coarse ("AI data"), Critical training corpora with PII and Low public eval sets are conflated; the program cannot tier without re-inventorying.

Level 2: - ❌ Tier-rubric dimensions are subjective ("sensitive"), reviewers tier differently; auditors don't trust it; tier movements feel political. - ❌ Tier-treatment matrix published but not enforced, Critical training corpora lack DPIA coverage and HSM-rooted encryption because enforcement never happened. - ❌ Scoreboard reported in aggregate, hiding that Critical-tier shadow data is present because overall averages look acceptable. - ❌ Tier upgrades get resistance from data teams because they trigger DPIA gates and encryption requirements, no governance on tier-movement means assets stay under-tiered. - ❌ Downstream practices treat tier as advisory, DR/IR/ST don't differentiate scope by data tier, defeating the purpose of L2. - ❌ Rubric over-engineered, too many dimensions, tier derivation becomes an oracle ritual.

Level 3: - ❌ Automation runs without a data-quality SLO, signal-driven inventory silently drifts and privacy/data teams stop trusting it. - ❌ Benchmarking chooses peers that flatter the program (comparing a startup to startup benchmarks when operating at enterprise scale and regulatory exposure). - ❌ Industry "contributions" are presentations, not technical artifacts that land in DAMA/EDM/ISO/NIST/CSA working groups. - ❌ Automated tiering rules encode historical bias, audit of rule inputs never happens; under-tiering of training-data assets persists silently. - ❌ ROI narrative decouples from reality, external benchmarks cited but program's own metrics are stale. - ❌ Tier-change downstream triggers fire on every schema edit, data teams disable the signal-source rather than fix rule sensitivity.

Practice Maturity Questions

Level 1: 1. Is there a published AI/HAI Data Assurance program charter with a named executive sponsor (CISO + DPO/CPO + Head of Data or Engineering), a cross-functional working group, and clear decision rights for approval, block, exception, and cross-border-flow approval across all seven data-domain archetypes? 2. Does a single AI/HAI data inventory exist, seeded from data catalogs, model-registry lineage, ETL/ELT pipeline metadata, object-store inventories, vector-store listings, classification-scanner findings, and prompt/completion log volumes, covering all seven archetypes with ≥90% coverage of discovered assets within 12 months? 3. Are the L1 outcome metrics baselined and reported quarterly to the sponsor, inventory coverage, shadow-data-in-AI ratio (≤15% and trending down), AI Data AUP attestation (≥95% of engineers and data scientists), data assets with named owning team (100%), and known regulated-data-in-AI exposure events?

Level 2: 1. Is every AI/HAI data asset in the inventory assigned a risk tier based on the seven auditable dimensions, data classification, lineage and provenance, volume and criticality, cross-border flows, use in training vs. inference vs. eval, decision-affecting use, and subject-access-rights exposure, with a published tier-treatment matrix driving differential intensity? 2. Is there a published tier-treatment matrix driving differential controls across PC, TA, SR, SA, DR, IR, ST, EH, ML, IM, with ≥95% of Critical-tier data assets receiving full-scope treatment (HSM-rooted encryption, full lineage, DPIA closure, retention enforcement, EU AI Act Art. 10 evidence) in the last 12 months? 3. Does the quarterly shadow-data-in-AI scoreboard report per tier and per archetype (with Critical-tier unclassified or ungoverned data assets in production explicitly tracked at zero), and does tier-movement get logged and reviewed by the program sponsor?

Level 3: 1. Does inventory and tier assignment auto-update from live catalog, lineage, classifier, and pipeline telemetry with a published data-quality SLO, and is ≥80% of curation handled automatically with exception-based human review? 2. Do you publish a semi-annual external-benchmarking brief comparing the program against ≥5 peer-comparable metrics via CDMC / EDM Council / DAMA / sector ISACs / ISO/IEC 23894, and does it drive program investment decisions? 3. Does the program contribute ≥4 substantive, anonymized artifacts per year to AI data-governance standards (DAMA, EDM Council, ISO/IEC 23894, NIST AI RMF, CSA AI Safety Initiative, OpenSSF AI, sector ISACs), and does the exec/board ROI narrative cite external benchmarks?

Document Version: HAIAMM v3.0 Practice: Strategy & Metrics (SM) Domain: Data Last Updated: 2026-05-13 Author: Verifhai