Design Review (DR) - Data Domain

HAIAMM Assessment Questionnaire v3.5

Generated from cells/DR-Data-L*.cell.json. Do not edit by hand. Canonical model: HAIAMM-v3.5-Framing.md · Handbook: HAIAMM-v3.5-Handbook.md.

---

Practice: Design Review (DR) Domain: Data Business Function: Verification Purpose: Establish a two-lane design review for data AI changes with a published checklist and service level, so review is predictable enough that teams route through it. Scoring Model: Evidence + Outcome Metrics (see Scoring Methodology below)

---

Instructions

• Answer from current, implemented practice, not plans or aspirations.
• Each question has two components: Evidence (what you did) and Outcome Metrics (how well it worked).
• Levels are derived, never self-assigned. Record evidence and metric values; the level falls out of the scoring below.
• Silence is not a pass. A metric you cannot produce is left blank and counts as not met. Do not estimate.
• Answer progressively. Complete all Level 1 questions before Level 2.

Evidence grade

Record how each answer is known, not only what is claimed (v3.5 framing §6). A level claim is graded at its weakest field.

Grade	Meaning
E0	Asserted by a practitioner, unevidenced
E1	Asserted, with a link to a named artifact
E2	Emitted by a system as telemetry
E3	Machine-attested and independently checkable

---

Scoring Methodology

Tier	Score	Criteria
Fully Mature	1.0	Evidence complete + ≥3 outcome metrics meet targets
Implemented	0.67	Evidence complete + 2 outcome metrics meet targets
Partial	0.33	Evidence partially complete + <2 outcome metrics meet targets
Not Implemented	0.0	No substantive evidence of practice

Practice maturity level achieved = the highest level where all 3 questions score ≥ 0.67.

---

Maturity Level 1, Foundational

Objective: Establish a two-lane design review for data AI changes with a published checklist and service level, so review is predictable enough that teams route through it.

Prerequisite cells: SM-Data-L1, TA-Data-L1, SR-Data-L1, SA-Data-L1

Question L1.1: Establish two review lanes with published service levels

Q1.1: Do two published review lanes operate with stated routing rules and service levels of 5 and 15 business days, with adherence measured and reported per lane and persistent breaches treated as a capacity problem? Evidence this in the Data domain against a full-lane decision on a cross-border corpus flow.

Evidence Required:

• [ ] Define a fast lane and a full lane, and publish the routing rule and the service level for each
• [ ] The fast lane handles designs adopting a sanctioned reference pattern without material deviation, cleared against a standard checklist within 5 business days
• [ ] The full lane handles novel architectures, pattern deviations and high-risk designs, with a named architect reviewer and a decision within 15 business days
• [ ] Publishing both numbers is the practice's single most important act: a review of unknown duration is a review teams plan around, and they do so quietly
• [ ] Measure and report adherence per lane, and treat a persistent breach as a capacity problem to be resourced rather than a communication problem to be explained

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L1.2: Publish the checklist and make decisions written

Q1.2: Is the review checklist published so teams self-assess before submitting, covering classification recorded before ingest, lawful basis stated, lineage capture through every transformation, retention righ, and does every review produce a written decision with named reviewer, rationale and tracked conditions?

Evidence Required:

• [ ] Publish the review checklist so teams can self-assess before submitting
• [ ] In this domain the checklist covers classification recorded before ingest, lawful basis stated, lineage capture through every transformation, retention right-sized to the longest applicable obligation, erasure propagation path to embeddings and derived artifacts, per-request authorization on retrieval, and redaction at log write
• [ ] Every review produces a written decision of approve, approve with conditions, or refuse, naming the reviewer and stating the rationale
• [ ] Conditions carry an owner and a date and are tracked to closure, because an approval with untracked conditions is an approval, and the conditions are the part that mattered
• [ ] Record the checklist items that most often fail, and feed that back to the education and architecture practices as the clearest available evidence of where design understanding is weak

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L1.3: Define review triggers and detect what bypassed review

Q1.3: Do review triggers fire from inventory, deployment and change signals rather than team initiative, and does monthly bypass reconciliation raise every unreviewed production asset as a finding with an owner? In Data, ask to see a full-lane decision on a cross-border corpus flow.

Evidence Required:

• [ ] Define what triggers a review rather than relying on teams to request one
• [ ] In this domain the triggers are a new data class entering a corpus, a new consuming model, a cross-border flow change, or a retention or sampling change, plus any new asset entering the inventory
• [ ] Wire the triggers to the signals that already exist, being inventory changes, deployment events and change records, so that a review obligation is raised by the system rather than remembered by a person
• [ ] Then detect bypass: reconcile assets reaching production against review decisions, and treat a production asset with no review record as a finding with an owner
• [ ] Bypass detection is what converts design review from a service teams may use into a gate whose absence is visible, and without it the practice measures only the designs it happened to see

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Level 1 Outcome Metrics

Metric	Baseline	Current	Target	Met?	Notes
AI design changes receiving a written decision within the applicable lane service level (fast lane 5 business days, full lane 15)	___	___	≥90% overall, reported per lane	☐
Production AI assets with no design review record	___	___	0, with every case raised as a finding with an owner	☐
Review decisions carrying a written rationale and named reviewer	___	___	100%	☐
Approval conditions carrying an owner and a date, tracked to closure	___	___	100%, with none past due	☐
Review triggers fired by system signal rather than by team request	___	___	≥80% of reviews trigger-initiated	☐

Where each number is read from:

• AI design changes receiving a written decision within the applicable lane service level (fast lane 5 business days, full lane 15) — Design review decision log with timestamps
• Production AI assets with no design review record — Asset inventory reconciled against the review decision log
• Review decisions carrying a written rationale and named reviewer — Design review decision log
• Approval conditions carrying an owner and a date, tracked to closure — Condition tracker joined to the review log
• Review triggers fired by system signal rather than by team request — Review intake source field

Level 1 30-second check (E0)

• ☐ Yes ☐ No — Is there a security review before an AI design gets built?
• ☐ Yes ☐ No — Do teams know how long a design review will take?
• ☐ Yes ☐ No — Would you notice an AI asset that reached production without review?

---

Maturity Level 2, Comprehensive

Objective: Calibrate data design review depth and reviewer authority to asset tier, and make review decisions traceable to the threats and requirements they turned on.

Prerequisite cells: DR-Data-L1, SM-Data-L2, TA-Data-L2, SR-Data-L2, SA-Data-L2

Question L2.1: Derive review depth and reviewer authority from tier

Q2.1: Is review lane and reviewer seniority derived automatically from asset tier rather than judged at intake under queue pressure, with at least 95% tier-derived routing and a published per-tier reviewer-load ceiling? Evidence this in the Data domain against a full-lane decision on a cross-border corpus flow.

Evidence Required:

• [ ] Publish what each tier receives: at Critical, a full-lane review with a named architect, mandatory threat-model and requirements-evidence review, and a decision recorded where the sponsor can see it
• [ ] At High, full lane where a deviation exists and fast lane otherwise
• [ ] At Low, a checklist pass with sampling
• [ ] Derive the lane from the inventory tier automatically rather than from reviewer judgment at intake, because intake routing under queue pressure drifts toward the fast lane and does so without anyone deciding to lower the bar
• [ ] Publish a reviewer-load ceiling per tier, and when it is breached add reviewers rather than shortening reviews, since the alternative is a gate that approves at the rate the queue demands
• [ ] In this domain the designs that genuinely need architect judgment are new corpus types and cross-border flows, while an additional consumer of an already-governed store usually does not

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L2.2: Make decisions traceable to threats and requirements

Q2.2: Do at least 90% of full-lane decisions cite the specific TA threat identifiers and SR requirement identifiers that drove each condition or refusal, making the reasoning reconstructable by a second reviewer and an auditor? For Data, the assessor should be shown a full-lane decision on a cross-border corpus flow.

Evidence Required:

• [ ] Require every full-lane decision to cite the specific threat identifiers from the TA library and requirement identifiers from the SR pack that it assessed, and to state which ones drove each condition or refusal
• [ ] This converts review from personal judgment into reconstructable reasoning, which matters for three reasons: a second reviewer can reach the same conclusion, an auditor can follow it, and the practice can tell which threats and requirements are actually doing work in review
• [ ] Report the threats and requirements most frequently driving conditions, since that is direct evidence of where designs are systematically weak and belongs in the architecture and education practices rather than only in the review log
• [ ] Automated design criteria read the catalogue entry, declared lawful basis, retention setting and lineage plan, which covers the conformance questions and leaves provenance judgment to a person

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L2.3: Govern deviations and make Critical decisions visible

Q2.3: Do deviation approvals carry a compensating control, tier-appropriate approval authority and an expiry with none past expiry, and are Critical-tier deviations reported to the sponsor alongside approvals? In Data, ask to see a full-lane decision on a cross-border corpus flow.

Evidence Required:

• [ ] Operate deviation approval as a distinct decision with its own record: what pattern or requirement is not met, why, what compensating control applies, who approved it at an authority proportionate to tier, and when it expires
• [ ] Review the deviation register monthly and report Critical-tier deviations to the sponsor alongside the approvals, because a review practice that reports only what it approved and not what it excused presents a misleading picture of the estate
• [ ] Track deviations later implicated in incidents as a distinct measure, which is the only honest feedback on whether deviation judgment is sound rather than merely procedurally correct
• [ ] Forward tracing in this domain most often implicates approvals that accepted a retention period without checking the longest applicable obligation

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Level 2 Outcome Metrics

Metric	Baseline	Current	Target	Met?	Notes
Reviews routed to a lane derived from asset tier rather than reviewer judgment at intake	___	___	≥95% tier-derived routing	☐
Critical-tier designs reviewed full-lane with a named architect	___	___	100%	☐
Full-lane decisions citing specific threat and requirement identifiers	___	___	≥90%	☐
Deviation approvals carrying compensating control, tier-appropriate authority and expiry	___	___	100%, with none past expiry	☐
Reviewer load against the published per-tier ceiling	___	___	no reviewer above ceiling for two consecutive weeks	☐
Per-tier service level adherence	___	___	≥90% per tier	☐

Where each number is read from:

• Reviews routed to a lane derived from asset tier rather than reviewer judgment at intake — Review intake routing source field
• Critical-tier designs reviewed full-lane with a named architect — Review log joined to inventory tier
• Full-lane decisions citing specific threat and requirement identifiers — Review decision records
• Deviation approvals carrying compensating control, tier-appropriate authority and expiry — Deviation register
• Reviewer load against the published per-tier ceiling — Review assignment log
• Per-tier service level adherence — Review log with timestamps by tier

Level 2 30-second check (E0)

• ☐ Yes ☐ No — Does a high-risk AI design get a more senior reviewer than a low-risk one?
• ☐ Yes ☐ No — Could someone reconstruct why a design was approved with conditions?
• ☐ Yes ☐ No — Is there a cap on how many designs one reviewer handles?

---

Maturity Level 3, Industry-Leading

Objective: Shift data design review left into the pipeline, reserve human review for genuine novelty, and measure whether review decisions were right rather than only timely.

Prerequisite cells: DR-Data-L2, SA-Data-L3, IM-Data-L2

Question L3.1: Automate the machine-checkable criteria

Q3.1: Are at least 75% of machine-checkable review criteria evaluated automatically with the non-automatable remainder listed explicitly, and do conformant Medium- and Low-tier designs clear without human review while sampling and a human veto are retained? Evidence this in the Data domain against a full-lane decision on a cross-border corpus flow.

Evidence Required:

• [ ] Convert every review criterion that can be evaluated from an artifact into an automated check running against the design representation, whether that is infrastructure-as-code, a configuration manifest, a service definition or a system card
• [ ] A design meeting all automated criteria and adopting a sanctioned pattern clears without human review at Medium and Low tier, and arrives at Critical and High review pre-verified so the reviewer spends attention on what is novel rather than on confirming what is conformant
• [ ] Publish the list of criteria that cannot be automated and review it quarterly for reduction, because that list is the honest boundary of the automation
• [ ] Keep a human veto and sampling at every tier, since a wholly automated gate encodes yesterday's criteria permanently and nothing in the system notices
• [ ] In this domain the designs that genuinely need architect judgment are new corpus types and cross-border flows, while an additional consumer of an already-governed store usually does not
• [ ] Novelty in this domain means a new corpus type or an unfamiliar provenance situation, not another consumer of a governed store

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L3.2: Measure decision quality, not throughput

Q3.2: Is decision quality measured by tracing approvals forward to incidents, implementation findings and test failures, reported to the sponsor alongside throughput and explicitly rather than throughput as the outcome measure? For Data, the assessor should be shown a full-lane decision on a cross-border corpus flow.

Evidence Required:

• [ ] Trace review decisions forward
• [ ] For every approval, record whether the asset subsequently generated an incident, an implementation-review finding or a security-test failure attributable to something the review should have caught
• [ ] Report false-negative rate, being approvals later implicated, and investigate false-positive pressure, being conditions imposed that evidence later showed unnecessary, since a gate that over-conditions trains teams to argue rather than comply
• [ ] Throughput and service-level adherence remain as health metrics but are explicitly not the practice's outcome measure, because a review function optimising for turnaround will achieve it by approving, and the metric will look excellent throughout
• [ ] Publish decision-quality measures to the sponsor alongside throughput
• [ ] Automated design criteria read the catalogue entry, declared lawful basis, retention setting and lineage plan, which covers the conformance questions and leaves provenance judgment to a person

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L3.3: Calibrate reviewers and concentrate attention on novelty

Q3.3: Does blind double-review run at least quarterly on Critical-tier designs with disagreement at or below 15% used to refine criteria rather than rank reviewers, and is at least 60% of reviewer time spent on genuinely novel designs? In Data, ask to see a full-lane decision on a cross-border corpus flow.

Evidence Required:

• [ ] Run blind double-review on a sample of Critical-tier designs at least quarterly and report the disagreement rate, because a gate whose outcome depends on which architect drew the review is a lottery rather than a control, and disagreement is invisible without deliberately measuring it
• [ ] Use the disagreement cases to refine criteria rather than to rank reviewers
• [ ] Reserve human review explicitly for novelty: designs using an unestablished pattern, a new archetype, a new model capability class or an unfamiliar threat surface, and route everything else to automated verification with sampling
• [ ] Track the share of reviewer hours spent on novel designs, which should rise steadily and is the clearest measure that the automation is doing its job
• [ ] Forward tracing in this domain most often implicates approvals that accepted a retention period without checking the longest applicable obligation

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Level 3 Outcome Metrics

Metric	Baseline	Current	Target	Met?	Notes
Machine-checkable review criteria evaluated automatically	___	___	≥75%, with the non-automatable remainder listed explicitly	☐
Medium- and Low-tier designs clearing without human review while meeting all automated criteria	___	___	≥70%, with sampling maintained	☐
Approvals later implicated in an incident, finding or test failure the review should have caught	___	___	tracked and falling year over year, reported to the sponsor	☐
Reviewer disagreement rate on blind double-reviewed Critical-tier designs	___	___	≤15%, with disagreements used to refine criteria	☐
Share of reviewer hours spent on genuinely novel designs	___	___	≥60% and rising	☐

Where each number is read from:

• Machine-checkable review criteria evaluated automatically — Automated design check registry
• Medium- and Low-tier designs clearing without human review while meeting all automated criteria — Review pipeline telemetry
• Approvals later implicated in an incident, finding or test failure the review should have caught — Review decisions joined to incident, IR and ST records
• Reviewer disagreement rate on blind double-reviewed Critical-tier designs — Double-review calibration records
• Share of reviewer hours spent on genuinely novel designs — Review time recording by design novelty classification

Level 3 30-second check (E0)

• ☐ Yes ☐ No — Does a conformant AI design clear review without a human reading it?
• ☐ Yes ☐ No — Do you know whether designs you approved later caused incidents?
• ☐ Yes ☐ No — Would two of your architects reach the same decision on the same design?

---

Summary Scorecard

Level	Q1	Q2	Q3	Level score	Weakest evidence grade	Achieved?
L1	___	___	___	___	___	☐
L2	___	___	___	___	___	☐
L3	___	___	___	___	___	☐

Design Review (DR) maturity in the Data domain: ____________

---

Document Version: HAIAMM v3.5 (generated)