Design Review (DR)

Vendors Domain - HAIAMM v3.0

Practice Overview

Objective: Operate a lightweight design checkpoint between intake approval and production rollout for every AI vendor integration, confirming the team picked a reference pattern, covered the SR requirements, and accepted only the residual risks the program can live with.

Description: DR-Vendors is the single moment where architecture (SA), requirements (SR), and threats (TA) meet an actual planned integration. At L1 the review is deliberately small: a structured design checkpoint with a named reviewer, a standard checklist keyed to the archetype, and a written decision (approve / approve-with-conditions / send back). It sits before implementation begins, catching issues when they cost hours to fix, not weeks.

Context: Without a design checkpoint, AI vendor integrations get discovered already in production. The pattern is skipped, the logging omitted, the permission boundary too broad, and retrofitting those after launch is expensive and visible. L1 DR-Vendors puts a small, predictable gate in front of rollout, timeboxed so it doesn't slow teams down.

Maturity Level 1

Objective: Run a standard design checkpoint per AI vendor integration before production, producing a written decision with traceability to SR, SA, and TA

At this level, design review is a habit, not a policy on paper, every AI vendor rollout over a small risk threshold gets a checkpoint, and the checkpoint outputs a decision artifact linked to the inventory record.

Dependencies

• SA-Vendors L1 (required): the reference patterns are what the checkpoint reviews the team's design against.
• SR-Vendors L1 (required): the requirements pack defines the bar.
• TA-Vendors L1 (required): the threat snapshot names what the design must defend against.
• EG-Vendors L1 (required): reviewers are trained on AI-vendor archetypes and TTPs.
• Supports / unblocks: IR-Vendors L1 (implementation reviews check against the approved design), ST-Vendors L1 (tests target the approved architecture), IM-Vendors L1 (incident triage references design assumptions).

Desired Outcomes

• Every AI vendor rollout above the triage threshold goes through a design checkpoint before production.
• Checkpoint decisions are written, versioned, and visible, not tribal knowledge.
• Design deviations from reference patterns are approved or rejected explicitly, with named reviewer and rationale.
• Recurring deviation themes feed back into SA (pattern updates) and SR (pack updates), the review generates organizational learning, not just per-ticket decisions.
• Review is timeboxed: teams know whether to expect a 30-minute async check or a 60-minute architect review based on risk tier.

Activities

A) Publish the AI Vendor Integration Design Checklist

One checklist per archetype (derived from the SA reference pattern). Each item is a yes/no with an evidence pointer.

Common spine across archetypes: - Pattern adherence, using the reference pattern, or documented deviation with rationale. - Data boundary, which data classes cross to the vendor; DLP/proxy inspection points defined. - Identity, SSO-backed human access; service-principal model for machine access; secret management. - Logging, prompt/completion (or equivalent) logs captured; human-oversight evidence trail present; retention meets policy. - Failure modes, what happens on vendor outage, model change, rate limit; fallback or kill-switch defined. - Permissions (agent archetype), tool allowlist, per-tool scope, human-in-the-loop gates for destructive or external actions. - Disclosure, Art. 50 disclosure where AI interaction is user-visible; customer-communication plan if regulated. - Residual risk, explicit list of residual risks, compensating controls, owners, and expiry.

B) Triage and route reviews by risk tier

Not every integration needs the same depth. L1 uses a two-lane model aligned to the risk tier assigned at intake: - Fast-lane (Low / Medium tier): async checklist review by the designated reviewer; target SLA ≤2 business days. - Full-lane (High / Critical tier): 45–60 minute architect review with the team walking the reference pattern; target SLA ≤5 business days. - Trigger for full-lane regardless of tier: deviation from the reference pattern, agent archetype, regulated-data involvement, external customer exposure.

Outputs for both lanes: one structured decision record, approve / approve-with-conditions (list) / send back (reasons), stored against the integration record.

C) Close the loop with SA, SR, and IM

Design review is a learning surface for the program: - Pattern update trigger, three deviations in the same direction for the same archetype auto-queue a pattern review with SA ownership. - Pack update trigger, a requirement repeatedly waived with a compensating control auto-queues an SR pack-revision review. - Incident-feedback loop, every IM-Vendors incident tagged to an AI vendor triggers a re-look at the design checkpoint that approved it: was the issue visible? what would catch it earlier?

Outcome Metrics (L1)

Process Metrics (leading)

• Reviewer population staffed and trained; named lead reviewer per archetype.
• Fast-lane vs. full-lane ratio monitored, a drift toward all-fast may indicate under-review; toward all-full may indicate over-review.
• Pattern-deviation rate tracked, feeds the SA pattern-update trigger.

Effectiveness Metrics (business value)

• Issues caught at design (vs. later stages), trend measured over quarters.
• Cycle-time impact, design review adds a small, predictable window, not an open-ended one; teams can plan.
• SA/SR update volume driven by DR feedback, the practice generates learning, not paperwork.

Success Criteria

• Design checklist per archetype published and versioned.
• Two-lane review model operational with published SLAs.
• ≥95% of AI vendor integrations going to production in the last 90 days have a completed checkpoint record.
• Pattern-update and pack-update triggers wired to SA and SR.
• Named reviewer population trained and active.

Maturity Level 2

Objective: Move design reviews from checklist to scenario-based walkthroughs; include vendor participation for Critical-tier; detect design drift between reviews

At this level, design review becomes a conversation about specific threat scenarios rather than a form-filling exercise. Critical-tier vendors are invited into joint design discussions. Drift between the approved design and the live integration is detected and re-routed to review.

Dependencies

• DR-Vendors L1 (required): design checkpoint, per-archetype checklists, two-lane review model.
• TA-Vendors L2 (required): per-vendor threat models power scenarios.
• SA-Vendors L2 (required): extended patterns referenced during review.

Desired Outcomes

• Critical-tier design reviews cover specific abuse scenarios, not only checklist conformance.
• Vendor architects participate in Critical-tier reviews when appropriate, joint ownership of mitigations.
• Drift from approved designs is detected automatically; material drift routes back to DR.

Activities

A) Scenario-based reviews for Critical/High tiers

• Reviewer walks 3–5 specific threat scenarios against the proposed design.
• Scenarios sourced from TA library + anonymized industry incidents.
• Review decision tied explicitly to how the design handles each scenario.

B) Vendor participation in Critical-tier reviews

• Pre-established comms channels with Critical-tier vendor security / architect teams.
• Joint design discussion for novel integrations; vendor-side FRIA cooperation for EU AI Act high-risk uses.
• Template NDA and sharing-agreement ready; Legal pre-approval.

C) Design-drift detection

• Compare live integration vs. approved design quarterly for Critical; annually for High.
• Material drift (pattern change, new tools, new data classes, new regions) auto-routes back to DR.

Outcome Metrics (L2)

Process Metrics (leading)

• Scenario library from TA refreshed quarterly.
• Vendor-participation program coverage growing.
• Drift-detection tooling health monitored.

Effectiveness Metrics (business value)

• Fewer surprise findings at IR (drift caught pre-IR).
• Critical-tier vendor relationships deepen, improving incident response.

Success Criteria

• 100% Critical reviews scenario-based.
• ≥70% Critical-tier vendors participating in joint reviews.
• Drift detection operating; 100% material drifts return to DR.

Maturity Level 3

Objective: Continuous design attestation via automated pattern-compliance telemetry, and contribute architecture-review patterns to industry

At this level, integrations don't get reviewed periodically, they attest continuously. Pattern-compliance is a telemetry signal; reviewers handle exceptions. Review patterns, rubrics, and vendor-cooperation frameworks are contributed back to industry.

Dependencies

• DR-Vendors L2 (required): scenario reviews + vendor participation + drift detection.
• SA-Vendors L3 (required): externalized patterns supply the attestation frame.

Desired Outcomes

• Critical-tier integrations' design posture is readable in near-real-time.
• Review patterns and vendor-cooperation frameworks cited by industry.
• Review backlog shrinks to exception work, routine attestation is automated.

Activities

A) Continuous design attestation

• Integrations declare compliance monthly via automated pattern-compliance scans and configuration audits.
• Deviations open a DR-exception ticket automatically.

B) Contribute review patterns to industry

• Publish review rubrics, scenario templates, vendor-cooperation frameworks to OpenSSF AI, CSA, Shared Assessments.

C) Pattern evolution driven by external + internal data

• External incident patterns (from ISACs, MITRE ATLAS) + internal IM incidents + ML telemetry drive design-pattern updates quarterly.

Outcome Metrics (L3)

Process Metrics (leading)

• Attestation-scan health monitored, % Critical integrations producing a fresh attestation signal in the last 7 days.
• Industry contribution pipeline, ≥1 artifact in-flight at any time (draft, in-review, published).
• External-signal ingestion cadence, ISAC and MITRE ATLAS feeds processed monthly into the pattern-update queue.
• Exception-queue freshness, DR-exception tickets opened by attestation deviations triaged within 3 business days.

Effectiveness Metrics (business value)

• Reviewer-hours per integration drop quarter-over-quarter as attestation absorbs routine work.
• External recognition, citations or adoption of published review patterns by peer organizations, regulators, or standards bodies.
• Critical-incident MTTR shortened by vendor-cooperation channels seeded at L2 and matured at L3.
• Talent signal, reviewer-track hires cite the program's externalized rubrics as a reason for joining.

Success Criteria

• Monthly auto-attestation for ≥90% of Critical integrations.
• ≥2 externally contributed review artifacts per year with documented adoption.
• Review backlog age inside target (≤7 days).
• Quarterly design-pattern evolution loop traceable to external (ISAC, ATLAS) and internal (IM, ML) signal sources.

Key Success Indicators

Level 1: - Per-archetype AI Vendor Integration Design Checklist published, versioned, and traceable to the applicable SA reference pattern, SR requirement pack, and TA threat snapshot. - Two-lane review model operational (fast-lane ≤2 BD, full-lane ≤5 BD) with named lead reviewers per archetype. - ≥95% of AI vendor integrations reaching production in the last 90 days carry a completed design-checkpoint decision record. - Pattern-update and pack-update triggers wired so recurring deviations and waived requirements feed SA and SR; every AI-vendor IM incident re-examines the approving checkpoint.

Level 2: - 100% of Critical-tier reviews are scenario walkthroughs sourced from TA library and anonymized industry incidents, with the decision tied to how the design handles each scenario. - ≥70% of eligible Critical-tier vendors participate in joint design reviews under pre-established NDA and sharing-agreement. - Drift detection runs quarterly for Critical and annually for High; 100% material drifts return to DR. - IR-stage surprises measurably down, drift caught before implementation review, not after.

Level 3: - ≥90% of Critical AI-vendor integrations produce monthly automated pattern-compliance and configuration-audit signals. - ≥2 externally contributed review artifacts per year (rubrics, scenario templates, vendor-cooperation frameworks) with documented adoption. - Review backlog age ≤7 days; routine attestation work has absorbed the pre-L3 review volume. - Pattern evolution driven by external (ISAC, MITRE ATLAS) and internal (IM, ML) signals on a quarterly cadence with a traceable change log.

Common Pitfalls

Level 1: - ❌ Design review treated as a gate to clear, not a learning surface, decisions get filed and nobody updates SA or SR from them. - ❌ Fast-lane becomes the default for everything, high-risk integrations slip through with a checklist instead of an architect conversation. - ❌ Reviewers anchored on classic SaaS patterns miss AI-specific concerns (training-data posture, tool-scope breadth, EU AI Act deployer duties). - ❌ Approve-with-conditions becomes a permanent state, action items age without named owners or expiry dates. - ❌ Reviews scheduled after implementation has begun, the checkpoint loses leverage because rework cost is already sunk.

Level 2: - ❌ "Scenario-based" review becomes another checklist disguised as a walkthrough, same questions, different format. - ❌ Vendor participation invited but never realized because NDA / Legal isn't pre-staged; the program reverts to L1 in practice. - ❌ Drift detection runs but findings dead-end, material drift opens no DR ticket, leaving the approved design as fiction. - ❌ Scenario library not refreshed quarterly, drifting from current TA threat models and industry incident reality. - ❌ Joint reviews dominated by vendor narrative; reviewer doesn't independently verify against the SR pack.

Level 3: - ❌ Attestation signal becomes telemetry theater, green dots without meaningful integrity checks behind them. - ❌ Externally contributed patterns evolve faster than internal practice, what's published doesn't match what reviewers actually do. - ❌ Exception queue overwhelms the team because attestation thresholds are too tight; reviewers chase low-signal alerts. - ❌ Industry contributions become marketing artifacts disconnected from internal IM and ML feedback loops.

Practice Maturity Questions

Level 1: 1. Is there a per-archetype AI Vendor Integration Design Checklist, published, versioned, and traceable to its SA reference pattern, SR requirement pack, and TA threat snapshot, with named lead reviewers per archetype? 2. Do ≥95% of AI vendor integrations going to production in the last 90 days have a completed design-checkpoint decision record (approve / approve-with-conditions / send-back) before production, with target SLAs of ≤2 BD fast-lane and ≤5 BD full-lane? 3. Are recurring deviations and repeatedly-waived requirements automatically queueing SA pattern-update and SR pack-update reviews, with every AI-vendor IM incident re-examining the design checkpoint that approved it?

Level 2: 1. Are 100% of Critical-tier design reviews conducted as scenario-based walkthroughs sourced from the TA library and anonymized industry incidents, with the decision tied explicitly to how the design handles each scenario? 2. Are ≥70% of eligible Critical-tier vendor architects participating in joint design reviews under pre-established NDA and sharing-agreement, with vendor-side FRIA cooperation available for EU AI Act high-risk uses? 3. Is design-drift detection operating quarterly for Critical and annually for High, with 100% material drifts (pattern change, new tools, new data classes, new regions) automatically re-routing back to DR?

Level 3: 1. Are ≥90% of Critical AI-vendor integrations producing monthly automated pattern-compliance and configuration-audit signals, with deviations auto-opening DR-exception tickets triaged within 3 business days? 2. Has the program contributed ≥2 substantive review artifacts per year (rubrics, scenario templates, vendor-cooperation frameworks) to industry bodies (OpenSSF AI, CSA, Shared Assessments) with documented adoption? 3. Are external incident patterns (ISAC, MITRE ATLAS), internal IM incidents, and ML telemetry feeding a quarterly design-pattern evolution cadence with a traceable change log?

Document Version: HAIAMM v3.0 Practice: Design Review (DR) Domain: Vendors Last Updated: 2026-05-12 Author: Verifhai