Design Review (DR) - Vendors Assessment

Assessment questionnaire for measuring maturity. Answer each question honestly based on current, implemented practices.

v3.0 framing: The canonical source-of-truth for Design Review (DR) in the Vendors domain is ../practices/DR-Vendors-OnePager.md. This questionnaire is authored against that one-pager. Canonical subject and through-lines: ../HAIAMM-v3.0-Framing.md §8.

Design Review (DR) - Vendors Domain

HAIAMM Assessment Questionnaire v3.0

Practice: Design Review (DR) Domain: Vendors Purpose: Assess organizational maturity in operating a design checkpoint before every AI vendor integration reaches production, confirming pattern adherence, data boundary, identity, logging, failure modes, agent permissions, Art. 50 disclosure, and residual risk are covered before implementation begins Scoring Model: Evidence + Outcome Metrics (see Scoring Methodology below)

Instructions

Answer each question honestly based on current, implemented practices (not plans or aspirations)
Each question has two components: Evidence (what you did) and Outcome Metrics (how well it worked)
Scoring uses 4 tiers: Fully Mature (1.0), Implemented (0.67), Partial (0.33), Not Implemented (0.0)
Answer progressively - Complete all Level 1 questions before Level 2
Level progression - Achieve ALL questions at lower level before advancing
Baseline first - Record current metric values before setting targets

Scoring Methodology

Score	Label	Criteria
1.0	Fully Mature	Evidence complete AND ≥3 outcome metrics meet targets
0.67	Implemented	Evidence complete AND 2 outcome metrics meet targets
0.33	Partial	Evidence partially complete OR <2 outcome metrics meet targets
0.0	Not Implemented	No evidence of practice

Level Score = average of question scores within a level Overall DR-Vendors Score = weighted average: L1 × 0.5 + L2 × 0.3 + L3 × 0.2

Maturity Level 1

Objective: Run a standard design checkpoint per AI vendor integration before production, producing a written decision with traceability to SR, SA, and TA

Question 1: AI Vendor Integration Design Checklist

Q1.1: Is there a published, versioned AI Vendor Integration Design Checklist per archetype, derived from the SA-Vendors reference pattern, traceable to the applicable SR-Vendors requirements pack and TA-Vendors threat snapshot?

Q1.2: Does the common spine of each checklist cover pattern adherence, data boundary (which data classes cross to the vendor; DLP/proxy inspection points), identity (SSO-backed human access; service-principal model; secrets management), logging (prompt/completion or equivalent; human-oversight trail; retention), failure modes (vendor outage, model change, rate limit; fallback or kill-switch), permissions for agent archetype (tool allowlist, per-tool scope, HITL gates for destructive or external actions), Art. 50 disclosure for user-visible AI interaction, and residual risk (explicit list with compensating controls, owners, expiry)?

Q1.3: Are checklists updated within 30 days of any SA-Vendors pattern change or SR-Vendors pack update?

Evidence Required: - [ ] Per-archetype checklist set published and version-controlled, one file or section per AI vendor archetype with an explicit version stamp - [ ] Agent-archetype checklist includes all three mandatory permissions items: tool allowlist, per-tool scope minimization, and HITL gates for destructive and external-network actions - [ ] Art. 50 disclosure item present for all archetypes where vendor AI interaction is user-visible - [ ] Each checklist item carries an evidence pointer traced to the SA-Vendors reference pattern or SR-Vendors requirement - [ ] Residual-risk section present in the checklist template (not optional) - [ ] Named lead reviewer per archetype confirmed (EG-Vendors L1 completion verified)

Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |--------|----------|---------|--------|------|-------| | % AI vendor integrations with a completed design-checkpoint record before production | % | % | ≥95% | ☐ | Integration tracker | | % checkpoint records referencing the applicable SA pattern and SR requirement pack | % | % | 100% | ☐ | Checkpoint records | | Median review turnaround, fast-lane | ___ BD | ___ BD | ≤2 BD | ☐ | Review SLA telemetry | | Median review turnaround, full-lane | ___ BD | ___ BD | ≤5 BD | ☐ | Review SLA telemetry |

Metric Collection Guidance: - DR coverage: Count AI vendor integrations reaching production with a dated design-checkpoint record predating implementation start, divided by total integrations promoted to production. Source: integration tracker joined to DR record store. Measured quarterly. - Pattern and requirement pack reference rate: Inspect checkpoint records for a hyperlink or identifier referencing the SA-Vendors pattern and SR-Vendors pack. Automated field validation preferred. - Fast-lane SLA: P50 of (checkpoint decision date − review submission date) for fast-lane reviews. Source: review-tracking system. - Full-lane SLA: P50 of same calculation for full-lane reviews (High/Critical tier, agent archetype, regulated-data involvement, external customer exposure always route full-lane).

Answer: - ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets) - ☐ Implemented (Evidence complete + 2 metrics meet targets) - ☐ Partial (Evidence partially complete + <2 metrics meet targets) - ☐ Not Implemented (No per-archetype checklist published)

Evidence Location	Validation Date	Notes

Question 2: Two-Lane Routing and Decision Records

Q2.1: Is a two-lane routing model operational, fast-lane (Low/Medium tier, on-pattern, ≤2 BD) and full-lane (High/Critical tier, pattern deviation, agent archetype, regulated-data involvement, external customer exposure, ≤5 BD), with routing criteria published and applied consistently?

Q2.2: Does every checkpoint decision record contain: decision (approve / approve-with-conditions / send-back); checklist completed with evidence pointers; deviations listed with rationale; residual risks with compensating controls, named owner, and expiry; reviewer name and date; links to integration inventory record, TA-Vendors threat snapshot, and SR-Vendors requirements pack?

Q2.3: Are open approve-with-conditions items tracked to resolution, with named owners, expiry dates, and an enforcement path before production?

Evidence Required: - [ ] Routing criteria document specifying which tier/archetype/deviation combinations trigger full-lane vs. fast-lane - [ ] Decision record template with all required fields used consistently for the last 10 reviews (sample auditable) - [ ] Approve-with-conditions items in a trackable backlog with named owner and expiry date per item - [ ] Sample of ≥5 decision records showing the residual-risk section populated with named owners (not blank) - [ ] Two-lane review SLAs communicated to integration teams - [ ] Reviewer training records confirming EG-Vendors L1 completion for all active reviewers

Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |--------|----------|---------|--------|------|-------| | % AI vendor integrations with a completed design-checkpoint record before production | % | % | ≥95% | ☐ | Integration tracker | | Open approve-with-conditions items aging >60 days | ___ | ___ | 0 | ☐ | Action-item backlog | | Median review turnaround, fast-lane | ___ BD | ___ BD | ≤2 BD | ☐ | Review SLA telemetry | | Median review turnaround, full-lane | ___ BD | ___ BD | ≤5 BD | ☐ | Review SLA telemetry |

Metric Collection Guidance: - Approve-with-conditions aging: Query action-item backlog for items where (today − condition creation date) > 60 days and status is not resolved. Measured weekly. - SaaS-admin handoff tracking: At L1, flag integrations that require an admin-console no-train confirmation, ensure the admin setting is verified (not assumed from contract language alone) before the checkpoint decision is recorded.

Evidence Location	Validation Date	Notes

Question 3: Loop-back to SA-Vendors, SR-Vendors, and IM-Vendors

Q3.1: Are recurring pattern deviations and repeatedly-waived SR-Vendors requirements automatically queuing SA-Vendors pattern-update and SR-Vendors pack-update reviews, with a threshold of three deviations in the same direction for the same archetype triggering a pattern-update review?

Q3.2: Does every IM-Vendors incident tagged to an AI vendor trigger a re-examination of the design checkpoint that approved it, asking was the issue visible at design time and what checklist item would have caught it?

Q3.3: Is the pattern-deviation rate tracked by archetype and surfaced in a regular program review?

Evidence Required: - [ ] Documented trigger rule: three same-direction deviations per archetype auto-queues SA-Vendors pattern-update review - [ ] SA-Vendors pattern-update queue and SR-Vendors pack-update queue showing items from DR feedback in the last 12 months - [ ] IM-Vendors incident post-mortems including a section linking back to the design checkpoint record with re-examination findings - [ ] Checklist updated in response to at least one IM-Vendors incident finding (version comparison available) - [ ] Pattern-deviation rate by archetype surfaced in a program review in the last quarter

Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |--------|----------|---------|--------|------|-------| | % checkpoint records referencing the applicable SA pattern and SR requirement pack | % | % | 100% | ☐ | Checkpoint records | | SA/SR update items queued from DR feedback in last 12 months | ___ | ___ | ≥1 | ☐ | SA/SR update queues | | % IM-Vendors AI incidents with a DR checkpoint re-examination step | % | % | 100% | ☐ | IM post-mortems | | Open approve-with-conditions items aging >60 days | ___ | ___ | 0 | ☐ | Action-item backlog |

Metric Collection Guidance: - SA/SR queue items from DR: Count items in SA-Vendors or SR-Vendors update queues with a DR-feedback source reference. Measured quarterly. - IM incident DR re-examination: Review IM-Vendors incident records and confirm each has a linked checkpoint record with a re-examination finding. Track coverage as a percentage. Measured after each incident closes.

Evidence Location	Validation Date	Notes

Maturity Level 2

Objective: Move design reviews from checklist to scenario-based walkthroughs; include vendor participation for Critical-tier; detect design drift between reviews; add SaaS-admin handoff review before tenant-wide AI feature enablement

Question 4: Scenario-Based Reviews for Critical and High-Tier Integrations

Q4.1: Are 100% of Critical-tier design reviews conducted as scenario-based walkthroughs, with 3–5 specific threat scenarios sourced from the TA-Vendors library and anonymized industry incidents, with the review decision tied explicitly to how the design handles each scenario?

Q4.2: Are scenario sources refreshed quarterly from TA-Vendors per-vendor threat models, anonymized IM-Vendors incidents, and applicable MITRE ATLAS technique candidates for the vendor archetype?

Q4.3: For High-tier integrations, is the standard full-lane review augmented with at least one scenario from the TA-Vendors archetype library?

Evidence Required: - [ ] DR records for Critical-tier integrations showing scenario-based walkthrough format with ≥3 named scenarios per review, each specific to the integration's tool set, data classes, and output paths - [ ] Each scenario maps to a design control or an accepted residual risk with named owner and expiry - [ ] Scenario library version-controlled with quarterly refresh dates and signal provenance (TA-Vendors model version, IM incident ID, or ATLAS technique cited) - [ ] TA-Vendors per-vendor threat model referenced in each Critical-tier DR record - [ ] High-tier DR records showing at least one augmenting scenario from the TA-Vendors archetype library - [ ] Reviewer population trained on scenario-based walkthrough technique

Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |--------|----------|---------|--------|------|-------| | % Critical reviews using scenario-based walkthrough | % | % | 100% | ☐ | DR records | | % Critical-tier vendors participating in joint reviews (eligible) | % | % | ≥70% | ☐ | DR records | | Drift-detection cadence met | quarterly Critical / annual High | | quarterly Critical / annual High | ☐ | Drift telemetry | | % drift findings returned to DR | % | % | 100% | ☐ | DR queue |

Metric Collection Guidance: - Scenario-based coverage: Count Critical-tier checkpoint records with a "scenarios" section listing ≥3 named threat scenarios, divided by total Critical-tier records. Measured quarterly. - Vendor participation rate: Count Critical-tier vendors who attended a joint review session in the last 12 months (under NDA or sharing agreement), divided by total eligible Critical-tier vendors. Measured annually.

Evidence Location	Validation Date	Notes

Question 5: Vendor Participation and SaaS-Admin Handoff Reviews

Q5.1: Are ≥70% of eligible Critical-tier vendor architects participating in joint design reviews, under pre-established NDA and sharing agreements, with vendor-side FRIA cooperation available for EU AI Act high-risk uses?

Q5.2: Before any SaaS AI feature (Copilot, Notion AI, Slack AI, Workspace AI, or equivalent) is enabled tenant-wide, does a dedicated DR handoff review confirm: the enable workflow is documented, the data scope is declared, conditional enablement is configured where possible, the admin-audit log captures the enablement event, and a drift-detection hook is in place?

Q5.3: Where the SaaS vendor does not expose admin-API controls for a feature, is the gap noted as a residual risk with a named owner and a compensating control in the DR record?

Evidence Required: - [ ] Pre-established NDA and sharing-agreement template ready; Legal pre-approval on file - [ ] Vendor participation log showing Critical-tier vendor architects invited and ≥70% participating for eligible integrations - [ ] SaaS-admin handoff review template with all five confirmation items: enable workflow, data scope, conditional enablement, admin-audit, drift-detection hook - [ ] At least one SaaS-admin handoff review record from the last 12 months - [ ] DR records for SaaS features where vendor admin-API controls are absent, residual risk with compensating control documented - [ ] FRIA cooperation request process established for EU AI Act high-risk vendor use cases

Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |--------|----------|---------|--------|------|-------| | % Critical-tier vendors participating in joint reviews (eligible) | % | % | ≥70% | ☐ | DR records | | % SaaS tenant-wide AI feature enablements with a prior DR handoff record | % | % | 100% | ☐ | SaaS admin log × DR records | | % Critical reviews using scenario-based walkthrough | % | % | 100% | ☐ | DR records | | % drift findings returned to DR | % | % | 100% | ☐ | DR queue |

Metric Collection Guidance: - SaaS handoff coverage: Count tenant-wide SaaS AI feature enablements in the last 12 months with a prior DR handoff record, divided by total such enablements. Source: SaaS admin audit log joined to DR record store. Measured quarterly. - Vendor participation: same as Q4 guidance above.

Evidence Location	Validation Date	Notes

Question 6: Design-Drift Detection

Q6.1: Is design-drift detection operating quarterly for Critical-tier and annually for High-tier integrations, comparing the live integration against the approved design for pattern changes, new tools, new data classes, and new regions?

Q6.2: Are material drift findings (pattern change, new tools or data classes, new regions, no-train setting reverted, Art. 50 disclosure removed, agent permissions expanded) automatically routing back to DR?

Q6.3: Does drift detection produce a staleness alert if a Critical integration has no drift check in the last 90 days?

Evidence Required: - [ ] Drift-detection schedule showing Critical integrations checked quarterly and High integrations annually - [ ] Drift check artifacts (written diffs) for ≥3 Critical-tier integrations in the last 12 months - [ ] Classification criteria defining which delta types are material vs. non-material for vendor integrations - [ ] At least one material drift finding that re-opened a DR record and routed to a new review - [ ] Staleness alert configuration confirmed (Critical integration silent for >90 days triggers alert) - [ ] No-train setting verification included as a drift-check item for applicable vendor integrations

Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |--------|----------|---------|--------|------|-------| | Drift-detection cadence met | quarterly Critical / annual High | | quarterly Critical / annual High | ☐ | Drift telemetry | | % drift findings returned to DR | % | % | 100% | ☐ | DR queue | | % Critical reviews using scenario-based walkthrough | % | % | 100% | ☐ | DR records | | % Critical-tier vendors participating in joint reviews (eligible) | % | % | ≥70% | ☐ | DR records |

Metric Collection Guidance: - Drift check cadence: Count Critical integrations with a documented drift check in the last 90 days, divided by total Critical integrations. Measured monthly. - Material drift re-routing: Count material drift findings with a corresponding DR re-review record, divided by total material drift findings. Measured per drift-check cycle.

Evidence Location	Validation Date	Notes

Maturity Level 3

Objective: Continuous design attestation via automated pattern-compliance telemetry; contribute architecture-review patterns to industry

Question 7: Continuous Design Attestation via Automated Pattern-Compliance Scans

Q7.1: Are ≥90% of Critical AI-vendor integrations producing monthly automated pattern-compliance and configuration-audit signals, confirming no-train setting, data-boundary controls, identity/secrets configuration, logging completeness, and agent-permission scope remain within the DR-approved posture, with deviations automatically opening DR-exception tickets triaged within 3 business days?

Q7.2: Are attestation artifacts machine-readable and surfaceable to regulators, producing EU AI Act Art. 9 risk-management evidence and ISO/IEC 42001 AIMS operational records without manual assembly?

Q7.3: Do human reviewers handle only: novel integrations not covered by existing attestation rules, accepted exceptions with documented rationale, and IM-Vendors escalations?

Evidence Required: - [ ] Attestation pipeline configuration showing monthly scan cadence for Critical-tier integrations - [ ] Coverage report: % of Critical-tier integrations producing a fresh attestation signal in the last 7 days - [ ] Sample attestation artifact covering: no-train setting, data-boundary controls, identity/secrets, logging completeness, agent-permission scope - [ ] DR-exception ticket queue showing tickets opened automatically on attestation deviation - [ ] Evidence that at least one DR-exception ticket was triaged within 3 business days of opening - [ ] Staleness alert confirmed (Critical integration without attestation in 7 days triggers alert)

Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |--------|----------|---------|--------|------|-------| | % Critical integrations with monthly auto-attestation | % | % | ≥90% | ☐ | Attestation telemetry | | Mean DR-exception ticket age from open to triage | ___ BD | ___ BD | ≤3 BD | ☐ | DR-exception queue | | Mean review backlog age | ___ days | ___ days | ≤7 days | ☐ | Review queue telemetry | | Quarterly pattern-evolution reviews conducted | ___ | ___ | 4 / year | ☐ | Pattern-update log |

Metric Collection Guidance: - Attestation coverage: Count Critical integrations with a completed attestation scan in the last 7 days, divided by total Critical integrations. Sourced from attestation pipeline run log. Measured weekly; alert if below 90%. - Exception ticket SLA: P50 of (triage timestamp − open timestamp) for DR-exception tickets. Measured weekly. - Review backlog age: P90 age of non-exception review queue items. Measured weekly.

Evidence Location	Validation Date	Notes

Question 8: Contribute Review Patterns to Industry

Q8.1: Has the program contributed ≥2 substantive review artifacts per year (review rubrics, scenario templates, vendor-cooperation frameworks) to industry bodies (OpenSSF AI, CSA AI Safety Initiative, Shared Assessments, or equivalent), with documented adoption?

Q8.2: Are internal rubrics and templates kept aligned to the published external versions, with internal deviations proposed as upstream changes rather than silently forked?

Q8.3: Is adoption tracked via citations, forks, or direct acknowledgment from peer organizations or standards bodies?

Evidence Required: - [ ] Contribution log showing ≥2 published artifacts in the last 12 months, review rubrics, scenario templates, or vendor-cooperation frameworks - [ ] Publication links (Apache 2.0 or equivalent) to OpenSSF AI, CSA AI Safety Initiative, Shared Assessments, or equivalent body - [ ] Adoption evidence: citations, forks, or written acknowledgment from a peer organization - [ ] Internal rubric version compared to external published version, confirmed aligned or upstream PR submitted for divergences - [ ] At least one artifact in draft, in-review, or published at the time of assessment

Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |--------|----------|---------|--------|------|-------| | Industry contributions per year (rubrics, scenario templates, vendor-cooperation frameworks) | 0 | ___ | ≥2 | ☐ | Contribution log | | % Critical integrations with monthly auto-attestation | % | % | ≥90% | ☐ | Attestation telemetry | | Mean DR-exception ticket age from open to triage | ___ BD | ___ BD | ≤3 BD | ☐ | DR-exception queue | | Quarterly pattern-evolution reviews conducted | ___ | ___ | 4 / year | ☐ | Pattern-update log |

Metric Collection Guidance: - Industry contributions: Count distinct published artifacts publicly accessible under an open license and attributed to this organization. Measured annually. - Adoption evidence: Log citations and forks quarterly. Vendor-cooperation framework adoption by peer assessors tracked as a key signal.

Evidence Location	Validation Date	Notes

Question 9: Quarterly Pattern Evolution Driven by External and Internal Signals

Q9.1: Is there a quarterly pattern-evolution review driven by external incident patterns (ISAC advisories, MITRE ATLAS technique additions relevant to vendor AI archetypes) and internal signals (IM-Vendors incident patterns, ML-Vendors telemetry anomalies, ST-Vendors findings), with a versioned change log?

Q9.2: Are downstream DR records for in-flight reviews notified of pattern changes that affect their vendor archetype?

Q9.3: Where an ISAC advisory or IM-Vendors incident reveals a checklist gap, is the gap propagated to SA-Vendors and SR-Vendors to maintain the traceability chain?

Evidence Required: - [ ] Quarterly pattern-evolution review calendar with at least 4 sessions completed in the last 12 months, each with a dated agenda and change log entry - [ ] Change log showing signal provenance (ISAC advisory reference, ATLAS technique ID, or IM incident ID) for each update - [ ] Evidence that ISAC and ATLAS feeds were ingested monthly into the pattern-update queue - [ ] In-flight DR review notifications sent when a pattern change affected the vendor archetype under review - [ ] SA-Vendors and SR-Vendors update items queued from pattern-evolution checklist gaps

Outcome Metrics: | Metric | Baseline | Current | Target | Met? | Notes | |--------|----------|---------|--------|------|-------| | Quarterly pattern-evolution reviews conducted | ___ | ___ | 4 / year | ☐ | Pattern-update log | | % Critical integrations with monthly auto-attestation | % | % | ≥90% | ☐ | Attestation telemetry | | Industry contributions per year | 0 | ___ | ≥2 | ☐ | Contribution log | | Mean review backlog age | ___ days | ___ days | ≤7 days | ☐ | Review queue telemetry |

Metric Collection Guidance: - Pattern-evolution cadence: Count completed quarterly reviews with a dated agenda and at least one change log entry citing an external signal source. Measured annually. - Signal provenance completeness: Spot-check 5 change log entries per quarter, verify each has a named source (ISAC advisory reference, ATLAS technique ID, or IM incident ID).

Evidence Location	Validation Date	Notes

Summary Scorecard

Question	Level	Score (0 / 0.33 / 0.67 / 1.0)	Notes
Q1: AI Vendor Integration Design Checklist	L1
Q2: Two-Lane Routing and Decision Records	L1
Q3: Loop-back to SA / SR / IM	L1
Q4: Scenario-Based Reviews (Critical/High)	L2
Q5: Vendor Participation and SaaS-Admin Handoff	L2
Q6: Design-Drift Detection	L2
Q7: Continuous Design Attestation	L3
Q8: Industry Contributions	L3
Q9: Quarterly Pattern Evolution	L3

Level 1 Score: ___ / 1.0 (average of Q1–Q3) Level 2 Score: ___ / 1.0 (average of Q4–Q6) Level 3 Score: ___ / 1.0 (average of Q7–Q9) Overall DR-Vendors Score: ___ / 1.0 (L1 × 0.5 + L2 × 0.3 + L3 × 0.2)

Current Maturity Level: ☐ L1 ☐ L2 ☐ L3 Assessment Date: Assessor: Next Review Date:

Document Version: HAIAMM v3.0 Practice: Design Review (DR) Domain: Vendors Questionnaire Date: 2026-05-15 Author: Verifhai

Instructions:

Answer based on current practices, not plans
“Yes” requires documented evidence
Complete all Level 1 questions before Level 2
Partial implementation = “No”

↓ Download as Markdown