Secure Architecture (SA) - Vendors Domain

HAIAMM Assessment Questionnaire v3.5

Generated from cells/SA-Vendors-L*.cell.json. Do not edit by hand. Canonical model: HAIAMM-v3.5-Framing.md · Handbook: HAIAMM-v3.5-Handbook.md.

---

Practice: Secure Architecture (SA) Domain: Vendors Business Function: Building Purpose: Publish reference architecture patterns and an antipattern catalogue for each vendors archetype, so the secure design is the default path rather than an achievement. Scoring Model: Evidence + Outcome Metrics (see Scoring Methodology below)

---

Instructions

• Answer from current, implemented practice, not plans or aspirations.
• Each question has two components: Evidence (what you did) and Outcome Metrics (how well it worked).
• Levels are derived, never self-assigned. Record evidence and metric values; the level falls out of the scoring below.
• Silence is not a pass. A metric you cannot produce is left blank and counts as not met. Do not estimate.
• Answer progressively. Complete all Level 1 questions before Level 2.

Evidence grade

Record how each answer is known, not only what is claimed (v3.5 framing §6). A level claim is graded at its weakest field.

Grade	Meaning
E0	Asserted by a practitioner, unevidenced
E1	Asserted, with a link to a named artifact
E2	Emitted by a system as telemetry
E3	Machine-attested and independently checkable

---

Scoring Methodology

Tier	Score	Criteria
Fully Mature	1.0	Evidence complete + ≥3 outcome metrics meet targets
Implemented	0.67	Evidence complete + 2 outcome metrics meet targets
Partial	0.33	Evidence partially complete + <2 outcome metrics meet targets
Not Implemented	0.0	No substantive evidence of practice

Practice maturity level achieved = the highest level where all 3 questions score ≥ 0.67.

---

Maturity Level 1, Foundational

Objective: Publish reference architecture patterns and an antipattern catalogue for each vendors archetype, so the secure design is the default path rather than an achievement.

Prerequisite cells: SM-Vendors-L1, TA-Vendors-L1, SR-Vendors-L1

Question L1.1: Publish reference patterns per archetype

Q1.1: Does a reference pattern exist for at least 85% of inventoried vendors archetypes, each stating components, boundaries, controls, the threats it mitigates by TA identifier, the requirements it satisfies and where it does not apply, validated with an engineer who has built that archetype?

Evidence Required:

• [ ] Author one reference pattern per archetype, covering the brokered-access pattern routing vendor AI through an organisational gateway, the data-minimisation pattern filtering before egress to the vendor, the fallback pattern surviving vendor unavailability, and the exit pattern keeping data portable by construction
• [ ] Each pattern states the components and their boundaries, the controls at each boundary, the threats it mitigates by identifier from the TA library, the requirements it satisfies from the SR pack, and the conditions under which it does not apply
• [ ] Keep each pattern to a length a team reads before designing rather than after, targeting three pages with a diagram
• [ ] Validate each with an engineer who has built that archetype, because a pattern authored entirely within security describes a system that is secure and unbuildable, and teams respond to that by ignoring it rather than by arguing
• [ ] Publish patterns in the repository where the work happens rather than in a separate architecture space

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L1.2: Publish the antipattern catalogue

Q1.2: Is there an antipattern catalogue of at least eight entries drawn from designs actually seen in review, each stating why the shape is attractive, what goes wrong, and which reference pattern replaces it? For Vendors, the assessor should be shown the brokered-access pattern and the data-minimisation filter.

Evidence Required:

• [ ] Author an antipattern catalogue naming the insecure shapes that actually recur in this domain, being direct vendor API access from application code with credentials embedded, unfiltered organisational data sent to a vendor endpoint, and a vendor dependency with no fallback and no tested exit
• [ ] For each antipattern, state why it is attractive, since it is always attractive for a reason and a catalogue that does not acknowledge that reads as naive
• [ ] What specifically goes wrong
• [ ] And which reference pattern replaces it
• [ ] The antipattern half of the library is the more useful and the more commonly omitted, because teams recognise the thing they are building far more readily than they recognise the thing they should have built instead
• [ ] Draw the catalogue from real designs seen in review rather than from literature, and add to it whenever review encounters a new recurring shape

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L1.3: Establish the untrusted-content boundary and a design-time self-check

Q1.3: Does every archetype ingesting content the organisation does not author have a documented untrusted-content boundary, and is a design-time self-check published and completed by at least 70% of designs arriving at review? In Vendors, ask to see the brokered-access pattern and the data-minimisation filter.

Evidence Required:

• [ ] Document explicitly where untrusted content enters each archetype and how it is kept separate from instruction, control and identity
• [ ] This is the single most consequential architectural decision in AI systems and the one most often left implicit, because the failure it prevents, content becoming instruction, does not look like a boundary violation in any conventional sense
• [ ] State the pattern for isolation, for provenance marking, and for what the system must never do with content it did not author
• [ ] Publish a short design-time self-check, at most ten questions, that a team can answer before requesting review, which both raises the quality of what arrives at review and tells the team what the review will ask, removing the asymmetry that makes design review feel arbitrary

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Level 1 Outcome Metrics

Metric	Baseline	Current	Target	Met?	Notes
Vendors archetypes with a published reference pattern	___	___	≥85% of inventoried archetypes within 9 months	☐
Patterns stating the threats they mitigate by TA identifier and the requirements they satisfy	___	___	100% of published patterns	☐
Antipatterns in the catalogue drawn from designs actually seen in review	___	___	≥8 antipatterns, each traced to an observed design	☐
Archetypes with a documented untrusted-content boundary	___	___	100% of archetypes that ingest content the organisation does not author	☐
Designs arriving at review having completed the design-time self-check	___	___	≥70%	☐

Where each number is read from:

• Vendors archetypes with a published reference pattern — Pattern library register joined to the asset inventory
• Patterns stating the threats they mitigate by TA identifier and the requirements they satisfy — Pattern library traceability audit
• Antipatterns in the catalogue drawn from designs actually seen in review — Antipattern catalogue with review references
• Archetypes with a documented untrusted-content boundary — Pattern library boundary section audit
• Designs arriving at review having completed the design-time self-check — Design review intake records

Level 1 Oversight Metrics

"A human approves it" is not a control unless you measure whether they refuse. A metric that cannot fail is not a metric.

Metric	Current	Oversight failure it detects	Met?
Share of designs where the reviewer recorded a required change before approval	___	rubber-stamping at design review, where approval is recorded without the review having altered anything, which is the default state of a review with no capacity to say no	☐
Designs approved with a human gate declared on a new data class flowing to a vendor but no stated refusal authority	___	a gate that exists structurally while the person holding it cannot actually refuse, which satisfies the control on paper and not in fact	☐

Level 1 30-second check (E0)

• ☐ Yes ☐ No — Is there a reference architecture for each kind of AI vendors asset?
• ☐ Yes ☐ No — Can a team tell at design time whether their shape is sanctioned?
• ☐ Yes ☐ No — Is it documented where untrusted content enters your AI systems?

---

Maturity Level 2, Comprehensive

Objective: Calibrate the vendors reference-pattern set by risk tier and scale the autonomy and oversight controls in each pattern with the asset's capability tier.

Prerequisite cells: SA-Vendors-L1, SM-Vendors-L2, TA-Vendors-L2, SR-Vendors-L2

Question L2.1: Declare tier applicability across the pattern library

Q2.1: Does every pattern declare its tier applicability and mandatory-versus-recommended status, with the applicable set re-derived automatically when an asset's tier changes? Evidence this in the Vendors domain against the brokered-access pattern and the data-minimisation filter.

Evidence Required:

• [ ] Annotate every pattern in the library with the tiers it applies to and the tiers at which it is mandatory rather than recommended
• [ ] Publish the mapping so that a team's pattern set is derived from the asset's tier rather than assembled by preference
• [ ] At Critical tier, pattern adherence is mandatory and any deviation requires a full-lane design review with a named architect
• [ ] At Low tier the pattern is recommended and deviation needs only a recorded note
• [ ] Stating this removes the ambiguity that otherwise resolves in favour of whatever is quickest to build
• [ ] Re-derive the applicable pattern set automatically when an asset's tier changes, because a Medium-to-Critical promotion that does not change the pattern obligations has changed nothing that matters

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L2.2: Scale autonomy and oversight controls with capability tier

Q2.2: Do the controls inside each pattern tighten as capability rises, with the human gate on a new data class flowing to a vendor specified numerically in authority, information and time budget rather than as a principle, and is a per-tier reviewer-load ceiling published?

Evidence Required:

• [ ] Within each pattern, define how its controls tighten as the asset's capability tier rises
• [ ] Bound blast radius more tightly, shorten the autonomy budget between human checkpoints, and require a human gate on a new data class flowing to a vendor at higher capability tiers where lower tiers need only logging
• [ ] Specify each gate numerically rather than in principle: who holds the authority to refuse, what information they receive, and how long they have
• [ ] A gate specified as a principle is compressed under delivery pressure until scrutiny is impossible, and because nothing in the specification objects, the erosion is invisible
• [ ] Publish a reviewer-load ceiling per tier above which review capacity must be added rather than reviewers exhorted, since a gate held by an overloaded reviewer is a gate in name only

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L2.3: Run the incident-to-pattern learning loop

Q2.3: Is every incident reaching a root cause checked against the library for a preventing pattern with negative findings recorded, and is the share attributable to an existing pattern not followed tracked separately from gaps in the library? In Vendors, ask to see the brokered-access pattern and the data-minimisation filter.

Evidence Required:

• [ ] Establish a standing review that takes every incident reaching a root cause and asks two questions: was there a pattern that should have prevented this, and if so why was it not followed, or if not what pattern should now exist
• [ ] Record the answer for every incident, including the answer that no pattern change is warranted, because the discipline of recording the negative case is what keeps the loop honest rather than performative
• [ ] Where a pattern existed and was not followed, the finding belongs to the review and adoption practices rather than to the library
• [ ] Where no pattern existed, author one and publish it within 60 days
• [ ] Track the share of incidents attributable to a pattern that existed and was not followed, which separates a library problem from an adoption problem and is the single most useful number this practice produces

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Level 2 Outcome Metrics

Metric	Baseline	Current	Target	Met?	Notes
Patterns carrying a declared tier applicability	___	___	100% of published patterns	☐
Critical-tier assets following a reference pattern or carrying a full-lane deviation review	___	___	≥95%, with no third category	☐
Pattern deviations carrying a named approver, recorded rationale and expiry	___	___	100%, with none past expiry	☐
Incidents checked against the pattern library for a preventing pattern	___	___	100% of incidents reaching root cause, including negative findings	☐
Critical-tier assets carrying no unbounded autonomy budget	___	___	100%	☐

Where each number is read from:

• Patterns carrying a declared tier applicability — Pattern library metadata
• Critical-tier assets following a reference pattern or carrying a full-lane deviation review — Pattern adoption manifest joined to inventory tier
• Pattern deviations carrying a named approver, recorded rationale and expiry — Deviation register
• Incidents checked against the pattern library for a preventing pattern — Incident review record joined to the pattern library
• Critical-tier assets carrying no unbounded autonomy budget — Pattern conformance audit

Level 2 Oversight Metrics

"A human approves it" is not a control unless you measure whether they refuse. A metric that cannot fail is not a metric.

Metric	Current	Oversight failure it detects	Met?
Reviewer override rate on pattern deviation requests, paired with sustained-override rate	___	deviation approval becoming automatic under delivery pressure, where pairing with sustained rate counters the gaming that follows once the rate becomes a target	☐
Design reviews per reviewer per week against the published ceiling	___	reviewer overload, where architectural scrutiny cannot survive the volume regardless of reviewer skill, which is a capacity defect rather than a personnel one	☐
Critical-tier designs where the human gate on a new data class flowing to a vendor specifies authority, information and a time budget numerically	___	oversight specified as a principle rather than a number, which is compressed under delivery pressure until scrutiny is impossible and nothing objects	☐

Level 2 30-second check (E0)

• ☐ Yes ☐ No — Does a high-risk AI asset have to follow a reference architecture?
• ☐ Yes ☐ No — Does your architecture demand more of AI that acts than AI that advises?
• ☐ Yes ☐ No — After an incident, does anyone ask whether a pattern should have prevented it?

---

Maturity Level 3, Industry-Leading

Objective: Publish vendors patterns as executable artifacts adopted by default, verify conformance continuously, and contribute the pattern library back to the field.

Prerequisite cells: SA-Vendors-L2, IR-Vendors-L2, ML-Vendors-L2

Question L3.1: Ship patterns as executable artifacts

Q3.1: Do at least 80% of reference patterns ship as executable artifacts with versioning and a migration path, and do at least 75% of new Critical- and High-tier assets adopt the artifact at creation rather than adopting the document? Evidence this in the Vendors domain against the brokered-access pattern and the data-minimisation filter.

Evidence Required:

• [ ] Convert each reference pattern into something a team runs rather than reads: infrastructure modules, project scaffolds, libraries or configuration bundles that instantiate the pattern correctly by default
• [ ] This is the level's central move and the one that changes adoption economics, because a pattern that is faster to adopt than to circumvent is adopted without enforcement, and no amount of mandate achieves what convenience achieves
• [ ] Version the artifacts, publish their change history, and provide a migration path when a pattern changes so that existing adopters are not stranded on an old version, which is the most common reason executable pattern libraries decay
• [ ] Measure adoption of the artifact rather than conformance to the document, since the artifact is now the pattern
• [ ] In this domain the executable artifact is a broker or gateway configuration that routes vendor access, filters the payload and applies limits, so using the vendor correctly is easier than integrating directly

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L3.2: Verify conformance continuously against the running system

Q3.2: Are at least 85% of pattern invariants verified continuously against the running system with unverifiable invariants listed explicitly, and are Critical-tier breaches detected within 5 business days and routed to issue management? For Vendors, the assessor should be shown the brokered-access pattern and the data-minimisation filter.

Evidence Required:

• [ ] Verify that what is running still matches the pattern that was approved, continuously rather than at review
• [ ] Draw the signals from configuration state, runtime telemetry and the deployment pipeline, and evaluate each pattern's invariants as checks that can fail
• [ ] Architecture drift is the defining failure of this practice at scale: designs are reviewed once and systems change continuously, so a library verified only at design time describes the past
• [ ] Publish a service level covering detection of a Critical-tier conformance breach within 5 business days, and route breaches to the issue-management practice with tier-appropriate severity rather than to a report nobody owns
• [ ] Where an invariant cannot be checked automatically, list it explicitly so the verification's real reach is known
• [ ] Conformance verification reads the integration path and payload schema against the pattern, which catches the direct integration added for latency beside the sanctioned broker

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L3.3: Contribute the executable pattern library back

Q3.3: Does the organisation contribute at least four accepted artifacts per year in majority-executable form, prioritising the patterns with the highest internal adoption and therefore the most operational evidence? In Vendors, ask to see the brokered-access pattern and the data-minimisation filter.

Evidence Required:

• [ ] Contribute at least four substantive artifacts per year, weighted toward executable form: reference implementations, infrastructure modules, scaffolds and the conformance checks that verify them
• [ ] Executable contributions carry disproportionate value in this practice, because a peer organisation can adopt a module and can only read a diagram, and the adoption gap between the two is the difference between influencing the field and describing it
• [ ] Work through OpenSSF AI, CNCF where infrastructure applies, OWASP projects and sector bodies
• [ ] Count only accepted artifacts, review each legally before release, and prefer contributing the patterns with the highest internal adoption, since those carry the most operational evidence and are the most likely to survive contact with another organisation's environment
• [ ] Vendor patterns contribute well as broker configurations and payload-filtering rules, which peers can adopt against the same small set of major providers

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Level 3 Outcome Metrics

Metric	Baseline	Current	Target	Met?	Notes
Reference patterns shipped as executable artifacts rather than documents alone	___	___	≥80% of patterns	☐
New assets adopting the executable pattern artifact at creation	___	___	≥75% of new Critical- and High-tier assets	☐
Pattern invariants verified continuously against the running system	___	___	≥85%, with unverifiable invariants listed explicitly	☐
Time to detect a Critical-tier architecture conformance breach	___	___	≤5 business days at the 90th percentile	☐
Accepted external contributions per year in executable form	___	___	≥4 accepted, majority executable	☐

Where each number is read from:

• Reference patterns shipped as executable artifacts rather than documents alone — Pattern artifact registry
• New assets adopting the executable pattern artifact at creation — Scaffold and module adoption telemetry
• Pattern invariants verified continuously against the running system — Conformance verification registry
• Time to detect a Critical-tier architecture conformance breach — Conformance verification event log
• Accepted external contributions per year in executable form — Contribution log with acceptance references

Level 3 Oversight Metrics

"A human approves it" is not a control unless you measure whether they refuse. A metric that cannot fail is not a metric.

Metric	Current	Oversight failure it detects	Met?
Pattern deviations later implicated in an incident, as a share of deviations approved	___	deviation approval that is procedurally correct and substantively wrong, which only post-hoc linkage to incidents can reveal	☐
Automated pattern conformance checks overridden by a human, with the override sustained	___	automation override becoming the routine path around a check that is inconvenient rather than wrong, which is how enforced patterns quietly become advisory	☐
Reviewer disagreement rate on a blind double-review sample of Critical-tier designs	___	reviewer drift, where the same design would be approved by one architect and refused by another, making the gate a lottery rather than a control	☐

Level 3 30-second check (E0)

• ☐ Yes ☐ No — Can a team adopt your secure architecture by running something rather than reading something?
• ☐ Yes ☐ No — Would you detect a running system drifting from its approved design?
• ☐ Yes ☐ No — Have you published an architecture module others can actually deploy?

---

Summary Scorecard

Level	Q1	Q2	Q3	Level score	Weakest evidence grade	Achieved?
L1	___	___	___	___	___	☐
L2	___	___	___	___	___	☐
L3	___	___	___	___	___	☐

Secure Architecture (SA) maturity in the Vendors domain: ____________

---

Document Version: HAIAMM v3.5 (generated)