Strategy & Metrics (SM) - Vendors Domain

HAIAMM Assessment Questionnaire v3.5

Generated from cells/SM-Vendors-L*.cell.json. Do not edit by hand. Canonical model: HAIAMM-v3.5-Framing.md · Handbook: HAIAMM-v3.5-Handbook.md.

---

Practice: Strategy & Metrics (SM) Domain: Vendors Business Function: Governance Purpose: Stand up vendor AI assurance, inventory every AI tool and service the organization consumes, and baseline the metrics that prove shadow AI adoption is decreasing. Scoring Model: Evidence + Outcome Metrics (see Scoring Methodology below)

---

Instructions

• Answer from current, implemented practice, not plans or aspirations.
• Each question has two components: Evidence (what you did) and Outcome Metrics (how well it worked).
• Levels are derived, never self-assigned. Record evidence and metric values; the level falls out of the scoring below.
• Silence is not a pass. A metric you cannot produce is left blank and counts as not met. Do not estimate.
• Answer progressively. Complete all Level 1 questions before Level 2.

Evidence grade

Record how each answer is known, not only what is claimed (v3.5 framing §6). A level claim is graded at its weakest field.

Grade	Meaning
E0	Asserted by a practitioner, unevidenced
E1	Asserted, with a link to a named artifact
E2	Emitted by a system as telemetry
E3	Machine-attested and independently checkable

---

Scoring Methodology

Tier	Score	Criteria
Fully Mature	1.0	Evidence complete + ≥3 outcome metrics meet targets
Implemented	0.67	Evidence complete + 2 outcome metrics meet targets
Partial	0.33	Evidence partially complete + <2 outcome metrics meet targets
Not Implemented	0.0	No substantive evidence of practice

Practice maturity level achieved = the highest level where all 3 questions score ≥ 0.67.

---

Maturity Level 1, Foundational

Objective: Stand up vendor AI assurance, inventory every AI tool and service the organization consumes, and baseline the metrics that prove shadow AI adoption is decreasing.

Question L1.1: Charter vendor AI assurance and open an amnesty window

Q1.1: Is there a published vendor AI assurance charter jointly sponsored by the CISO and the procurement or IT leader, naming all five vendor archetypes, four decision rights and a completed amnesty window of at least 60 days paired with a sanctioned path faster than self-service adoption?

Evidence Required:

• [ ] Publish a charter within the first 30 days sponsored by the CISO jointly with the Chief Procurement Officer or CIO, co-signed by Privacy counsel
• [ ] Name the five in-scope archetypes explicitly, because the most commonly missed category is the AI feature that appeared inside an already-approved SaaS product without a new contract, a new invoice or any procurement event at all
• [ ] Establish the working group across Security, Procurement, Legal and Privacy, IT, Data Governance and one business-unit representative, and define decision rights over who sanctions an AI vendor, who blocks one, who grants exceptions and who owns offboarding
• [ ] Open an amnesty window of at least 60 days, publicized through all-hands and team channels, under which anyone can disclose an AI tool they are using with no disciplinary consequence
• [ ] Amnesty is the single highest-yield discovery mechanism in this domain and it only works once, so pair it with a sanctioned path that is genuinely faster than signing up with a credit card, or disclosure will not repeat

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L1.2: Discover across five signals and inventory by archetype

Q1.2: Does one AI vendor inventory reconcile monthly against expense and procurement records, identity provider catalogues, network egress, endpoint and browser-extension audits, and the admin consoles of major approved SaaS products, reaching 90% coverage or better within 12 months?

Evidence Required:

• [ ] Build one AI vendor inventory recording per vendor the archetype, business owner, sanction status, data classes shared, whether inputs are used for provider training and how that is contractually established, sub-processor chain, data residency and transfer basis, model provider behind the service, seat count and utilisation, contract and renewal dates, and links to the TA snapshot and PC compliance mapping
• [ ] Reconcile monthly against five discovery signals: expense and procurement records including corporate card transactions, which catch the self-service adoptions that never reached procurement
• [ ] Identity provider application catalogues and single-sign-on grants, which catch tools users authenticated into
• [ ] Network egress to known AI provider domains, which catches everything with a browser or API footprint
• [ ] Endpoint management and browser extension audits, which catch locally installed assistants
• [ ] And the admin consoles of major approved SaaS products, which is the only way to see AI features that were switched on inside a product the organization already owns
• [ ] Treat any vendor appearing in a signal but not the inventory as shadow AI until attributed

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L1.3: Baseline the scoreboard and open the vendor spend ledger

Q1.3: Do all vendors receiving confidential or regulated data have a contractually established training-data position, and is a consumption ledger open recording cost per active seat with utilisation, committed-spend terms, overage rates and exit cost?

Evidence Required:

• [ ] Instrument and report quarterly: AI vendor inventory coverage against discovery reconciliation, the shadow-AI ratio as the domain's primary L1 outcome, acceptable-use attestation across the workforce, the share of sanctioned vendors with a documented training-data position, and known data-exposure events to unsanctioned AI tools
• [ ] Open the vendor consumption ledger recording cost per active seat and cost per token or unit of service, attributed to the business owner, together with committed-spend terms, overage rates, price-change notice periods and exit cost
• [ ] This domain is the only one where consumption is contractually governed rather than merely observed, which makes the ledger a negotiating instrument as well as a control: concentration risk becomes a number in dollars, and lock-in becomes a measured switching cost rather than an asserted worry
• [ ] Track seat utilisation alongside seat cost, since provisioned-and-never-used seats are simultaneously wasted spend and unnecessary data exposure

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Level 1 Outcome Metrics

Metric	Baseline	Current	Target	Met?	Notes
AI vendor inventory coverage against discovery reconciliation	___	___	≥90% within 12 months	☐
Shadow AI ratio (unsanctioned AI vendors in use ÷ total AI vendors in use)	___	___	≤15% and falling for two consecutive quarters	☐
Workforce with an acknowledged AI Acceptable Use Policy	___	___	≥95% of workforce	☐
Sanctioned AI vendors with a documented, contractually established training-data position	___	___	100% of vendors receiving confidential or regulated data	☐
Known data-exposure events to unsanctioned AI tools	___	___	falling quarter over quarter, with zero involving regulated data	☐

Where each number is read from:

• AI vendor inventory coverage against discovery reconciliation — Inventory reconciled against expense, IdP, egress, endpoint and SaaS-admin signals; where no central identity provider covers the tool, per-vendor admin console user lists, recorded at E1
• Shadow AI ratio (unsanctioned AI vendors in use ÷ total AI vendors in use) — Inventory sanction-status field
• Workforce with an acknowledged AI Acceptable Use Policy — HR / LMS attestation records
• Sanctioned AI vendors with a documented, contractually established training-data position — Contract register joined to inventory data-class field
• Known data-exposure events to unsanctioned AI tools — DLP alerts and incident tracker

Level 1 Consumption Metrics

Absolute spend is a ledger input, never a metric. Every row is divided by a unit of business work. If AI spend is not allocated to an owning team, record this level as blocked on allocation rather than scoring it down (v3.5 framing §12).

Metric	Denominator	Current	Economic failure it detects	Met?
Cost per active seat, by vendor and archetype	per active seat	___	licence waste where seats are provisioned and never used, which is wasted spend and standing data exposure at the same time	☐
Cost per unit of service against committed-spend terms and overage rates	per governed workload	___	overage exposure and unbudgeted price changes on renewal, and concentration that is only visible once consumption is priced per vendor	☐
Assured Spend Ratio for vendor AI: consumption through sanctioned vendors	per dollar of vendor AI consumption	___	a healthy vendor count masking that most actual AI consumption flows through tools the program has not sanctioned	☐

Level 1 30-second check (E0)

• ☐ Yes ☐ No — Do you have one list of every AI tool your organization pays for?
• ☐ Yes ☐ No — Would you know if an approved SaaS product switched on an AI feature?
• ☐ Yes ☐ No — Can you say whether any vendor trains on the data you send them?

---

Maturity Level 2, Comprehensive

Objective: Tier AI vendors by data exposure and dependency, calibrate assessment depth per tier, and govern concentration, committed spend and exit cost as measured positions.

Prerequisite cells: SM-Vendors-L1, PC-Vendors-L1, TA-Vendors-L1

Question L2.1: Publish the vendor risk-tier rubric

Q2.1: Is every AI vendor assigned a tier from a published rubric covering data exposure, dependency criticality, service autonomy, sub-processor depth and switching cost, re-derived when data classes or service scope change?

Evidence Required:

• [ ] Derive four tiers from five auditable dimensions
• [ ] Data exposure is the floor: a vendor receiving regulated or customer-confidential content sits at Critical or High regardless of spend, because contract value and risk exposure are only weakly correlated in this domain and a cheap tool can carry the whole customer base through it
• [ ] Dependency criticality elevates where the vendor sits on a revenue path or a customer-facing flow
• [ ] Service autonomy elevates where the vendor's AI acts rather than advises, which is what distinguishes an agent platform from a drafting assistant
• [ ] Sub-processor depth elevates where the vendor's own model provider and its sub-processors extend the chain beyond the point where the organization has visibility or contractual reach
• [ ] Switching cost elevates where exit would take longer than a quarter or require re-engineering, because a dependency that cannot be exited is a dependency that cannot be governed by the threat of exit
• [ ] Publish the rubric and re-derive when data classes or service scope change

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L2.2: Calibrate assessment depth and contract terms per tier

Q2.2: Is there a published tier-treatment matrix covering assessment depth, required contractual terms including training-data prohibition and model-change notification, monitoring, reassessment cadence and tested exit runbooks, with at least 95% Critical-tier contractual conformance?

Evidence Required:

• [ ] Publish a tier-treatment matrix covering per tier: assessment depth from questionnaire to evidence-backed review with technical validation
• [ ] Required contractual terms including training-data prohibition, sub-processor notification and approval rights, audit and evidence rights, incident-notification windows, model-change notification, data-residency commitments and deletion-on-termination obligations
• [ ] Monitoring obligations including provider status and advisory tracking
• [ ] Reassessment cadence and the events that force reassessment ahead of schedule
• [ ] And exit requirements including data-export format and a tested exit runbook for Critical tier
• [ ] Add the consumption row: Critical- and High-tier vendors carry governed committed-spend terms with a defined overage rate, a price-change notice period and a documented exit cost
• [ ] Treat a vendor's model change, a sub-processor addition or a material pricing change as a material change triggering reassessment inside the tier window, since each alters the risk profile that the original assessment described

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L2.3: Price concentration and exit, and report per tier

Q2.3: Does a sponsor-signed concentration and dependency statement price the share of consumption carried by each vendor and each underlying model provider, with measured exit time and switching cost for every Critical-tier vendor and an explicit accept, mitigate or migrate decision?

Evidence Required:

• [ ] Produce a standing concentration and dependency statement, refreshed quarterly and signed by the sponsor, that prices exposure rather than describing it: the share of AI consumption carried by each vendor and each underlying model provider, the single points of failure that follow, the measured exit time and switching cost for every Critical-tier vendor, and an explicit accept, mitigate or migrate decision for each
• [ ] Measuring exit cost requires actually specifying the exit, which is why most organizations never do it, and why the number is worth having
• [ ] Report the scoreboard per tier quarterly covering inventory and tier state, shadow AI ratio per tier with unsanctioned Critical-tier vendors targeted at zero, contractual-term conformance per tier, reassessment currency, seat utilisation by tier, and the Assured Spend Ratio for vendor consumption
• [ ] Maintain a tier-movement log recording the triggering dimension and review it with the sponsor

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Level 2 Outcome Metrics

Metric	Baseline	Current	Target	Met?	Notes
AI vendors carrying a current tier assignment	___	___	100% of vendors in use, with tier age under 12 months	☐
Critical- and High-tier vendors with all required contractual terms in force	___	___	≥95% of Critical, ≥85% of High	☐
Unsanctioned Critical-tier AI vendors in use	___	___	0, sustained	☐
Critical-tier vendors with a measured exit time and documented switching cost	___	___	100%, refreshed annually	☐
Reassessment currency: Critical- and High-tier vendors reassessed within their tier cadence	___	___	≥95%	☐
Seat utilisation at Critical and High tier	___	___	≥80% of provisioned seats active in the last 30 days	☐

Where each number is read from:

• AI vendors carrying a current tier assignment — Inventory tier field
• Critical- and High-tier vendors with all required contractual terms in force — Contract register conformance review joined to inventory tier
• Unsanctioned Critical-tier AI vendors in use — Inventory tier and sanction-status fields
• Critical-tier vendors with a measured exit time and documented switching cost — Concentration and dependency statement
• Reassessment currency: Critical- and High-tier vendors reassessed within their tier cadence — TPRM assessment records joined to inventory tier
• Seat utilisation at Critical and High tier — SaaS management platform joined to identity provider activity; where none is in use, expense records reconciled against identity grants, recorded at E1

Level 2 Consumption Metrics

Absolute spend is a ledger input, never a metric. Every row is divided by a unit of business work. If AI spend is not allocated to an owning team, record this level as blocked on allocation rather than scoring it down (v3.5 framing §12).

Metric	Denominator	Current	Economic failure it detects	Met?
Cost per active seat by tier, with utilisation	per active seat	___	licence waste concentrated at high tiers, where an unused seat carries standing access to the vendors handling the most sensitive data	☐
Overage as a share of committed spend, by vendor	per governed workload	___	consumption growth arriving as an unbudgeted invoice, and pricing changes taking effect without the notice period being enforced	☐
Concentration: share of vendor AI consumption carried by the largest vendor and the largest underlying model provider	per dollar of vendor AI consumption	___	dependency accumulated by convenience rather than decided, where the underlying model provider concentration is often higher than the vendor concentration suggests	☐
Measured exit cost as a multiple of the vendor's annualised contract value	per Critical-tier vendor relationship	___	lock-in that is asserted as a worry but never priced, so the threat of exit carries no weight in negotiation and migration is never budgeted	☐

Level 2 30-second check (E0)

• ☐ Yes ☐ No — Does every AI vendor carry a risk tier?
• ☐ Yes ☐ No — Do you know what it would cost to leave your largest AI vendor?
• ☐ Yes ☐ No — Are you paying for AI seats nobody has used this month?

---

Maturity Level 3, Industry-Leading

Objective: Drive vendor discovery and tiering from live identity, expense and egress feeds, benchmark vendor unit economics, and contribute AI vendor assurance intelligence back.

Prerequisite cells: SM-Vendors-L2, ML-Vendors-L2, EG-Vendors-L2

Question L3.1: Make discovery continuous across identity, expense and egress

Q3.1: Is discovery continuous across identity-provider grants, expense feeds, egress telemetry and SaaS admin APIs, with adoption-to-inventory latency at or below 48 hours at the 90th percentile and at least 75% of entries auto-curated?

Evidence Required:

• [ ] Replace the monthly sweep with continuous feeds
• [ ] Stream identity-provider application grants and first-authentication events, so a new single-sign-on grant to an AI service raises a discovery event the same day
• [ ] Stream expense and corporate-card feeds filtered against a maintained AI-vendor merchant list
• [ ] Stream network egress against a maintained AI-provider domain list that is itself updated from threat-intelligence and community sources rather than by hand
• [ ] And poll the admin APIs of major approved SaaS products for newly enabled AI features, which remains the only reliable way to see AI that arrives without a transaction
• [ ] Wire tier derivation to re-run automatically when a vendor's recorded data classes, service scope or sub-processor chain changes, and to open downstream reassessment obligations on tier increase
• [ ] Publish a data-quality objective covering time from first adoption signal to inventory record, targeting 48 hours, and reserve human curation for ambiguous merchant records, shared-account adoptions and genuinely novel vendor archetypes

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L3.2: Benchmark vendor unit economics and assessment depth

Q3.2: Does the semi-annual benchmarking brief compare at least five peer-comparable metrics including cost per active seat and seat utilisation by archetype, with the cohort named and the deltas delivered to the renewal calendar before Critical-tier negotiations?

Evidence Required:

• [ ] Publish a semi-annual brief comparing the program against named peers on at least five metrics: vendor inventory coverage and discovery latency, shadow AI ratio, contractual-term conformance at Critical tier, cost per active seat by archetype, and seat utilisation
• [ ] Vendor unit economics benchmark unusually well because the products are the same across organizations, so a materially higher cost per active seat for the same archetype is a negotiating position rather than an observation, and a materially lower one usually indicates either better terms worth sharing or unused capacity worth reclaiming
• [ ] Draw comparison data from peer chief information security officer communities, sector information-sharing bodies, the FinOps Foundation SaaS and AI scopes, and Shared Assessments or equivalent third-party risk communities
• [ ] Name the cohort, defend its selection, and record where the program trails
• [ ] Feed the deltas directly into the renewal calendar so that comparisons arrive before negotiations rather than after them

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L3.3: Contribute assessment intelligence and compute the vendor return

Q3.3: Does the program contribute at least four externally accepted vendor assurance artifacts per year, and is the vendor return computed from invoice arithmetic covering reclaimed licence waste, avoided overage and consolidation savings, reconciled with Finance and Procurement?

Evidence Required:

• [ ] Contribute at least four substantive artifacts per year: AI vendor assessment criteria and questionnaire modules published through Shared Assessments, CSA AI Safety Initiative or sector bodies
• [ ] Observed vendor-layer techniques such as silently enabled AI features, training-data term changes on renewal, and sub-processor chain extensions submitted to MITRE ATLAS or community advisories
• [ ] Anonymized contractual-term benchmarks that help peers negotiate
• [ ] And OWASP LLM working-group input on third-party and supply-chain risks
• [ ] Count only externally accepted artifacts
• [ ] Compute the vendor-domain return from invoice and ledger arithmetic: reclaimed licence waste from deprovisioned unused seats, which is usually the largest and most defensible component
• [ ] Avoided overage where committed-spend governance caught consumption growth before invoicing
• [ ] Consolidation savings where duplicate archetype coverage was retired
• [ ] And avoided price increases where notice periods were enforced
• [ ] Reconcile with Finance and Procurement annually and publish alongside the domain Assurance Cost Ratio

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Level 3 Outcome Metrics

Metric	Baseline	Current	Target	Met?	Notes
Time from first adoption signal to an inventory record	___	___	≤48 hours at the 90th percentile	☐
Vendor inventory entries created without human curation	___	___	≥75% auto-curated	☐
Tier re-derivation triggered automatically on vendor scope or sub-processor change	___	___	100% within 5 business days of the notification	☐
Peer-comparable metrics in the semi-annual brief	___	___	≥5 metrics with a named and defended peer cohort	☐
Externally accepted vendor assurance contributions per year	___	___	≥4 accepted, excluding talks and press	☐

Where each number is read from:

• Time from first adoption signal to an inventory record — Discovery event pipeline telemetry
• Vendor inventory entries created without human curation — Curation telemetry from the discovery pipeline
• Tier re-derivation triggered automatically on vendor scope or sub-processor change — Tier derivation event log
• Peer-comparable metrics in the semi-annual brief — Published benchmarking brief
• Externally accepted vendor assurance contributions per year — Contribution log with external acceptance references

Level 3 Consumption Metrics

Absolute spend is a ledger input, never a metric. Every row is divided by a unit of business work. If AI spend is not allocated to an owning team, record this level as blocked on allocation rather than scoring it down (v3.5 framing §12).

Metric	Denominator	Current	Economic failure it detects	Met?
Cost per active seat by archetype, benchmarked against the peer cohort	per active seat	___	paying materially above the peer rate for an identical product, which is invisible internally because the invoice looks the same every month	☐
Reclaimed licence waste from deprovisioned unused seats	per active seat recovered	___	seats accumulating through joiner-mover-leaver drift, carrying both cost and standing access to vendor-held data	☐
Avoided overage under committed-spend governance	per governed workload	___	consumption growth crossing a commitment threshold unnoticed and arriving as an unbudgeted invoice at a punitive overage rate	☐

Level 3 30-second check (E0)

• ☐ Yes ☐ No — Would a new AI tool adopted today show up in your inventory this week?
• ☐ Yes ☐ No — Do you know whether you pay above or below peer rates for your AI tools?
• ☐ Yes ☐ No — Have you reclaimed unused AI seats this quarter?

---

Summary Scorecard

Level	Q1	Q2	Q3	Level score	Weakest evidence grade	Achieved?
L1	___	___	___	___	___	☐
L2	___	___	___	___	___	☐
L3	___	___	___	___	___	☐

Strategy & Metrics (SM) maturity in the Vendors domain: ____________

---

Document Version: HAIAMM v3.5 (generated)