# Strategy & Metrics (SM) - Vendors Domain
## HAIAMM Assessment Questionnaire v3.0

> Canonical source-of-truth: [`../practices/SM-Vendors-OnePager.md`](../practices/SM-Vendors-OnePager.md). This questionnaire's questions, evidence requirements, and outcome metrics are derived from that one-pager. The canonical v3.0 model: [`../HAIAMM-v3.0-Framing.md`](../HAIAMM-v3.0-Framing.md).

---

**Practice:** Strategy & Metrics (SM)
**Domain:** Vendors
**Purpose:** Stand up a Vendor AI Assurance program that discovers, inventories, and strategically governs all AI/HAI tools and services provided by vendors, with shadow AI prevention as the primary L1 outcome and a defensible risk-tier rubric as the primary L2 deliverable.
**Scoring Model:** Evidence + Outcome Metrics (see Scoring Methodology below)

---

## Instructions
- Answer each question honestly based on **current, implemented practices** (not plans or aspirations)
- Each question has two components: **Evidence** (what you did) and **Outcome Metrics** (how well it worked)
- **Scoring uses 4 tiers:** Fully Mature (1.0), Implemented (0.67), Partial (0.33), Not Implemented (0.0)
- **Answer progressively**, Complete all Level 1 questions before Level 2
- **Level progression**, Achieve ALL questions at lower level before advancing
- **Baseline first**, Record current metric values before setting targets

---

## Scoring Methodology

| Tier | Score | Criteria |
|---|---|---|
| **Fully Mature** | 1.0 | Evidence complete + ≥3 outcome metrics meet targets |
| **Implemented** | 0.67 | Evidence complete + 2 outcome metrics meet targets |
| **Partial** | 0.33 | Evidence partially complete + <2 outcome metrics meet targets |
| **Not Implemented** | 0.0 | No substantive evidence of practice |

Practice maturity level achieved = the highest level where all 3 questions score ≥ 0.67.

---

## Maturity Level 1
**Objective:** Stand up the Vendor AI Assurance program, build an AI vendor inventory, and establish baseline metrics that prove shadow AI is decreasing

### Question 1: Charter the Vendor AI Assurance program

**Q1.1:** Do you have a published Vendor AI Assurance program charter that names the problem (shadow AI, uncontrolled data sharing with AI vendors, unassessed third-party AI risk), defines the six in-scope AI vendor categories, names an executive sponsor (typically CISO, CIO, or Chief Procurement Officer, co-sponsored by Legal/Privacy), establishes a cross-functional working group, and defines decision rights for approval, block, and exception?

**Evidence Required:**
- [ ] Published program charter with named executive sponsor (CISO / CIO / CPO co-sponsored by Legal/Privacy)
- [ ] Problem statement covering rapid adoption, hidden AI features in existing SaaS, data-leakage by design, and regulatory exposure under EU AI Act deployer duties
- [ ] Six in-scope AI vendor categories listed: consumer GenAI, AI-embedded SaaS, AI coding assistants, AI APIs/models, AI agent platforms, AI-native point solutions
- [ ] Working group roster: Security, Procurement, Legal/Privacy, IT, Data Governance, one business-unit representative
- [ ] Decision rights defined: who approves an AI vendor, who blocks one, who handles exceptions
- [ ] Year-one success definition with a numerical target for L1 outcome metrics (e.g., "≥90% of AI vendors in use discovered and inventoried within 12 months")

**Outcome Metrics:**
| Metric | Baseline | Current | Target | Met? | Notes |
|---|---|---|---|---|---|
| AI vendor inventory coverage (% of discovered AI vendors in the inventory) | ___ | ___ | ≥90% within 12 months | ☐ | |
| Shadow AI ratio (unsanctioned AI vendors ÷ total AI vendors in use) | ___ | ___ | ≤15% and trending down | ☐ | |
| % employees covered by an acknowledged AI Acceptable Use Policy | ___ | ___ | ≥95% of workforce | ☐ | |
| Known data-exposure events to unsanctioned AI tools (per quarter) | ___ | ___ | trending down QoQ | ☐ | |

**Metric Collection Guidance:**
- **Inventory coverage**: Reconcile inventory count against all discovery-source signals (expense/procurement, SSO/IdP, DNS/egress, endpoint MDM/EDR, SaaS admin consoles). Formula: `inventory_count / discovered_count × 100`
- **Shadow AI ratio**: From inventory status field, count AI vendors in use with status not "Sanctioned"; divide by total AI vendors in use across all discovery sources
- **AUP attestation**: HR/LMS acknowledgment records for the AI Acceptable Use Policy; denominator is total workforce headcount; updated each quarter
- **Data-exposure events to unsanctioned AI**: Aggregate DLP alerts, incident-tracker entries for AI-related data-sharing events to unsanctioned tools; count per quarter and trend

**Answer:**
- ☐ **Fully Mature** (Evidence complete + ≥3 metrics meet targets)
- ☐ **Implemented** (Evidence complete + 2 metrics meet targets)
- ☐ **Partial** (Evidence partially complete + <2 metrics meet targets)
- ☐ **Not Implemented** (No evidence)

**Evidence Location:** _________________________________
**Metric Validation Date:** _________________________________
**Notes:** ___________________________________________

---

### Question 2: Build the AI vendor inventory and discover shadow AI

**Q1.2:** Do you maintain a single AI vendor inventory seeded from expense and procurement data, SSO/IdP sign-in logs, DNS/proxy egress logs for AI domains, MDM/EDR endpoint app inventory, and SaaS admin consoles for AI features in already-approved vendors, covering all six in-scope categories with a minimum field set including vendor name, AI capability, business owner, data classes sent, contract/DPA status, and approval status?

**Evidence Required:**
- [ ] Single authoritative AI vendor inventory with minimum fields: vendor name, product, AI capability, business owner, user count, data classes sent (public/internal/confidential/regulated), contract/DPA status and AI-specific addendum status, vendor-trains-on-customer-data flag, risk tier assignment, approval status (Sanctioned/Provisional/Under review/Prohibited)
- [ ] Expense and procurement data used as discovery signal: credit-card reports, SaaS expense platforms (Vendr, Zylo, Torii, Productiv), AP ledgers filtered for AI vendors
- [ ] SSO/IdP app catalog and sign-in logs (Okta/Entra) reviewed for AI SaaS sign-in activity
- [ ] DNS/proxy egress logs monitored for high-signal AI domains (openai.com, anthropic.com, gemini.google.com, cursor.sh, etc.); CASB used if available
- [ ] MDM/EDR endpoint app inventory filtered for AI desktop apps and browser extensions
- [ ] SaaS admin consoles audited for AI features in already-approved vendors (Notion AI, Slack AI, Zoom AI Companion, M365 Copilot, Google Workspace Gemini); amnesty window publicized through internal communications

**Outcome Metrics:**
| Metric | Baseline | Current | Target | Met? | Notes |
|---|---|---|---|---|---|
| AI vendor inventory coverage (% of discovered AI vendors in the inventory) | ___ | ___ | ≥90% within 12 months | ☐ | |
| Shadow AI ratio (unsanctioned AI vendors ÷ total AI vendors in use) | ___ | ___ | ≤15% and trending down | ☐ | |
| % employees covered by an acknowledged AI Acceptable Use Policy | ___ | ___ | ≥95% of workforce | ☐ | |
| Known data-exposure events to unsanctioned AI tools (per quarter) | ___ | ___ | trending down QoQ | ☐ | |

**Metric Collection Guidance:**
- **Inventory coverage**: Monthly reconciliation comparing inventory records to all discovery-source outputs; unmatched signals (new AI domain in egress, new app on MDM, new expense line) are shadow-AI candidates; report gap as coverage %
- **Shadow AI ratio**: Filter inventory for vendors in use with approval status != "Sanctioned"; divide by total in-use vendors across all sources; trend quarterly
- **AUP attestation**: HR/LMS query for workforce who completed AI AUP acknowledgment; denominator is total workforce headcount from HR system; updated each quarter
- **Data-exposure events**: DLP system events for AI-tool data-sharing incidents and incident-tracker AI-related entries; count per quarter; trend downward is the goal

**Answer:**
- ☐ **Fully Mature** (Evidence complete + ≥3 metrics meet targets)
- ☐ **Implemented** (Evidence complete + 2 metrics meet targets)
- ☐ **Partial** (Evidence partially complete + <2 metrics meet targets)
- ☐ **Not Implemented** (No evidence)

**Evidence Location:** _________________________________
**Metric Validation Date:** _________________________________
**Notes:** ___________________________________________

---

### Question 3: Establish foundational metrics and the shadow AI scoreboard

**Q1.3:** Do you baseline and report quarterly to the executive sponsor a shadow AI scoreboard covering inventory state (total/sanctioned/provisional/prohibited), new vendors discovered and their intake status, shadow AI ratio trend over the last four quarters, AUP attestation coverage, and the top five data-exposure risks to unsanctioned AI tools with remediation owners?

**Evidence Required:**
- [ ] Quarterly shadow AI scoreboard published and delivered to the executive sponsor, at least two consecutive quarters on record
- [ ] Scoreboard covers: AI vendors in inventory (total/sanctioned/provisional/prohibited), new AI vendors discovered this quarter and intake status, shadow AI ratio trend (last 4 quarters), AUP attestation coverage, top 5 data-exposure risks to unsanctioned AI tools with remediation owner
- [ ] Shadow AI ratio trended over last 4 quarters with commentary on direction
- [ ] AUP attestation percentage reported with workforce headcount denominator
- [ ] Intake SLA tracked: new AI vendor intake triaged within 5 business days; inventory freshness at ≥80% of records reviewed/updated in the last 90 days

**Outcome Metrics:**
| Metric | Baseline | Current | Target | Met? | Notes |
|---|---|---|---|---|---|
| AI vendor inventory coverage (% of discovered AI vendors in the inventory) | ___ | ___ | ≥90% within 12 months | ☐ | |
| Shadow AI ratio (unsanctioned AI vendors ÷ total AI vendors in use) | ___ | ___ | ≤15% and trending down | ☐ | |
| % employees covered by an acknowledged AI Acceptable Use Policy | ___ | ___ | ≥95% of workforce | ☐ | |
| Known data-exposure events to unsanctioned AI tools (per quarter) | ___ | ___ | trending down QoQ | ☐ | |

**Metric Collection Guidance:**
- **Scoreboard delivery cadence**: Confirm last two quarters have a dated scoreboard delivered to exec sponsor with their acknowledgment on record
- **Shadow AI ratio trend**: Four-quarter chart or table; downward trend is the L1 success signal; source is inventory status field reconciled monthly against discovery sweeps
- **AUP attestation**: Percentage with workforce headcount denominator explicitly stated; HR/LMS is the authoritative source; updated each quarter
- **Top-5 data-exposure risks**: Each entry lists the unsanctioned vendor or data-exposure scenario, named remediation owner, and current status (open / in-progress / mitigated)
- **Intake SLA**: Intake tracker report showing % of new vendor requests triaged within 5 BD; any backlog older than 5 BD flagged

**Answer:**
- ☐ **Fully Mature** (Evidence complete + ≥3 metrics meet targets)
- ☐ **Implemented** (Evidence complete + 2 metrics meet targets)
- ☐ **Partial** (Evidence partially complete + <2 metrics meet targets)
- ☐ **Not Implemented** (No evidence)

**Evidence Location:** _________________________________
**Metric Validation Date:** _________________________________
**Notes:** ___________________________________________

---

## Maturity Level 2
**Objective:** Risk-tier the AI vendor inventory, calibrate the program's intensity per tier, and measure practice maturity and shadow-AI reduction per tier, not only in aggregate

### Question 1: Define the AI vendor risk-tier rubric

**Q2.1:** Do you have a published risk-tier rubric (Critical / High / Medium / Low) assigning a tier to every AI vendor in the inventory based on six auditable dimensions, data sensitivity reaching the vendor, decision-affecting use (GDPR Art. 22 / EU AI Act Annex III), agentic capability, user exposure, regulatory scope, and concentration/criticality, with tier derivation deterministic from inputs, human overrides recorded with rationale, and 100% of inventory records carrying a current tier?

**Evidence Required:**
- [ ] Published tier-rubric document listing all six auditable dimensions with deterministic assignment logic (not reviewer vibes)
- [ ] 100% of inventory records carry a current tier assignment derived from the rubric
- [ ] Data sensitivity dimension: regulated data (PHI/PCI/regulated PII/source code/customer confidential) reaching the vendor → Critical or High
- [ ] Decision-affecting use dimension: EU AI Act Annex III high-risk use or GDPR Art. 22 materially affected decision → Critical
- [ ] Agentic capability dimension: agent platform or multi-tool function surface acting on org systems → elevate tier
- [ ] Regulatory scope dimension: sector-specific triggers (HIPAA BAA, PCI service provider, FINRA/SEC model risk, HHS/FDA clinical) → elevate
- [ ] Human override log maintained: all overrides recorded with rationale and reviewed by working group

**Outcome Metrics:**
| Metric | Baseline | Current | Target | Met? | Notes |
|---|---|---|---|---|---|
| % of inventory with a current tier assignment | ___ | ___ | 100% | ☐ | |
| Tier-treatment matrix adherence, % Critical vendors with full-scope treatment completed in last 12 months | ___ | ___ | ≥95% | ☐ | |
| Tier-weighted shadow AI ratio (Critical-weighted) | ___ | ___ | Critical = 0 unsanctioned; overall trending down | ☐ | |
| Per-tier SLA adherence (intake, DR, IR, ST, ML) | ___ | ___ | ≥90% per tier | ☐ | |
| Tier drift rate (tier changes per year) | ___ | ___ | tracked; unexplained changes = 0 | ☐ | |

**Metric Collection Guidance:**
- **% inventory with tier assignment**: Automated check, records with null risk_tier flagged for remediation within 5 BD; stale tier (not re-confirmed after a material change) also flagged
- **Tier-treatment matrix adherence**: Cross-reference Critical-tier vendor records against evidence of full-scope treatment: full SR pack completed, per-vendor TA deep model completed, DR completed, IR within cadence, full ST battery complete
- **Tier-weighted shadow AI ratio**: Critical-tier vendors in use with status != "Sanctioned" must be 0; overall ratio across all tiers should trend down; both reported in the quarterly scoreboard
- **Per-tier SLA adherence**: From intake tracker, DR queue, IR schedule, ST tracker, and ML system; % of SLA-bound actions completed on time per tier; report monthly
- **Tier drift rate**: Governance log reviewed at each working-group meeting; any change without a recorded dimension-change rationale is unexplained; target = 0

**Answer:**
- ☐ **Fully Mature** (Evidence complete + ≥3 metrics meet targets)
- ☐ **Implemented** (Evidence complete + 2 metrics meet targets)
- ☐ **Partial** (Evidence partially complete + <2 metrics meet targets)
- ☐ **Not Implemented** (No evidence)

**Evidence Location:** _________________________________
**Metric Validation Date:** _________________________________
**Notes:** ___________________________________________

---

### Question 2: Calibrate program intensity per tier

**Q2.2:** Do you have a published tier-treatment matrix defining differential program intensity across all downstream Vendors-domain practices (PC, TA, DR, IR, ST, ML) for each tier, and is this matrix enforced, with Critical-tier vendors receiving full SR pack + full REM at intake, a per-vendor deep TA model, full-lane DR, semi-annual IR plus review on material change, a full ST battery with quarterly red-team probe, all detections tuned, and mandatory re-review within 14 days of any vendor material change?

**Evidence Required:**
- [ ] Tier-treatment matrix published covering all downstream practices with explicit treatment per tier
- [ ] Critical-tier treatment documented: full SR pack + full REM; per-vendor deep TA model; full-lane DR required; semi-annual IR + on change; full ST battery + quarterly red-team probe; all detections tuned for the integration; mandatory re-review within 14 days of material change
- [ ] Low-tier fast-track documented: base SR pack only; archetype TA snapshot; no DR required; go-live IR only; spot-check ST; baseline logging; re-review at annual review only
- [ ] Evidence that Critical-tier vendors are actually receiving full-scope treatment (vendor × treatment evidence in program tracker for ≥95% of Critical vendors in last 12 months)
- [ ] Downstream practices (PC, TA, DR, IR, ST, ML) each acknowledged the calibration via working-group decision record

**Outcome Metrics:**
| Metric | Baseline | Current | Target | Met? | Notes |
|---|---|---|---|---|---|
| % of inventory with a current tier assignment | ___ | ___ | 100% | ☐ | |
| Tier-treatment matrix adherence, % Critical vendors with full-scope treatment completed in last 12 months | ___ | ___ | ≥95% | ☐ | |
| Tier-weighted shadow AI ratio (Critical-weighted) | ___ | ___ | Critical = 0 unsanctioned; overall trending down | ☐ | |
| Per-tier SLA adherence (intake, DR, IR, ST, ML) | ___ | ___ | ≥90% per tier | ☐ | |
| Tier drift rate (tier changes per year) | ___ | ___ | tracked; unexplained changes = 0 | ☐ | |

**Metric Collection Guidance:**
- **Tier-treatment matrix adherence**: Build a cross-reference table, for each Critical-tier vendor, list last DR date, last IR date, ST coverage status, ML detection status; ≥95% must show all Critical-tier treatments completed in last 12 months
- **Per-tier SLA adherence**: Aggregate SLA data from intake, DR, IR, and ST trackers; calculate on-time completion % per tier; report monthly
- **Tier-weighted shadow AI ratio**: Critical-tier unsanctioned vendors must be 0; overall shadow AI ratio should decrease; both reported in quarterly scoreboard
- **Tier drift rate**: Governance log cross-checked at each working-group meeting; unexplained tier changes (no dimension-change rationale) flagged for sponsor visibility; target = 0

**Answer:**
- ☐ **Fully Mature** (Evidence complete + ≥3 metrics meet targets)
- ☐ **Implemented** (Evidence complete + 2 metrics meet targets)
- ☐ **Partial** (Evidence partially complete + <2 metrics meet targets)
- ☐ **Not Implemented** (No evidence)

**Evidence Location:** _________________________________
**Metric Validation Date:** _________________________________
**Notes:** ___________________________________________

---

### Question 3: Per-tier scoreboard and governance

**Q2.3:** Does the quarterly shadow AI scoreboard report inventory state per tier with Critical-tier unsanctioned AI explicitly tracked at zero, include a tier-movement log with rationale, report per-tier SLA adherence for intake/DR/IR/ST/ML, and is it reviewed by the executive sponsor who explicitly discusses tier-balance?

**Evidence Required:**
- [ ] Quarterly scoreboard includes a tier-level breakdown showing vendors by tier (Critical/High/Medium/Low) with sanctioned/provisional/prohibited counts per tier
- [ ] Critical-tier unsanctioned AI vendors in use is a named metric in the scoreboard; target is 0; any non-zero value is a headline finding requiring sponsor action
- [ ] Tier-movement log included: vendors that moved up or down in the quarter, with dimension(s) that changed and rationale for each move
- [ ] SLA adherence per tier reported for intake, DR, IR, ST, and ML
- [ ] Quarterly executive review documented (agenda + minutes) showing tier-balance discussion: is the program's effort matching the risk profile?
- [ ] Tier calibration exercise documented: at least quarterly, a sample of 20 vendors re-tiered by a second reviewer; drift tracked

**Outcome Metrics:**
| Metric | Baseline | Current | Target | Met? | Notes |
|---|---|---|---|---|---|
| % of inventory with a current tier assignment | ___ | ___ | 100% | ☐ | |
| Tier-treatment matrix adherence, % Critical vendors with full-scope treatment completed in last 12 months | ___ | ___ | ≥95% | ☐ | |
| Tier-weighted shadow AI ratio (Critical-weighted) | ___ | ___ | Critical = 0 unsanctioned; overall trending down | ☐ | |
| Per-tier SLA adherence (intake, DR, IR, ST, ML) | ___ | ___ | ≥90% per tier | ☐ | |
| Tier drift rate (tier changes per year) | ___ | ___ | tracked; unexplained changes = 0 | ☐ | |

**Metric Collection Guidance:**
- **Per-tier scoreboard delivery**: Last two consecutive quarterly scoreboards must include a tier-level breakdown (not just aggregate); each scoreboard shows delta from prior quarter
- **Critical-tier unsanctioned count**: Named metric in each scoreboard; source is inventory filtered on tier=Critical AND status != Sanctioned AND in-use; target = 0
- **Tier-movement log completeness**: Each entry must have vendor name, prior tier, new tier, dimension(s) that changed, reviewer name, and date; unexplained changes target = 0
- **SLA adherence per tier**: Pulled from intake, DR, IR, ST, and ML systems; aggregated per tier; reported as % on-time per tier per quarter
- **Executive review**: Filed governance document confirming exec sponsor reviewed tier-balance section and issued follow-up actions or signed off

**Answer:**
- ☐ **Fully Mature** (Evidence complete + ≥3 metrics meet targets)
- ☐ **Implemented** (Evidence complete + 2 metrics meet targets)
- ☐ **Partial** (Evidence partially complete + <2 metrics meet targets)
- ☐ **Not Implemented** (No evidence)

**Evidence Location:** _________________________________
**Metric Validation Date:** _________________________________
**Notes:** ___________________________________________

---

## Maturity Level 3
**Objective:** Automate inventory and tier maintenance from live signals, benchmark the program against external peers, and contribute anonymized AI-vendor ecosystem intelligence back to the industry

### Question 1: Continuous inventory and tier automation

**Q3.1:** Does the AI vendor inventory auto-update from live expense/procurement feeds, SSO/IdP app catalog, DNS/egress telemetry, SaaS admin audit feeds, endpoint AI-tool inventory, and intake, with tier assignments rule-based on versioned, replayable logic, tier changes adjusting automatically when dimensional inputs change, and a published data-quality SLO of ≥99% of active AI vendors correctly tiered within 48 hours of a material change?

**Evidence Required:**
- [ ] Published data-quality SLO: ≥99% of active AI vendors correctly tiered within 48 hours of a material change
- [ ] Automated feeds operational: expense/procurement feeds, SSO/IdP app catalog, DNS/egress telemetry, SaaS admin audit feeds (for parent vendors like M365, Slack, Notion), endpoint AI-tool inventory from MDM/EDR, intake system, self-attestation
- [ ] Tier rules documented as versioned, replayable logic; rule changes are change-logged and can be replayed against historical inventory state
- [ ] Human curation queue defined for: new archetypes, ambiguous discoveries, dimensional-input conflicts
- [ ] Automation health dashboard: signal-feed freshness and error rate monitored; on-call paged when a feed staleness threshold is exceeded
- [ ] Tier changes adjust automatically when dimensional inputs change (e.g., a vendor gains agentic capability, data sensitivity classification changes); humans intervene only on exception cases

**Outcome Metrics:**
| Metric | Baseline | Current | Target | Met? | Notes |
|---|---|---|---|---|---|
| Inventory auto-update latency | ___ | ___ | ≤48h for material changes | ☐ | |
| % inventory entries auto-curated vs. human-curated | ___ | ___ | ≥80% auto | ☐ | |
| External benchmarks tracked | ___ | ___ | ≥5 | ☐ | |
| Industry contributions per year | ___ | ___ | ≥4 substantive | ☐ | |
| Executive-level ROI narrative refreshed with external benchmarks | ___ | ___ | semi-annual | ☐ | |

**Metric Collection Guidance:**
- **Auto-update latency**: Measure time from a known material change event (new AI vendor approved in SSO, new AI domain appears in egress logs, new expense line for an AI vendor) to the corresponding inventory record update; P95 across sampled events per quarter
- **% auto-curated**: From the inventory curation log, count records updated by automated feeds vs. human-initiated edits; report as a ratio per quarter
- **Automation health**: Signal-feed freshness report showing last successful pull time for each feed; alert thresholds defined and on-call paging tested
- **Data-quality SLO adherence**: At each quarter-end, spot-check 20 vendors against their expected tier from the rubric; report % correctly tiered

**Answer:**
- ☐ **Fully Mature** (Evidence complete + ≥3 metrics meet targets)
- ☐ **Implemented** (Evidence complete + 2 metrics meet targets)
- ☐ **Partial** (Evidence partially complete + <2 metrics meet targets)
- ☐ **Not Implemented** (No evidence)

**Evidence Location:** _________________________________
**Metric Validation Date:** _________________________________
**Notes:** ___________________________________________

---

### Question 2: External benchmarking

**Q3.2:** Do you publish a semi-annual external-benchmarking brief comparing the program against at least five peer-comparable metrics via sector ISACs with AI-vendor working groups, ISO/IEC 42001 community, NIST AI RMF implementations, CSA AI Safety Initiative, Shared Assessments AI-vendor track, and formal peer roundtables, and do benchmark deltas explicitly inform program investment decisions?

**Evidence Required:**
- [ ] Semi-annual benchmarking brief published, two most recent on file with dates, each containing ≥5 peer-comparable metrics from named external sources
- [ ] Benchmarking sources include at least two of: sector ISACs (FS-ISAC, H-ISAC, IT-ISAC) AI-vendor working groups / ISO/IEC 42001 community / NIST AI RMF implementations / CSA AI Safety Initiative / Shared Assessments AI-vendor track / formal peer roundtables
- [ ] Metrics benchmarked cover: shadow AI ratio, AUP coverage, intake SLA, per-tier depth of treatment, automation level, and procurement cycle time for approved AI vendors
- [ ] Benchmark deltas explicitly referenced in a program investment or prioritization decision; documentation filed within 90 days of each brief
- [ ] "How we compare" brief refreshed semi-annually; peer selection rationale documented, peers chosen to stretch, not flatter

**Outcome Metrics:**
| Metric | Baseline | Current | Target | Met? | Notes |
|---|---|---|---|---|---|
| Inventory auto-update latency | ___ | ___ | ≤48h for material changes | ☐ | |
| % inventory entries auto-curated vs. human-curated | ___ | ___ | ≥80% auto | ☐ | |
| External benchmarks tracked | ___ | ___ | ≥5 | ☐ | |
| Industry contributions per year | ___ | ___ | ≥4 substantive | ☐ | |
| Executive-level ROI narrative refreshed with external benchmarks | ___ | ___ | semi-annual | ☐ | |

**Metric Collection Guidance:**
- **External benchmarks tracked**: Each brief lists ≥5 named benchmark data points; each traceable to an ISAC report, ISO/IEC 42001 community publication, NIST AI RMF guidance, CSA controls document, or named peer roundtable
- **Benchmark-driven investment**: Program planning or budget document explicitly citing a benchmark delta as rationale; filed within 90 days of each brief
- **Semi-annual cadence**: Two briefs within a 12-month window; no gap > 7 months between consecutive briefs
- **Executive ROI narrative**: Annual exec/board briefing deck or memo includes benchmark comparisons and avoided-loss examples; procurement cycle time for approved AI vendors is a key effectiveness metric

**Answer:**
- ☐ **Fully Mature** (Evidence complete + ≥3 metrics meet targets)
- ☐ **Implemented** (Evidence complete + 2 metrics meet targets)
- ☐ **Partial** (Evidence partially complete + <2 metrics meet targets)
- ☐ **Not Implemented** (No evidence)

**Evidence Location:** _________________________________
**Metric Validation Date:** _________________________________
**Notes:** ___________________________________________

---

### Question 3: Contribute anonymized AI-vendor ecosystem intelligence

**Q3.3:** Does the program contribute at least four substantive, anonymized artifacts per year to the AI-vendor assurance ecosystem through MITRE ATLAS, OWASP LLM / Agentic Top 10, NIST AI RMF Playbook, AI Vulnerability Database submissions, or sector ISAC feeds, with each contribution anonymized, legally vetted, and traceable to a published working-group output or advisory?

**Evidence Required:**
- [ ] Contribution log maintained listing all submissions: target body (MITRE ATLAS, OWASP, NIST, AVID, ISAC), submission type (new AI-vendor-ecosystem TTP, real-world telemetry, discovery signal pattern, detection pattern), date submitted, anonymization review completed, status
- [ ] At least 4 substantive contributions per year in the most recent 12-month window; each is a technical artifact accepted or in active review by the named body; conference talks and press releases do not count
- [ ] Each contribution has a legal/privacy review sign-off confirming anonymization before submission
- [ ] Contributions traceable to published outputs: MITRE ATLAS technique entries, OWASP review comments incorporated, NIST AI RMF Playbook references, AVID entries, ISAC advisory feeds
- [ ] Contribution pipeline shows ≥2 items in-flight (draft, in-review, or being prepared) at any working-group review

**Outcome Metrics:**
| Metric | Baseline | Current | Target | Met? | Notes |
|---|---|---|---|---|---|
| Inventory auto-update latency | ___ | ___ | ≤48h for material changes | ☐ | |
| % inventory entries auto-curated vs. human-curated | ___ | ___ | ≥80% auto | ☐ | |
| External benchmarks tracked | ___ | ___ | ≥5 | ☐ | |
| Industry contributions per year | ___ | ___ | ≥4 substantive | ☐ | |
| Executive-level ROI narrative refreshed with external benchmarks | ___ | ___ | semi-annual | ☐ | |

**Metric Collection Guidance:**
- **Industry contributions per year**: Count entries in the contribution log for trailing 12 months where status = submitted or accepted to a named body; only substantive technical artifacts count
- **Contribution pipeline health**: At any working-group meeting, pipeline log shows ≥2 items not yet in submitted status; noted in working-group minutes
- **Legal/privacy review**: Each contribution log entry must have reviewer name and date; no contribution submitted without this sign-off
- **Executive ROI narrative**: Filed annually to exec/board; references external benchmarks and avoided-loss examples (data-exposure incidents avoided, regulatory exposure mitigated, procurement cycle time reduced); sponsor decisions citing benchmark data indicate program maturity

**Answer:**
- ☐ **Fully Mature** (Evidence complete + ≥3 metrics meet targets)
- ☐ **Implemented** (Evidence complete + 2 metrics meet targets)
- ☐ **Partial** (Evidence partially complete + <2 metrics meet targets)
- ☐ **Not Implemented** (No evidence)

**Evidence Location:** _________________________________
**Metric Validation Date:** _________________________________
**Notes:** ___________________________________________

---

## Summary Scorecard

| Level | Q1 | Q2 | Q3 | Avg | Achieved? |
|---|---|---|---|---|---|
| L1 | __ | __ | __ | __ | ☐ |
| L2 | __ | __ | __ | __ | ☐ |
| L3 | __ | __ | __ | __ | ☐ |

**Practice maturity level achieved:** ___ (highest level where all 3 questions score ≥ 0.67)

---

**Document Version:** HAIAMM v3.0
**Practice:** Strategy & Metrics (SM)
**Domain:** Vendors
**Last Updated:** 2026-05-15
**Author:** Verifhai