> **v3.0 framing:** The canonical source-of-truth for Security Testing (ST) in the Vendors domain is [`../practices/ST-Vendors-OnePager.md`](../practices/ST-Vendors-OnePager.md). Outcome metrics, activities, and success criteria below are verbatim from that one-pager. Canonical subject: Vendor AI Assurance, security of AI/HAI tools the organization consumes from vendors, with shadow-AI discovery and per-integration test battery as the primary L1 outcomes. Canonical subject and through-lines: [`../HAIAMM-v3.0-Framing.md`](../HAIAMM-v3.0-Framing.md).

---

# Security Testing (ST) - Vendors Domain
## HAIAMM Assessment Questionnaire v3.0

**Practice:** Security Testing (ST)
**Domain:** Vendors
**Purpose:** Assess organizational maturity in running per-integration acceptance test batteries and shadow-AI discovery tests against AI vendor integrations, from foundational per-archetype batteries through continuous automated red-teaming
**Scoring Model:** Evidence + Outcome Metrics (see Scoring Methodology below)

---

## Instructions

- Answer each question honestly based on **current, implemented practices** (not plans or aspirations)
- Each question has two components: **Evidence** (what you did) and **Outcome Metrics** (how well it worked)
- **Scoring uses 4 tiers:** Fully Mature (1.0), Implemented (0.67), Partial (0.33), Not Implemented (0.0)
- **Answer progressively** - Complete all Level 1 questions before Level 2
- **Level progression** - Achieve ALL questions at lower level before advancing
- **Baseline first** - Record current metric values before setting targets

---

## Scoring Methodology

| Score | Label | Criteria |
|-------|-------|----------|
| 1.0 | Fully Mature | Evidence complete + ≥3 outcome metrics meet targets |
| 0.67 | Implemented | Evidence complete + 2 outcome metrics meet targets |
| 0.33 | Partial | Evidence partially complete + <2 outcome metrics meet targets |
| 0.0 | Not Implemented | No evidence of the activity |

**Level Score** = average of the 3 question scores for that level
**Overall ST-Vendors Score** = weighted average: L1 × 0.5 + L2 × 0.3 + L3 × 0.2

---

## Maturity Level 1
**Objective:** Run a foundational AI-vendor test battery at go-live and quarterly; operate program-level shadow-AI discovery tests at least quarterly; feed findings into IM

---

### Question 1: Per-Archetype Foundational Test Battery

**Q1.1:** Is a per-archetype foundational test battery published with ≤6 test classes per archetype, covering data-egress canary, no-train verification, prompt-injection probe, permission-boundary/tool-scope test, logging-completeness test, kill-switch/rate-limit test, and toggle-drift test, each with defined inputs, pass/fail criteria, and evidence artifact, and are 100% of new integrations required to pass the battery before production?

**Evidence Required:**
- [ ] Test battery document published per archetype (≤6 test classes per archetype); covers per-archetype probes including data-egress canary (send synthetic tagged payload; verify logging, DLP interception, and vendor-side retention behavior), no-train verification (admin-console state + DPA reference + behavioral probe where testable), prompt-injection resilience probe (curated set of prompt-injection test strings; verify system prompts and tool permissions hold), permission-boundary/tool-scope test for agent archetype (attempt actions outside allowlist; verify deny + log), logging-completeness test (verify every required event type produces a log line in the org-side store with correct attribution), kill-switch/rate-limit test (exercise kill-switch or rate-limit path; verify behavior), and toggle-drift test (re-check toggle state and users-with-access list vs. approved scope)
- [ ] Each test class records: inputs, expected output, pass/fail criteria, evidence artifact (log snippet, screenshot, or trace ID)
- [ ] Each test class mapped to a TA-Vendors threat and an SR-Vendors requirement
- [ ] Battery linked from DR-Vendors and IR-Vendors artifacts for each AI vendor integration
- [ ] Named battery owner per archetype documented; go-live battery run required before production cutover
- [ ] 100% of AI vendor integrations reaching production in the last 90 days have a passed go-live battery on record

**Outcome Metrics:**
| Metric | Baseline | Current | Target | Met? | Notes |
|--------|----------|---------|--------|------|-------|
| % active AI vendor integrations with a current-quarter test-battery pass | ___% | ___% | ≥90% | ☐ | |
| % archetypes with a published L1 test battery | ___/5 | ___/5 | 5/5 | ☐ | |
| % of test failures converted to an IM issue within 1 business day | ___% | ___% | 100% | ☐ | |
| CI automation coverage of battery items (% running without human intervention) | ___% | ___% | ≥60% | ☐ | |

**Metric Collection Guidance:**
- **Current-quarter pass rate**: Count active integrations with a battery run record dated within the current quarter vs. total active integrations. Source: test-run registry
- **Archetype battery publication**: Count archetypes with a published, linked test battery document vs. target archetype count. Source: test library
- **IM routing rate**: For test failures in the last quarter, check IM ticket creation timestamps. Formula: `failures_with_IM_ticket_within_1BD / total_failures × 100`. Source: test failure log × IM system
- **CI automation**: Count battery items with an automated trigger vs. total battery items. Source: battery registry + CI configuration

**Answer:**
- ☐ **Fully Mature** (Evidence complete + ≥3 metrics meet targets)
- ☐ **Implemented** (Evidence complete + 2 metrics meet targets)
- ☐ **Partial** (Evidence partially complete + <2 metrics meet targets)
- ☐ **Not Implemented** (No evidence of per-archetype test battery)

**Evidence Location:** _______________
**Validation Date:** _______________
**Notes:** _______________

---

### Question 2: Quarterly Battery Re-Run and Triggers

**Q2.1:** Is the battery re-run at least quarterly for all active AI vendor integrations, with post-change and post-incident re-run triggers wired to the change-management and IM processes, and are ≥90% of integrations carrying a current-quarter pass?

**Evidence Required:**
- [ ] Battery re-run schedule on calendar: quarterly runs for all active AI vendor integrations; run records on file for the last two quarters
- [ ] Post-change re-run trigger confirmed: vendor major version bump, model-family change, admin-console redesign, or org-plan migration triggers a re-run; evidence: at least one post-change re-run record in the last 6 months
- [ ] Post-incident re-run trigger confirmed: any IM-Vendors incident involving the integration triggers a re-run of the relevant subset before incident closure; evidence: at least one post-incident re-run record in the last 6 months (or confirmation that no relevant incidents occurred)
- [ ] Regression corpora for the vendor battery versioned in source control with a named corpus owner; monthly refresh cadence evidenced through change-log showing updates from internal observations (IR-Vendors findings, IM-Vendors incidents) and external sources (OWASP LLM Top 10, ATLAS technique examples)
- [ ] Named battery owner per archetype; battery automation coverage ≥60%
- [ ] ≥90% of active AI vendor integrations carry a current-quarter battery pass

**Outcome Metrics:**
| Metric | Baseline | Current | Target | Met? | Notes |
|--------|----------|---------|--------|------|-------|
| % active AI vendor integrations with a current-quarter test-battery pass | ___% | ___% | ≥90% | ☐ | |
| % post-change re-runs completed within 14 days of change | ___% | ___% | ≥90% | ☐ | |
| % post-incident re-runs completed before incident closure | ___% | ___% | 100% | ☐ | |
| Corpus refresh cadence, months since last update | ___ | ___ | ≤1 month | ☐ | |

**Metric Collection Guidance:**
- **Current-quarter pass rate**: Test-run registry filtered to current quarter vs. active integration count. Source: test-run registry
- **Post-change re-run**: For vendor changes in the last 6 months, check whether a re-run was triggered and completed within 14 days. Source: change management system × test-run registry
- **Post-incident re-run**: For IM-Vendors incidents in the last 6 months, verify re-run was completed before incident closure. Source: IM-Vendors × test-run registry
- **Corpus refresh**: Inspect git log for battery corpora files; compute days since last commit adding new entries. Source: VCS change-log

**Answer:**
- ☐ **Fully Mature** (Evidence complete + ≥3 metrics meet targets)
- ☐ **Implemented** (Evidence complete + 2 metrics meet targets)
- ☐ **Partial** (Evidence partially complete + <2 metrics meet targets)
- ☐ **Not Implemented** (No quarterly re-run cadence in place)

**Evidence Location:** _______________
**Validation Date:** _______________
**Notes:** _______________

---

### Question 3: Shadow-AI Discovery Tests

**Q3.1:** Are shadow-AI discovery exercises run at least quarterly, validating that synthetic unsanctioned-AI scenarios are detected within SLA, with failures feeding ML-Vendors' detection backlog and results reviewed by the program sponsor?

**Evidence Required:**
- [ ] Shadow-AI discovery exercise conducted at least quarterly; exercise records on file for the last two quarters
- [ ] Exercise scope covers four synthetic scenarios (with sponsor approval and scope limits): test account attempts to pay for an unsanctioned consumer GenAI subscription via expense pathways; test endpoint downloads a known-AI-tool installer; test SaaS admin toggles on an AI feature in a sandbox workspace; test egress to a known AI vendor domain from an unmanaged path
- [ ] Each scenario records: pass/fail result and time-to-detect measurement
- [ ] Failures feed ML-Vendors' detection backlog: scenario failures result in IM-Vendors detection-gap tickets with named owner and remediation plan
- [ ] Results reviewed by the program sponsor within 5 business days of exercise completion; review artifact on file
- [ ] Median time-to-detect for shadow-AI scenarios measured and trending toward target

**Outcome Metrics:**
| Metric | Baseline | Current | Target | Met? | Notes |
|--------|----------|---------|--------|------|-------|
| Shadow-AI discovery test pass rate (scenarios detected within SLA) | ___% | ___% | ≥80% by end of year 1 | ☐ | |
| Median time-to-detect in shadow-AI tests | ___ days | ___ days | ≤14 days | ☐ | |
| % shadow-AI scenario failures converted to IM-Vendors detection-gap tickets within 1 BD | ___% | ___% | 100% | ☐ | |
| Shadow-AI exercise cadence, quarters since last exercise | ___ | ___ | ≤1 quarter | ☐ | |

**Metric Collection Guidance:**
- **Shadow-AI pass rate**: For each scenario in the last exercise, count scenarios where detection fired within the declared SLA vs. total scenarios. Formula: `detected_within_SLA / total_scenarios × 100`. Source: quarterly exercise results
- **Median time-to-detect**: Median detection time across scenarios in the last exercise. Source: exercise telemetry
- **Detection-gap ticket rate**: For failed scenarios, count those with an IM-Vendors detection-gap ticket created within 1 business day. Source: exercise failure log × IM-Vendors system
- **Exercise cadence**: Date of last exercise vs. current date. Source: exercise log

**Answer:**
- ☐ **Fully Mature** (Evidence complete + ≥3 metrics meet targets)
- ☐ **Implemented** (Evidence complete + 2 metrics meet targets)
- ☐ **Partial** (Evidence partially complete + <2 metrics meet targets)
- ☐ **Not Implemented** (No shadow-AI discovery exercise program in place)

**Evidence Location:** _______________
**Validation Date:** _______________
**Notes:** _______________

---

## Maturity Level 2
**Objective:** Dedicated AI-vendor red team for Critical tier quarterly; maintained regression corpus for jailbreaks and prompt-injection; bug-bounty integration where applicable

---

### Question 4: Quarterly Red-Team for Critical-Tier Integrations

**Q4.1:** Are 100% of Critical-tier AI vendor integrations red-teamed at least quarterly with scope covering prompt-injection chains, indirect-prompt-injection via RAG, agent tool abuse, jailbreak regression, and data-egress canaries, with findings routed to IM and remediation tracked?

**Evidence Required:**
- [ ] Red-team schedule on calendar covering all Critical-tier AI vendor integrations quarterly; no Critical-tier integration skipped in the last 12 months
- [ ] Red-team scope documented per exercise: written rules of engagement, test plan reviewed with integration owner, scope derived from TA-Vendors L2 per-vendor threat models; covers prompt-injection chains (ATLAS TA0001/TA0003), indirect-prompt-injection via RAG retrieval, agent tool abuse (ATLAS TA0004), jailbreak regression, and data-egress canaries (ATLAS TA0013)
- [ ] ATLAS tactic IDs referenced in scope documentation: TA0001 Reconnaissance, TA0003 Initial Access, TA0004 ML Model Access, TA0012 ML Attack Staging, TA0013 Exfiltration
- [ ] Red-team execution log and structured findings report on file: severity, root cause, ATLAS tactic ID, SR-Vendors requirement traced, remediation recommendation
- [ ] Findings routed to IM-Vendors with severity tag and named integration owner as assignee; remediation tracked
- [ ] Scheduled red-team exercises confirmed: quarterly for Critical, no Critical integration skipped

**Outcome Metrics:**
| Metric | Baseline | Current | Target | Met? | Notes |
|--------|----------|---------|--------|------|-------|
| % Critical integrations red-teamed in last 90 days | ___% | ___% | 100% | ☐ | |
| % High/Critical integrations running regression corpus weekly in CI | ___% | ___% | ≥90% | ☐ | |
| Regression corpus size / change rate | ___ | ___ | growing; ≥1 update/month | ☐ | |
| % red-team findings (Critical/High severity) converted to corpus entries within 30 days | ___% | ___% | ≥90% | ☐ | |

**Metric Collection Guidance:**
- **Critical red-team rate**: Count Critical-tier integrations with a red-team report dated within the last 90 days vs. total Critical-tier integrations. Source: ST records
- **CI corpus run rate**: For Critical/High-tier integrations, verify weekly corpus run in CI. Formula: `integrations_with_weekly_CI_corpus / total_Critical_High × 100`. Source: CI telemetry
- **Corpus growth**: Git log of jailbreak/prompt-injection corpus; count commits adding new entries per calendar month. Source: VCS change-log
- **Finding to corpus conversion**: For Critical/High severity findings, count those with a corresponding corpus entry committed within 30 days. Source: finding → corpus pipeline telemetry

**Answer:**
- ☐ **Fully Mature** (Evidence complete + ≥3 metrics meet targets)
- ☐ **Implemented** (Evidence complete + 2 metrics meet targets)
- ☐ **Partial** (Evidence partially complete + <2 metrics meet targets)
- ☐ **Not Implemented** (No scheduled red-team function for Critical-tier integrations)

**Evidence Location:** _______________
**Validation Date:** _______________
**Notes:** _______________

---

### Question 5: Maintained Regression Corpus

**Q5.1:** Is a versioned jailbreak/prompt-injection regression corpus maintained with ≥1 update per month and running in CI at least weekly against ≥90% of Critical/High integrations, with pass/fail trend visible to the program sponsor?

**Evidence Required:**
- [ ] Versioned jailbreak and prompt-injection corpus in source control; corpus entries include input, expected safe output, threat tag (HAI TTP + ATLAS tactic ID), source, date added
- [ ] Monthly refresh cadence evidenced: change-log showing ≥1 new entry per month from internal observations (IR-Vendors findings, IM-Vendors incidents, red-team results) and external sources (OWASP LLM Top 10, HackAPrompt dataset, public jailbreak research, ATLAS technique examples)
- [ ] CI wiring confirmed: corpus runs at least weekly against Critical/High-tier integrations; CI job configuration on file
- [ ] Pass/fail trend dashboard or report visible to program sponsor; last report on file
- [ ] Corpus changes go through PR review with a named corpus owner
- [ ] Regression corpus growth driven by red-team findings: Critical/High severity red-team findings converted to corpus entries within 30 days

**Outcome Metrics:**
| Metric | Baseline | Current | Target | Met? | Notes |
|--------|----------|---------|--------|------|-------|
| Regression corpus size / change rate | ___ entries | ___ entries | growing; ≥1 update/month | ☐ | |
| % High/Critical integrations running regression weekly in CI | ___% | ___% | ≥90% | ☐ | |
| Bug-bounty findings consumed into library per quarter | ___ | ___ | ≥4 | ☐ | |
| % red-team findings (Critical/High severity) converted to corpus entries within 30 days | ___% | ___% | ≥90% | ☐ | |

**Metric Collection Guidance:**
- **Corpus growth**: Git log of corpus file; count commits adding new entries per calendar month. Source: VCS change-log
- **CI corpus run rate**: CI job run history for weekly corpus runs; filter to Critical/High-tier integrations. Source: CI telemetry
- **Bug-bounty consumption**: Count new corpus entries in the last quarter traceable to vendor-side or internal bug-bounty findings. Source: corpus change-log × bug-bounty finding log
- **Finding to corpus conversion**: IM-Vendors Critical/High findings × corpus commit dates. Source: IM-Vendors × corpus change-log

**Answer:**
- ☐ **Fully Mature** (Evidence complete + ≥3 metrics meet targets)
- ☐ **Implemented** (Evidence complete + 2 metrics meet targets)
- ☐ **Partial** (Evidence partially complete + <2 metrics meet targets)
- ☐ **Not Implemented** (No maintained regression corpus with weekly CI runs)

**Evidence Location:** _______________
**Validation Date:** _______________
**Notes:** _______________

---

### Question 6: Bug-Bounty Integration

**Q6.1:** Are ≥4 bug-bounty findings per quarter consumed into the test library from both vendor-side and internal programs, and are those findings traceable to TA-Vendors library updates?

**Evidence Required:**
- [ ] Participation in vendor-side bug bounties where AI-vendor programs exist; enrollment confirmation on file
- [ ] Internal bug-bounty program in place for custom AI integrations; program charter and scope on file
- [ ] Bug-bounty finding review process: incoming reports reviewed on a defined cadence; no findings queue aging >30 days without triage
- [ ] ≥4 bug-bounty-sourced findings consumed into the test library per quarter; traceable entries in the corpus or test battery change-log with source tag "bug-bounty"
- [ ] Findings traceable to TA-Vendors library updates: at least one TA-Vendors library-gap ticket or update per quarter from bug-bounty-sourced findings
- [ ] Red-team findings to corpus pipeline: Critical/High red-team findings converted to corpus entries within 30 days

**Outcome Metrics:**
| Metric | Baseline | Current | Target | Met? | Notes |
|--------|----------|---------|--------|------|-------|
| Bug-bounty findings consumed into library per quarter | ___ | ___ | ≥4 | ☐ | |
| % Critical integrations red-teamed in last 90 days | ___% | ___% | 100% | ☐ | |
| % High/Critical integrations running regression weekly in CI | ___% | ___% | ≥90% | ☐ | |
| Regression corpus growth rate (new entries per month) | ___ | ___ | ≥1 per month | ☐ | |

**Metric Collection Guidance:**
- **Bug-bounty consumption**: Corpus/battery change-log entries tagged "bug-bounty" in the last quarter. Source: VCS change-log with source tags
- **TA-Vendors updates from bug-bounty**: TA-Vendors library-gap tickets traceable to bug-bounty findings. Source: issue tracker
- **Regression corpus growth**: Git log per corpus file; monthly commit count. Source: VCS change-log
- **Red-team rate**: Red-team report dates vs. Critical-tier integration list. Source: ST records

**Answer:**
- ☐ **Fully Mature** (Evidence complete + ≥3 metrics meet targets)
- ☐ **Implemented** (Evidence complete + 2 metrics meet targets)
- ☐ **Partial** (Evidence partially complete + <2 metrics meet targets)
- ☐ **Not Implemented** (No bug-bounty integration in place)

**Evidence Location:** _______________
**Validation Date:** _______________
**Notes:** _______________

---

## Maturity Level 3
**Objective:** Continuous automated red-teaming for Critical-tier; publish anonymized findings to industry; host industry-shared red-team exercises

---

### Question 7: Continuous Automated Red-Team

**Q7.1:** Are ≥80% of Critical-tier AI vendor integrations under continuous automated red-team with daily probe execution, using prompt-injection generators, jailbreak ladders, and indirect-injection seeded content, with new TTPs from probe findings triaged into the TA-Vendors library at least weekly and high-severity automated findings routed to IM within 24 hours?

**Evidence Required:**
- [ ] Automated red-team harness deployed and producing daily probe results against Critical-tier AI vendor integrations: prompt-injection generators (mutation of regression corpus + template variation), jailbreak ladders (generated role-override, persona-switch, authority-claim, encoding-bypass sequences), and indirect-injection seeded content (crafted retrieval-path or tool-response payloads for agent-archetype integrations)
- [ ] Continuous red-team telemetry: harness health dashboard showing % Critical-tier integrations with a fresh automated probe result within the last 24 hours; on-call paged when feed goes stale >24 hours
- [ ] Finding triage process: named ST owner reviewing automated findings at least weekly; novel TTP patterns forwarded to TA-Vendors library within 14 days
- [ ] High-severity automated findings route to IM-Vendors within 24 hours; evidence: IM ticket timestamps vs. harness alert timestamps
- [ ] ATLAS tactic IDs covered by harness: TA0001, TA0003, TA0004, TA0012, TA0013
- [ ] Harness scope confirmed against current Critical-tier integration list from SM-Vendors inventory signals

**Outcome Metrics:**
| Metric | Baseline | Current | Target | Met? | Notes |
|--------|----------|---------|--------|------|-------|
| % Critical integrations under continuous automated red-team (daily probe execution) | ___% | ___% | ≥80% | ☐ | |
| New-TTP ingestion lead time (automated finding to TA-Vendors library entry) | ___ days | ___ days | ≤14 days | ☐ | |
| % high-severity automated findings routed to IM within 24 hours | ___% | ___% | 100% | ☐ | |
| Continuous harness health (% Critical integrations with fresh probe result in last 24 hours) | ___% | ___% | ≥95% | ☐ | |

**Metric Collection Guidance:**
- **Continuous coverage**: Harness telemetry, count Critical-tier integrations with a probe result in the last 24 hours vs. total Critical-tier integrations. Source: ST telemetry
- **TTP ingestion lead time**: For novel patterns identified by the harness, measure time from harness alert to TA-Vendors library entry. Source: harness to TA pipeline telemetry
- **IM routing within 24h**: Compare harness alert timestamp to IM ticket creation timestamp for high-severity findings. Source: harness alert log × IM-Vendors system
- **Harness health**: Monitoring dashboard last-probe-time per Critical-tier integration; stale feeds trigger on-call page

**Answer:**
- ☐ **Fully Mature** (Evidence complete + ≥3 metrics meet targets)
- ☐ **Implemented** (Evidence complete + 2 metrics meet targets)
- ☐ **Partial** (Evidence partially complete + <2 metrics meet targets)
- ☐ **Not Implemented** (No continuous automated red-team harness in place)

**Evidence Location:** _______________
**Validation Date:** _______________
**Notes:** _______________

---

### Question 8: Industry Contributions

**Q8.1:** Has the program contributed ≥4 anonymized, legally-vetted findings per year to MITRE ATLAS, AI Vulnerability Database, or OWASP LLM/Agentic Top 10, with at least one accepted as a new or refined technique?

**Evidence Required:**
- [ ] Contribution log on file: ≥4 submissions per year to MITRE ATLAS (novel AI-vendor-specific prompt-injection variants, tool-scope abuse mechanics, data-egress canary bypass techniques, with ATLAS tactic IDs), AVID (structured disclosure submissions for novel vulnerabilities in AI vendor integrations), or OWASP LLM/Agentic Top 10 (real-world telemetry evidence during revision cycles)
- [ ] At least one submission accepted as a new or refined technique; evidence: ATLAS, AVID, or OWASP acknowledgment or technique ID assignment
- [ ] Legal-vetting record for each contribution: org identity scrubbed; coordinated disclosure completed for vendor-side component involvement
- [ ] Industry-contribution pipeline: at least one anonymized finding in-preparation, in-legal-review, or submitted at all times
- [ ] AVID submissions for novel vulnerabilities in AI vendor integrations include: coordinated disclosure completion record, vendor notification date, and public disclosure date

**Outcome Metrics:**
| Metric | Baseline | Current | Target | Met? | Notes |
|--------|----------|---------|--------|------|-------|
| Industry contributions per year (MITRE ATLAS / AVID / OWASP LLM / Agentic Top 10) | ___ | ___ | ≥4 | ☐ | |
| Contributions accepted as new or refined techniques | ___ | ___ | ≥1 per year | ☐ | |
| % Critical integrations under continuous automated red-team | ___% | ___% | ≥80% | ☐ | |
| Industry-contribution pipeline active (finding in preparation/review/submitted) | No / Yes | No / Yes | Yes | ☐ | |

**Metric Collection Guidance:**
- **Contribution count**: Contribution log filtered to the last 12 months; count submissions with submission confirmation. Source: contribution log
- **Acceptance**: Cross-reference contribution log with ATLAS/AVID/OWASP acknowledgment records. Source: contribution log × external acknowledgments
- **Pipeline activity**: Check whether at least one finding is in-preparation, in-legal-review, or submitted at the time of assessment. Source: contribution pipeline status
- **Continuous coverage**: Harness telemetry as described in Q7 metrics

**Answer:**
- ☐ **Fully Mature** (Evidence complete + ≥3 metrics meet targets)
- ☐ **Implemented** (Evidence complete + 2 metrics meet targets)
- ☐ **Partial** (Evidence partially complete + <2 metrics meet targets)
- ☐ **Not Implemented** (No industry contributions in place)

**Evidence Location:** _______________
**Validation Date:** _______________
**Notes:** _______________

---

### Question 9: Industry-Shared Exercises

**Q9.1:** Has the program hosted at least 1 industry-shared red-team exercise per year and participated in ≥2 additional cross-org exercises, with documented cross-org detection-benchmark improvement data from participants?

**Evidence Required:**
- [ ] Exercise log on file: ≥1 hosted exercise per year (ISAC AI-vendor tabletop, OWASP AI chapter, or ATLAS practitioner table); ≥2 additional cross-org exercises participated
- [ ] Hosted exercise documented: agenda, participant list, benchmark methodology, measurement of detection-benchmark improvement before and after the exercise
- [ ] Cross-org detection-benchmark improvement data collected from participants and documented in exercise report
- [ ] Industry-exercise calendar: next hosted or co-hosted exercise scheduled at least 60 days in advance
- [ ] Participation in sector ISAC AI-vendor tabletops or cross-org red-team benchmarks with anonymized findings on record
- [ ] Published anonymized regression corpus (jailbreak/prompt-injection) cited or used by at least one external organization; adoption documented

**Outcome Metrics:**
| Metric | Baseline | Current | Target | Met? | Notes |
|--------|----------|---------|--------|------|-------|
| Industry-shared exercises per year | ___ | ___ | ≥1 hosted + ≥2 participated | ☐ | |
| Cross-org detection-benchmark improvement documented | No / Yes | No / Yes | Yes | ☐ | |
| Industry contributions per year | ___ | ___ | ≥4 | ☐ | |
| Next hosted exercise scheduled ≥60 days in advance | No / Yes | No / Yes | Yes | ☐ | |

**Metric Collection Guidance:**
- **Exercise count**: Exercise log filtered to the last 12 months; count hosted and participated exercises. Source: exercise log
- **Improvement documentation**: Hosted exercise report; participant pre/post detection benchmark data. Source: exercise reports
- **Contribution count**: Contribution log as described in Q8 metrics. Source: contribution log
- **Calendar check**: Confirm a specific future exercise date ≥60 days out is on the program calendar. Source: program calendar

**Answer:**
- ☐ **Fully Mature** (Evidence complete + ≥3 metrics meet targets)
- ☐ **Implemented** (Evidence complete + 2 metrics meet targets)
- ☐ **Partial** (Evidence partially complete + <2 metrics meet targets)
- ☐ **Not Implemented** (No industry-shared exercises conducted or planned)

**Evidence Location:** _______________
**Validation Date:** _______________
**Notes:** _______________

---

## Summary Scorecard

| Level | Q# | Question | Score | Weight |
|-------|-----|----------|-------|--------|
| L1 | Q1 | Per-Archetype Foundational Test Battery | ___ | |
| L1 | Q2 | Quarterly Battery Re-Run and Triggers | ___ | |
| L1 | Q3 | Shadow-AI Discovery Tests | ___ | |
| **L1 Score** | | | **___** | 0.5 |
| L2 | Q4 | Quarterly Red-Team for Critical-Tier Integrations | ___ | |
| L2 | Q5 | Maintained Regression Corpus | ___ | |
| L2 | Q6 | Bug-Bounty Integration | ___ | |
| **L2 Score** | | | **___** | 0.3 |
| L3 | Q7 | Continuous Automated Red-Team | ___ | |
| L3 | Q8 | Industry Contributions | ___ | |
| L3 | Q9 | Industry-Shared Exercises | ___ | |
| **L3 Score** | | | **___** | 0.2 |
| **Overall ST-Vendors Score** | | | **___** | |

**Maturity Level Achieved:** ☐ L1 ☐ L2 ☐ L3

**Assessment Date:** _______________
**Assessor:** _______________
**Next Review Date:** _______________

---

**Document Version:** HAIAMM v3.0
**Practice:** Security Testing (ST)
**Domain:** Vendors
**Questionnaire Authored:** 2026-05-15
**Author:** Verifhai