Security Testing (ST)

Vendors Domain - HAIAMM v3.0

Practice Overview

Objective: Exercise the AI vendor integration end-to-end with foundational tests that directly target the top archetype threats (data egress, prompt injection, permission-boundary abuse, logging completeness, shadow-AI discovery), so reviewed configurations are not only correct on paper but observed to behave correctly.

Description: ST-Vendors operates at two levels at L1: per-integration acceptance tests that every approved AI vendor integration must pass before production, and program-level shadow AI discovery tests that exercise the organization's ability to detect unsanctioned AI use. The test scope is deliberately narrow at L1, five or six recurring test classes, automated where possible, and tied directly to threats in the TA library.

Context: Most AI vendor integrations are tested only against the happy path. Prompt injection, tool-abuse, data-exfil via helpful behavior, and "feature silently enabled" scenarios are rarely exercised. L1 ST-Vendors installs the minimum repeatable test battery that turns these from theoretical threats into observed pass/fail outcomes.

Maturity Level 1

Objective: Run a foundational AI-vendor test battery at go-live and quarterly; operate program-level shadow-AI discovery tests at least quarterly; feed findings into IM

At this level, testing is a small, repeatable, automated-where-possible battery, not a bespoke exercise per vendor.

Dependencies

• TA-Vendors L1 (required): tests target specific threats from the archetype library.
• SR-Vendors L1 (required): tests verify requirements are enforced, not aspirational.
• SA-Vendors L1 (required): tests exercise the reference-pattern control points (proxy, DLP, logging).
• IR-Vendors L1 (required): tests assume the integration is configured as designed, IR precedes ST.
• Supports / unblocks: IM-Vendors L1 (test fails become issues), ML-Vendors L1 (detections are validated by the shadow-AI discovery tests), EH-Vendors L1 (hardening informed by test findings).

Desired Outcomes

• Every production AI vendor integration has passed a documented test battery against its archetype's top threats.
• Shadow AI detection is exercised, not assumed to work because the pipeline exists.
• Test failures become issues with named owners, not PDFs in a drive.
• The battery is automated enough to re-run on demand (post-change, post-incident, on cadence) without a bespoke effort each time.

Activities

A) Publish the foundational AI-vendor test battery per archetype

Ship a short, repeatable battery, L1 target ≤6 tests per archetype, each tied to a TA threat and an SR requirement. Tests are automated where feasible; manual where required but documented.

Common test classes: - Data egress test, send a synthetic "canary" payload (tagged, not real regulated data) and verify logging, vendor-side retention behavior, and DLP interception where applicable. - No-train verification, confirm the no-train setting is honored (admin-console state, DPA reference, and where testable, a behavioral probe). - Prompt-injection resilience probe, submit a curated set of prompt-injection test strings; verify that system prompts/tool permissions hold and that logging captures attempts. - Permission-boundary / tool-scope test (agent archetype), attempt actions outside the allowlist and outside the scoped parameters; verify deny + log. - Logging completeness test, verify every required event type produces a log line that reaches the org-side store with correct attribution and retention. - Kill-switch / rate-limit test, exercise the kill-switch or rate-limit path; verify behavior. - AI-embedded SaaS toggle-drift test, re-check the toggle state and users-with-access list vs. approved scope.

Each test: inputs, expected output, pass/fail criteria, evidence artifact (log snippet / screenshot / trace ID).

B) Run the battery at go-live and at least quarterly

Run triggers: - Go-live, all applicable tests pass before production cutover. - Quarterly, all active AI vendor integrations re-run the battery; failures routed to IM. - Post-change, vendor major version bump, model-family change, admin-console redesign, org-plan migration. - Post-incident, any IM-Vendors incident re-runs the relevant subset before the incident is closed.

C) Operate shadow-AI discovery tests at program level

ST-Vendors also tests the program's ability to detect unsanctioned AI use, validating the detections ML-Vendors depends on.

Quarterly, run synthetic scenarios (with sponsor approval and scope limits): - A test account attempts to pay for an unsanctioned consumer GenAI subscription via expense pathways, is it flagged? - A test endpoint downloads a known-AI-tool installer, is it detected by endpoint inventory? - A test SaaS admin toggles on an AI feature in a sandbox workspace, is the SaaS-admin-audit detection signal raised? - A test egress to a known AI vendor domain from an unmanaged path, is it observed?

Each scenario: pass/fail + time-to-detect. Failures feed ML-Vendors' detection backlog and, where structural, SA-Vendors' pattern updates.

Outcome Metrics (L1)

Process Metrics (leading)

• Battery owner named per archetype; quarterly runs scheduled in advance.
• Test-automation coverage, target ≥60% of battery items automated at L1.
• Shadow-AI exercise cadence honored (at least quarterly).

Effectiveness Metrics (business value)

• Observed threats mitigated, test failures that uncovered a misconfiguration before exploitation.
• Detection confidence, shadow-AI discovery pass rate trend informs the program sponsor's risk narrative.
• Reduced incident volume, vendors passing the battery have a materially lower IM incident rate (tracked over time).

Success Criteria

• Foundational battery published per archetype and linked from DR/IR artifacts.
• 100% of AI vendor integrations reaching production in the last 90 days have a passed go-live battery.
• Shadow-AI discovery exercise run at least once in the last 90 days with results reviewed by the program sponsor.
• Named battery owner per archetype.
• Test failures routed to IM with 1-day handoff SLA.

Maturity Level 2

Objective: Dedicated AI-vendor red team for Critical tier quarterly; maintained regression corpus for jailbreaks and prompt-injection; bug-bounty integration where applicable

At this level, testing adds a dedicated red-team function for Critical-tier AI-vendor integrations. Jailbreak and prompt-injection regression corpora are versioned, shared, and runnable on demand. Bug bounties, vendor-side and internal, feed findings into the program.

Dependencies

• ST-Vendors L1 (required): foundational per-archetype test battery.
• TA-Vendors L2 (required): per-vendor threat models shape red-team scope.
• IR-Vendors L2 (required): config validated before ST probes run.

Desired Outcomes

• Every Critical-tier AI vendor integration is red-teamed by the program at least quarterly.
• Jailbreak and prompt-injection regressions run weekly in CI; pass-fail trend visible.
• Bug-bounty learnings, vendor-side and internal, flow into the program's test library.

Activities

A) Quarterly red-team for Critical-tier integrations

• Scope: prompt-injection chains, indirect-prompt-injection via RAG, agent tool abuse, jailbreak regression, data-egress canaries.
• Deliverable: red-team report with findings, reproducibility notes, remediation recommendations; routes to IM.

B) Maintained regression corpus

• Versioned jailbreak / prompt-injection corpus (internal + external sources); expands monthly.
• Runs in CI against Critical/High integrations weekly.
• Pass/fail trend reported.

C) Bug-bounty integration

• Participate in vendor-side bug bounties where AI-vendor programs exist.
• Run internal bug-bounty for custom AI integrations.
• Findings feed the program's test library and TA updates.

Outcome Metrics (L2)

Process Metrics (leading)

• Red-team schedule on calendar; no Critical skipped.
• Corpus review cadence, monthly.
• Bug-bounty participation active.

Effectiveness Metrics (business value)

• Critical-tier incident rate decreases as red-team catches pre-prod.
• Regression corpus catches model-update regressions early.

Success Criteria

• Quarterly Critical-tier red-team for 100% of integrations.
• Regression corpus running weekly in CI for ≥90% of Critical/High.
• ≥4 bug-bounty-sourced findings consumed per quarter.

Maturity Level 3

Objective: Continuous automated red-teaming for Critical-tier; publish anonymized findings to industry; host industry-shared red-team exercises

At this level, testing runs continuously rather than periodically. An automated red-team harness probes Critical-tier integrations daily. Anonymized findings are contributed to industry (MITRE ATLAS, AIVD). The program hosts or co-hosts industry-shared exercises (benchmarking, cross-org red-team).

Dependencies

• ST-Vendors L2 (required): red-team function, regression corpus, bug-bounty integration.
• TA-Vendors L3 (required): telemetry-driven threat library.
• ML-Vendors L2+ (required): detections instrumented to catch red-team activity.

Desired Outcomes

• Critical-tier adversarial posture measured continuously.
• Program-originated TTPs recognized in MITRE ATLAS / AIVD.
• Industry-shared exercises improve cross-org detection maturity.

Activities

A) Continuous automated red-team

• Automated probes (prompt-injection generators, jailbreak ladders, indirect-injection seeded content) run daily against Critical-tier.
• Findings triaged; new TTPs feed the TA library.

B) Contribute findings to industry

• Anonymized, legally-vetted contributions to MITRE ATLAS, AI Vulnerability Database, OWASP LLM / Agentic Top 10.
• Target ≥4 contributions/year.

C) Industry-shared exercises

• Host or co-host cross-org red-team benchmarks.
• Participate in ISAC AI-vendor tabletops.

Outcome Metrics (L3)

Process Metrics (leading)

• Continuous red-team harness health, % Critical integrations producing a fresh automated probe result within the last 24 hours.
• New-TTP ingestion cadence, findings from automated probes reviewed and triaged into the TA library at least weekly.
• Industry-contribution pipeline, at least one anonymized finding in-preparation, in-legal-review, or submitted at any time.
• Industry-exercise calendar, next hosted or co-hosted exercise scheduled at least 60 days in advance.

Effectiveness Metrics (business value)

• Mean time to detect novel AI-vendor attack techniques decreases quarter-over-quarter as continuous harness catches model-update regressions within hours, not sprint cycles.
• Program-originated TTPs recognized in MITRE ATLAS or AI Vulnerability Database demonstrate external validation of testing rigor.
• Cross-org exercise participants cite improved detection benchmarks, measurable uplift in peer organizations' defenses from shared corpora.
• Critical-tier incidents attributable to prompt injection, tool-scope abuse, or data-egress drop as continuous harness closes the gap between quarterly red-team cycles.

Success Criteria

• ≥80% Critical under continuous automated red-team.
• ≥4 industry contributions/year.
• ≥1 hosted + ≥2 participated industry exercises per year.

Key Success Indicators

Level 1: - Per-archetype foundational test battery published and linked from DR/IR artifacts, covering data-egress canary, no-train verification, prompt-injection resilience probe, permission-boundary/tool-scope test (agent archetype), logging-completeness test, kill-switch/rate-limit test, and toggle-drift test. - 100% of AI vendor integrations going to production in the last 90 days have a passed go-live battery on record; ≥90% of active integrations carry a current-quarter battery pass. - Shadow-AI discovery exercise run at least quarterly with results reviewed by the program sponsor; ≥80% of shadow-AI scenarios detected within SLA by end of year one. - All test failures routed to IM within 1 business day with named owner; battery automation coverage ≥60% at L1.

Level 2: - 100% of Critical-tier AI vendor integrations red-teamed in the last 90 days, covering prompt-injection chains, indirect-prompt-injection via RAG, agent tool abuse, jailbreak regression, and data-egress canaries. - Versioned jailbreak/prompt-injection regression corpus growing at ≥1 update per month; running weekly in CI against ≥90% of Critical/High integrations with pass/fail trend reported. - ≥4 bug-bounty-sourced findings consumed into the test library per quarter from vendor-side and internal programs.

Level 3: - ≥80% of Critical-tier integrations under continuous automated red-team with daily probe execution; new TTPs triaged into the TA library weekly. - ≥4 anonymized, legally-vetted findings contributed per year to MITRE ATLAS, AI Vulnerability Database, or OWASP LLM/Agentic Top 10, with at least one acknowledged as a new technique or variant. - At least 1 industry-shared exercise hosted per year plus ≥2 participated; cross-org detection-benchmark improvement documented.

Common Pitfalls

Level 1: - ❌ Test battery reduced to a logging-completeness check and a screenshot of the admin console, no behavioral probes (data-egress canary, prompt-injection, kill-switch) ever actually exercised. - ❌ Go-live battery runs but quarterly re-runs are skipped, test coverage erodes as configurations change post-launch. - ❌ Shadow-AI discovery scenarios written but never actually run against live detection infrastructure; the program assumes detections work without validating them. - ❌ Test failures logged in a spreadsheet separate from IM, no SLA enforcement, no aging visibility, no owner accountability. - ❌ "No-train verification" passes because the admin-console toggle is checked but no behavioral probe is attempted; the setting could be ignored by the vendor without detection.

Level 2: - ❌ Red-team scope defined as "prompt-injection probes" but indirect-prompt-injection via RAG retrieval, tool-parameter smuggling, and multi-turn jailbreak chains are excluded. - ❌ Regression corpus seeded at L2 launch and never updated, jailbreak corpus doesn't include techniques from the last 6 months, so model-update regressions pass undetected. - ❌ Bug-bounty participation passive, the program is enrolled but nobody reviews incoming reports; findings queue up without reaching the test library. - ❌ Red-team schedule slips because no Critical integration is ever skipped, scope is so broad that the team can't maintain quarterly cadence without dedicated capacity.

Level 3: - ❌ Continuous automated red-team harness runs prompt-injection probes that the vendor's content filter trivially blocks, coverage metric looks good but the probes aren't probing the real threat surface. - ❌ Industry contributions are legal-vetted case-study summaries rather than actionable, reproducible technique descriptions, ATLAS reviewers cannot map them to a technique ID. - ❌ Hosted industry exercises become vendor-showcase events rather than detection-benchmarking sessions, no measurable improvement data collected from participants. - ❌ New-TTP ingestion from automated probes to the TA library is manual and quarterly, by the time a novel technique reaches SR and SA updates, it is already in the wild.

Practice Maturity Questions

Level 1: 1. Is a per-archetype foundational test battery published with ≤6 test classes per archetype, data-egress canary, no-train verification, prompt-injection probe, permission-boundary/tool-scope test, logging-completeness test, kill-switch test, each with defined inputs, pass/fail criteria, and evidence artifact, and are 100% of new integrations required to pass the battery before production? 2. Is the battery re-run at least quarterly for all active AI vendor integrations, with post-change and post-incident re-run triggers wired to the change-management and IM processes, and are ≥90% of integrations carrying a current-quarter pass? 3. Are shadow-AI discovery exercises run at least quarterly, validating that synthetic unsanctioned-AI scenarios are detected within SLA, with failures feeding ML-Vendors' detection backlog and results reviewed by the program sponsor?

Level 2: 1. Are 100% of Critical-tier AI vendor integrations red-teamed at least quarterly with scope covering prompt-injection chains, indirect-prompt-injection via RAG, agent tool abuse, jailbreak regression, and data-egress canaries, with findings routed to IM and remediation tracked? 2. Is a versioned jailbreak/prompt-injection regression corpus maintained with ≥1 update per month and running in CI at least weekly against ≥90% of Critical/High integrations, with pass/fail trend visible to the program sponsor? 3. Are ≥4 bug-bounty findings per quarter consumed into the test library from both vendor-side and internal programs, and are those findings traceable to TA library updates?

Level 3: 1. Are ≥80% of Critical-tier AI vendor integrations under continuous automated red-team with daily probe execution, with new TTPs from probe findings triaged into the TA library at least weekly and traceable to SR/SA updates? 2. Has the program contributed ≥4 anonymized, legally-vetted findings per year to MITRE ATLAS, AI Vulnerability Database, or OWASP LLM/Agentic Top 10, with at least one accepted as a new or refined technique? 3. Has the program hosted at least 1 industry-shared red-team exercise per year and participated in ≥2 additional cross-org exercises, with documented detection-benchmark improvement data from participants?

Document Version: HAIAMM v3.0 Practice: Security Testing (ST) Domain: Vendors Last Updated: 2026-05-12 Author: Verifhai