Security Testing (ST) - Processes Domain

HAIAMM Assessment Questionnaire v3.5

Generated from cells/ST-Processes-L*.cell.json. Do not edit by hand. Canonical model: HAIAMM-v3.5-Framing.md · Handbook: HAIAMM-v3.5-Handbook.md.

---

Practice: Security Testing (ST) Domain: Processes Business Function: Verification Purpose: Baseline a foundational security test battery per processes archetype that asserts the controls actually hold, and run it before every release. Scoring Model: Evidence + Outcome Metrics (see Scoring Methodology below)

---

Instructions

• Answer from current, implemented practice, not plans or aspirations.
• Each question has two components: Evidence (what you did) and Outcome Metrics (how well it worked).
• Levels are derived, never self-assigned. Record evidence and metric values; the level falls out of the scoring below.
• Silence is not a pass. A metric you cannot produce is left blank and counts as not met. Do not estimate.
• Answer progressively. Complete all Level 1 questions before Level 2.

Evidence grade

Record how each answer is known, not only what is claimed (v3.5 framing §6). A level claim is graded at its weakest field.

Grade	Meaning
E0	Asserted by a practitioner, unevidenced
E1	Asserted, with a link to a named artifact
E2	Emitted by a system as telemetry
E3	Machine-attested and independently checkable

---

Scoring Methodology

Tier	Score	Criteria
Fully Mature	1.0	Evidence complete + ≥3 outcome metrics meet targets
Implemented	0.67	Evidence complete + 2 outcome metrics meet targets
Partial	0.33	Evidence partially complete + <2 outcome metrics meet targets
Not Implemented	0.0	No substantive evidence of practice

Practice maturity level achieved = the highest level where all 3 questions score ≥ 0.67.

---

Maturity Level 1, Foundational

Objective: Baseline a foundational security test battery per processes archetype that asserts the controls actually hold, and run it before every release.

Prerequisite cells: SM-Processes-L1, TA-Processes-L1, SR-Processes-L1, SA-Processes-L1

Question L1.1: Build the foundational battery per archetype

Q1.1: Does a foundational security test battery exist for at least 85% of inventoried processes archetypes, with each test written as an assertion about a control and verified as able to fail by deliberately introducing its condition once?

Evidence Required:

• [ ] Author a foundational test battery per archetype covering a seeded-error test measuring whether reviewers catch known-bad AI recommendations, an authority test asserting the workflow actually permits refusal, a time-budget test asserting the workflow allows the declared time per decision, a disclosure test asserting affected people are told, a contestability test walking the appeal route end to end, and a fallback exercise asserting the manual path works at realistic volume
• [ ] Write each test as an assertion about a control rather than as a check on behaviour, and verify that each test can fail: introduce the condition deliberately once and confirm the test catches it, which is the only way to distinguish a test that asserts something from one that passes because its condition never arises
• [ ] Keep the battery small enough to run on every workflow configuration change at the tiers that need it, because a battery too slow to run continuously will be moved to a nightly job and then to a weekly one and then to nowhere
• [ ] Version the battery alongside the reference patterns so that a pattern change and its test change travel together

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L1.2: Map tests to threats and publish the coverage gaps

Q1.2: Is every test mapped to the TA threat identifiers it exercises, with threats having no test published as gaps and threat coverage reported rather than test count? For Processes, the assessor should be shown the seeded-error exercise and the measured reviewer catch rate.

Evidence Required:

• [ ] Map each test to the threat identifiers from the TA library that it exercises, and publish the threats that have no test
• [ ] The coverage map is what converts a test suite into an assurance argument: without it the battery asserts that the tests pass, which is not the same claim as that the threats are addressed
• [ ] Report threat coverage rather than test count, since test count rises with effort while coverage rises only with effect
• [ ] Treat an untested threat at a high-risk archetype as an open finding with an owner rather than as a backlog item, and feed coverage gaps back to the threat assessment practice so that the library and the battery evolve together

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L1.3: Run before release, with a governed waiver path

Q1.3: Does the battery run before every release and on the change types that invalidate prior results, with failures blocking Critical- and High-tier releases and a governed waiver path carrying named authority, reason and expiry? In Processes, ask to see the seeded-error exercise and the measured reviewer catch rate.

Evidence Required:

• [ ] Run the battery before every release, and trigger it on the changes that actually invalidate prior results, being a volume increase, a change in the AI's role in the decision, an oversight reduction, and a workflow user-interface change
• [ ] Failures block release for Critical- and High-tier assets and warn for the rest
• [ ] Provide a governed waiver path, because an unwaivable gate is bypassed in ways nobody sees: a waiver needs a named authority, a stated reason, an expiry and automatic notification
• [ ] Review waivers weekly and treat a test waived across more than one release cycle as a finding, since the common failure here is not a single waiver but a test quietly disabled to unblock a release and never re-enabled, which reduces coverage without anyone deciding to reduce it

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Level 1 Outcome Metrics

Metric	Baseline	Current	Target	Met?	Notes
Processes archetypes with a baselined foundational security test battery	___	___	≥85% of inventoried archetypes within 9 months	☐
Tests verified as able to fail, by deliberately introducing the condition once	___	___	100% of tests in the battery	☐
Threats from the TA library exercised by at least one test	___	___	≥75%, with untested threats published as gaps	☐
Releases of Critical- and High-tier assets proceeding with a failing battery test and no waiver	___	___	0	☐
Test waivers carrying a named authority, reason and expiry	___	___	100%, with none past expiry	☐

Where each number is read from:

• Processes archetypes with a baselined foundational security test battery — Test registry joined to the asset inventory
• Tests verified as able to fail, by deliberately introducing the condition once — Test verification records
• Threats from the TA library exercised by at least one test — Threat-to-test coverage map
• Releases of Critical- and High-tier assets proceeding with a failing battery test and no waiver — CI results joined to release records
• Test waivers carrying a named authority, reason and expiry — Waiver log

Level 1 Oversight Metrics

"A human approves it" is not a control unless you measure whether they refuse. A metric that cannot fail is not a metric.

Metric	Current	Oversight failure it detects	Met?
Failing tests waived to permit a release, with a named waiver authority	___	waiver becoming the routine release path, where a failing security test is treated as a scheduling obstacle rather than a result	☐
Tests marked as skipped or quarantined for more than one release cycle	___	silent test decay, where a test is disabled to unblock a release and never re-enabled, so coverage falls without any decision being taken	☐

Level 1 30-second check (E0)

• ☐ Yes ☐ No — Is there a security test that would fail if your AI controls stopped working?
• ☐ Yes ☐ No — Has anyone checked that your security tests can actually fail?
• ☐ Yes ☐ No — Would a failing security test stop a release?

---

Maturity Level 2, Comprehensive

Objective: Calibrate the processes test battery to asset tier, run it continuously in CI, and add adversarial testing where the automated battery cannot reach.

Prerequisite cells: ST-Processes-L1, SM-Processes-L2, TA-Processes-L2, IR-Processes-L2

Question L2.1: Calibrate battery depth and gating to tier

Q2.1: Is battery depth and gating behaviour published per tier and derived automatically from the inventory rather than selected by teams under release pressure? Evidence this in the Processes domain against the seeded-error exercise and the measured reviewer catch rate.

Evidence Required:

• [ ] Publish what each tier receives: Critical takes the full battery on every workflow configuration change, with failures blocking the change, plus quarterly adversarial testing
• [ ] High takes the full battery in the workflow change process with failures blocking
• [ ] Medium takes a defined subset
• [ ] Derive the applicable battery from the inventory tier automatically rather than letting teams select, because selection under release pressure converges on the fastest subset
• [ ] Publish gating behaviour per tier so that nobody negotiates it at release time
• [ ] Where battery runtime makes per-build execution impractical at Critical tier, treat that as a test-performance problem to be solved rather than a reason to reduce frequency, since reduced frequency is how continuous testing becomes periodic testing
• [ ] Adversarial work in this domain is conducted with people rather than against systems: seeded-error exercises measuring reviewer catch rates, which is uncomfortable to run and the only way to establish whether oversight is real

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L2.2: Add adversarial testing where automation cannot reach

Q2.2: Is every Critical-tier asset exercised by adversarial testing at least annually with scope set by the testing party, and does every adversarial finding become a new automated test within 30 days so the automated floor rises permanently? For Processes, the assessor should be shown the seeded-error exercise and the measured reviewer catch rate.

Evidence Required:

• [ ] Schedule adversarial testing by people against Critical-tier assets at least quarterly, scoped from the threat model and executed by testers who did not build the asset
• [ ] The purpose is specifically to find what the automated battery cannot: an automated suite tests the techniques it encodes and therefore converges, passing continuously while the attack surface moves underneath it
• [ ] Measure the adversarial exercise by what it found that the battery missed, and feed every such finding back into the battery as a new test, so that the exercise permanently raises the automated floor rather than producing a report
• [ ] Scope must be set by the testing party rather than the tested team, because a team that steers scope away from what it knows is weak produces a clean result that means nothing
• [ ] The regression trigger that matters most here is a volume increase, because catch rates fall with load and nothing else in the process signals it

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L2.3: Keep corpora fresh and remove tests that assert nothing

Q2.3: Are regression corpora refreshed within 90 days and triggered by invalidating change types, and are tests that have never failed across a quarter audited for assertion strength rather than assumed to be stable controls? In Processes, ask to see the seeded-error exercise and the measured reviewer catch rate.

Evidence Required:

• [ ] Refresh regression corpora on a defined cadence, targeting 90 days, drawing new cases from published research, incidents, adversarial findings and threat-intelligence updates
• [ ] A prompt-injection or adversarial-input corpus that has not changed in a year tests the techniques of a year ago and passes with increasing confidence as the real surface moves
• [ ] Separately, audit the battery for tests that have never failed across a full quarter and examine each: some are genuinely stable controls and some assert nothing at all, and the two are indistinguishable from a dashboard
• [ ] Remove or strengthen the latter
• [ ] Trigger regression runs on a volume increase, a change in the AI's role in the decision, an oversight reduction, and a workflow user-interface change in addition to the scheduled cadence
• [ ] Process test content contributes well as seeded-error methodologies and measured catch rates, which is close to absent in the public literature

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Level 2 Outcome Metrics

Metric	Baseline	Current	Target	Met?	Notes
Assets running the battery depth derived from their tier	___	___	100% of Critical and High tier assets	☐
Critical-tier assets exercised by adversarial testing in the last 12 months	___	___	100%, scoped by the testing party	☐
Adversarial findings fed back into the automated battery as new tests	___	___	100% within 30 days of the finding	☐
Regression corpus entries with a last-refresh date within 90 days	___	___	100%	☐
Tests with a 100% pass rate across a full quarter, audited for assertion strength	___	___	100% audited, with non-asserting tests removed or strengthened	☐
Waived Critical-tier tests carrying an authority proportionate to tier and an expiry	___	___	100%, with none past expiry	☐

Where each number is read from:

• Assets running the battery depth derived from their tier — Battery assignment joined to inventory tier
• Critical-tier assets exercised by adversarial testing in the last 12 months — Adversarial engagement register
• Adversarial findings fed back into the automated battery as new tests — Test registry change log joined to engagement findings
• Regression corpus entries with a last-refresh date within 90 days — Corpus metadata
• Tests with a 100% pass rate across a full quarter, audited for assertion strength — Test audit records
• Waived Critical-tier tests carrying an authority proportionate to tier and an expiry — Waiver log joined to asset tier

Level 2 Oversight Metrics

"A human approves it" is not a control unless you measure whether they refuse. A metric that cannot fail is not a metric.

Metric	Current	Oversight failure it detects	Met?
Critical-tier test waivers, with the waiver authority and expiry recorded	___	waiver authority drifting downward under release pressure, so the decision to ship with a failing security test is taken by whoever is available	☐
Waived tests later implicated in a production incident	___	waiver judgment that is procedurally correct and substantively wrong, which only forward tracing can reveal	☐
Test suites whose pass rate is 100% across every run for a full quarter	___	tests that assert nothing, which are indistinguishable from passing tests on every dashboard and provide false assurance at exactly the tiers that need real assurance	☐

Level 2 30-second check (E0)

• ☐ Yes ☐ No — Has a person tried to break your highest-risk AI asset this year?
• ☐ Yes ☐ No — When did your prompt-injection test cases last change?
• ☐ Yes ☐ No — Do you know which of your security tests have never once failed?

---

Maturity Level 3, Industry-Leading

Objective: Run continuous adversarial testing against processes AI assets, measure the battery against what people can still find, and contribute test corpora back to the field.

Prerequisite cells: ST-Processes-L2, TA-Processes-L3, ML-Processes-L2

Question L3.1: Run adversarial testing continuously

Q3.1: Is adversarial testing continuous against Critical-tier assets rather than scheduled, measured by time-to-find and surprise rate rather than by engagement count? Evidence this in the Processes domain against the seeded-error exercise and the measured reviewer catch rate.

Evidence Required:

• [ ] Move from scheduled engagements to continuous adversarial operation against Critical-tier assets, whether by a standing internal team, a contracted continuous service or an automated adversarial harness that evolves its own cases
• [ ] Measure the practice by time-to-find rather than by engagement count, and track the surprise rate, being the share of findings the automated battery should have caught but did not
• [ ] A falling surprise rate indicates the battery is converging on reality
• [ ] A flat one at high volume indicates the battery has converged on itself
• [ ] Feed every finding into the battery within days rather than within the 30-day window that L2 allowed, since continuous operation makes the slower loop the binding constraint
• [ ] Adversarial work in this domain is conducted with people rather than against systems: seeded-error exercises measuring reviewer catch rates, which is uncomfortable to run and the only way to establish whether oversight is real

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L3.2: Close the automation gap deliberately

Q3.2: Is the automation gap treated as the primary work item, with each human finding categorised as a missing case, missing technique class, weak assertion or unreachable surface so that investment goes to the right remedy? For Processes, the assessor should be shown the seeded-error exercise and the measured reviewer catch rate.

Evidence Required:

• [ ] Treat the gap between what automation catches and what humans still find as the practice's primary work item rather than as an inevitable residue
• [ ] For each human finding, ask specifically why the battery missed it: a missing case, a missing technique class, an assertion too weak, or a surface the harness cannot reach
• [ ] Record the category, because the distribution tells you whether to extend the corpus, strengthen assertions or rebuild the harness, and these are very different investments
• [ ] Publish the gap measure alongside coverage so that a rising coverage number with a static gap is visible as the illusion it represents
• [ ] The regression trigger that matters most here is a volume increase, because catch rates fall with load and nothing else in the process signals it

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L3.3: Contribute corpora and harnesses back

Q3.3: Do new public techniques reach the corpus within 14 days at the 90th percentile, and does the organisation contribute at least four accepted artifacts per year in majority-runnable form with disclosure coordinated? In Processes, ask to see the seeded-error exercise and the measured reviewer catch rate.

Evidence Required:

• [ ] Contribute at least four substantive artifacts per year, weighted toward runnable form: adversarial corpora, test harnesses, assertion libraries and reproducible technique demonstrations
• [ ] Shared adversarial corpora are the scarcest resource in AI security and the one where contribution most directly raises the whole field's floor, because a corpus a peer can run is worth far more than a paper describing what it contains
• [ ] Work through OWASP projects, MITRE ATLAS, OpenSSF AI and sector bodies
• [ ] Anonymise, legally review and coordinate disclosure with affected providers before release
• [ ] Count only accepted artifacts, and prioritise contributing the cases that most often catch real defects internally, since those carry the most evidence
• [ ] Process test content contributes well as seeded-error methodologies and measured catch rates, which is close to absent in the public literature

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Level 3 Outcome Metrics

Metric	Baseline	Current	Target	Met?	Notes
Critical-tier assets under continuous adversarial testing rather than scheduled campaigns	___	___	100%	☐
Surprise rate: findings the automated battery should have caught but did not	___	___	tracked and falling year over year	☐
Human findings fed back into the automated battery	___	___	100% within 5 business days	☐
Time from a technique appearing publicly to a corpus entry exercising it	___	___	≤14 days at the 90th percentile	☐
Accepted external contributions per year in runnable form	___	___	≥4 accepted, majority runnable corpora or harnesses	☐

Where each number is read from:

• Critical-tier assets under continuous adversarial testing rather than scheduled campaigns — Adversarial operation register
• Surprise rate: findings the automated battery should have caught but did not — Adversarial findings classified against battery coverage
• Human findings fed back into the automated battery — Test registry change log joined to findings
• Time from a technique appearing publicly to a corpus entry exercising it — Corpus change log joined to publication dates
• Accepted external contributions per year in runnable form — Contribution log with acceptance references

Level 3 Oversight Metrics

"A human approves it" is not a control unless you measure whether they refuse. A metric that cannot fail is not a metric.

Metric	Current	Oversight failure it detects	Met?
Adversarial findings that the automated battery should have caught but did not	___	a battery that has converged on what it already tests, which passes continuously while the real attack surface moves	☐
Automated test results overridden at merge, with the override sustained across builds	___	override becoming the normal path around an inconvenient check, which converts a gate into advice without any decision	☐
Share of red-team scope determined by the team being tested	___	scope capture, where the tested team steers the exercise away from what it knows is weak, which makes a clean result meaningless	☐

Level 3 30-second check (E0)

• ☐ Yes ☐ No — Is someone continuously trying to break your highest-risk AI?
• ☐ Yes ☐ No — Do you know what your automated tests keep missing?
• ☐ Yes ☐ No — Have you published an adversarial corpus others can run?

---

Summary Scorecard

Level	Q1	Q2	Q3	Level score	Weakest evidence grade	Achieved?
L1	___	___	___	___	___	☐
L2	___	___	___	___	___	☐
L3	___	___	___	___	___	☐

Security Testing (ST) maturity in the Processes domain: ____________

---

Document Version: HAIAMM v3.5 (generated)