Implementation Review (IR) - Endpoints Domain

HAIAMM Assessment Questionnaire v3.5

Generated from cells/IR-Endpoints-L*.cell.json. Do not edit by hand. Canonical model: HAIAMM-v3.5-Framing.md · Handbook: HAIAMM-v3.5-Handbook.md.

---

Practice: Implementation Review (IR) Domain: Endpoints Business Function: Verification Purpose: Review what was actually built against what was designed and required for endpoints AI assets, at go-live, with findings tracked to closure. Scoring Model: Evidence + Outcome Metrics (see Scoring Methodology below)

---

Instructions

• Answer from current, implemented practice, not plans or aspirations.
• Each question has two components: Evidence (what you did) and Outcome Metrics (how well it worked).
• Levels are derived, never self-assigned. Record evidence and metric values; the level falls out of the scoring below.
• Silence is not a pass. A metric you cannot produce is left blank and counts as not met. Do not estimate.
• Answer progressively. Complete all Level 1 questions before Level 2.

Evidence grade

Record how each answer is known, not only what is claimed (v3.5 framing §6). A level claim is graded at its weakest field.

Grade	Meaning
E0	Asserted by a practitioner, unevidenced
E1	Asserted, with a link to a named artifact
E2	Emitted by a system as telemetry
E3	Machine-attested and independently checkable

---

Scoring Methodology

Tier	Score	Criteria
Fully Mature	1.0	Evidence complete + ≥3 outcome metrics meet targets
Implemented	0.67	Evidence complete + 2 outcome metrics meet targets
Partial	0.33	Evidence partially complete + <2 outcome metrics meet targets
Not Implemented	0.0	No substantive evidence of practice

Practice maturity level achieved = the highest level where all 3 questions score ≥ 0.67.

---

Maturity Level 1, Foundational

Objective: Review what was actually built against what was designed and required for endpoints AI assets, at go-live, with findings tracked to closure.

Prerequisite cells: SM-Endpoints-L1, SR-Endpoints-L1, SA-Endpoints-L1, DR-Endpoints-L1

Question L1.1: Publish the implementation review checklist and run it at go-live

Q1.1: Does every endpoints AI asset receive an implementation review before go-live against a published per-archetype checklist that verifies controls are present and effective rather than documented, with controls the environment cannot expose recorded as unverifiable rather than passed?

Evidence Required:

• [ ] Publish a checklist per archetype that examines the built artifact rather than its documentation
• [ ] In this domain the review looks at the capability allowlist enforced by endpoint management rather than published as policy, device compliance genuinely required before access, data-loss prevention applied to input as well as egress, per-user and per-session limits configured with the declared at-limit behaviour, and session logging emitting at the declared level
• [ ] The discipline that makes this practice worth running is verifying presence and effect rather than presence alone: a logging configuration that exists but emits nothing, an allowlist enforced in audit mode, a kill switch wired to nothing, all pass a documentation check and fail a real one
• [ ] Run the review before go-live for every asset, and make go-live conditional on blocking findings being closed
• [ ] Where the review cannot verify a control because the environment does not expose it, record that explicitly as unverifiable rather than as passed, since silence is not a pass

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L1.2: Compare the built system against the approved design

Q1.2: Is the built system reconciled against the approved design and requirements pack, with every in-scope requirement given a verified, not-verified or unverifiable disposition and the design-to-implementation gap measured as a metric in its own right? For Endpoints, the assessor should be shown the go-live review record showing the allowlist in enforcing rather than audit mode.

Evidence Required:

• [ ] Reconcile what was built against the design review decision and the requirements pack: every requirement in scope for the asset's archetype gets a verified, not-verified or unverifiable disposition, and every condition attached to the design approval gets confirmed as met
• [ ] Record the design-to-implementation gap as a measure in its own right, because a gap that is never measured is assumed to be zero and never is
• [ ] Where implementation diverges from the approved design, treat it as a finding against the change rather than as a re-opening of the design decision, and route material divergence back to design review rather than absorbing it in the implementation record

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L1.3: Track findings to closure with severity and ownership

Q1.3: Do all findings carry an owner, severity, date and disposition in the shared issue backlog, with blocking findings closed before go-live and every accepted finding carrying a named acceptor with proportionate authority? In Endpoints, ask to see the go-live review record showing the allowlist in enforcing rather than audit mode.

Evidence Required:

• [ ] Give every finding a severity, an owner, a remediation date and a disposition of remediated, accepted or deferred
• [ ] An accepted finding requires a named acceptor with authority proportionate to the severity, because a finding that is neither fixed nor formally accepted leaves residual risk that nobody owns, which is the most common way implementation review findings evaporate
• [ ] Feed all findings into the issue-management backlog rather than maintaining a separate review register, so that AI findings age and escalate on the same rails as everything else
• [ ] Report finding density by archetype and by checklist item quarterly, which tells the architecture and education practices where implementations systematically diverge rather than where reviewers happened to look

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Level 1 Outcome Metrics

Metric	Baseline	Current	Target	Met?	Notes
Endpoints AI assets receiving an implementation review before go-live	___	___	100% of assets reaching production	☐
Blocking findings closed before go-live	___	___	100%	☐
Requirements given a verified, not-verified or unverifiable disposition at review	___	___	100% of in-scope requirements, with unverifiable recorded explicitly	☐
Findings carrying an owner, severity, date and disposition	___	___	100%	☐
Accepted findings carrying a named acceptor with proportionate authority	___	___	100%	☐

Where each number is read from:

• Endpoints AI assets receiving an implementation review before go-live — Review records reconciled against the deployment log
• Blocking findings closed before go-live — Finding tracker joined to deployment records
• Requirements given a verified, not-verified or unverifiable disposition at review — Implementation review records
• Findings carrying an owner, severity, date and disposition — Issue tracker records
• Accepted findings carrying a named acceptor with proportionate authority — Finding disposition records

Level 1 Oversight Metrics

"A human approves it" is not a control unless you measure whether they refuse. A metric that cannot fail is not a metric.

Metric	Current	Oversight failure it detects	Met?
Share of implementation reviews recording at least one finding before sign-off	___	rubber-stamping at implementation review, where sign-off is recorded without the review having examined anything, which is the default outcome when review is a release-checklist item	☐
Findings accepted as risk at sign-off rather than remediated, with a named acceptor	___	silent acceptance, where a finding is noted and neither fixed nor formally accepted, so nobody owns the residual risk	☐

Level 1 30-second check (E0)

• ☐ Yes ☐ No — Does anyone check the built AI system before it goes live?
• ☐ Yes ☐ No — Could you show that a control is working rather than configured?
• ☐ Yes ☐ No — Does an accepted security finding have someone's name against it?

---

Maturity Level 2, Comprehensive

Objective: Calibrate endpoints implementation review cadence to asset tier and detect configuration drift continuously, so controls verified at go-live stay verified.

Prerequisite cells: IR-Endpoints-L1, SM-Endpoints-L2, SR-Endpoints-L2, SA-Endpoints-L2

Question L2.1: Set re-review cadence by tier and material change

Q2.1: Is re-review cadence published per tier and driven by material-change triggers wired to change records and deployment events, with the calendar serving as a floor rather than the mechanism? Evidence this in the Endpoints domain against the go-live review record showing the allowlist in enforcing rather than audit mode.

Evidence Required:

• [ ] Publish the re-review cadence per tier: Critical at go-live, semi-annually and on every material change
• [ ] High at go-live, annually and on material change
• [ ] Medium at go-live and annually
• [ ] Define material change concretely for this domain, being an allowlist moved from enforcing to audit mode, an extension permission escalated at update, a limit raised for a demonstration and left raised, and logging reduced to control ingestion cost, and wire the triggers to change records and deployment events so the obligation is raised by the system rather than remembered
• [ ] The calendar is the weakest trigger available and should be the floor rather than the mechanism, because an asset that changed materially in month two carries a verified-once record for ten months and the record looks identical to an accurate one

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L2.2: Detect configuration drift continuously on high-tier assets

Q2.2: Are at least 90% of Critical-tier assets under continuous configuration drift detection with a 5 business day detection service level, and are detections routed into the shared backlog with tier-appropriate severity? For Endpoints, the assessor should be shown the go-live review record showing the allowlist in enforcing rather than audit mode.

Evidence Required:

• [ ] Stand up continuous drift detection for Critical- and High-tier assets covering the controls whose silent disablement would matter most
• [ ] In this domain the drift that recurs is an allowlist moved from enforcing to audit mode, an extension permission escalated at update, a limit raised for a demonstration and left raised, and logging reduced to control ingestion cost
• [ ] Detect it from configuration state, infrastructure-as-code divergence, policy enforcement mode and runtime telemetry, and alert on change rather than on schedule
• [ ] Drift detection is what converts implementation review from a point-in-time assertion into a continuing one, and it is the single highest-value activity at this level, because implementations rarely fail review, they pass and then change
• [ ] Publish a detection service level of 5 business days at Critical tier and route detections into the issue backlog with tier-appropriate severity rather than into a report

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L2.3: Apply tier-proportionate service levels and deferral authority

Q2.3: Do remediation service levels vary by tier and severity with adherence at or above 90% per tier, does deferral authority rise with tier, and are deferred findings later implicated in incidents tracked as a distinct measure? In Endpoints, ask to see the go-live review record showing the allowlist in enforcing rather than audit mode.

Evidence Required:

• [ ] Set remediation service levels by finding severity and asset tier, tighter at Critical, and report adherence rather than only backlog size
• [ ] Require deferral authority to rise with tier and severity, so a deferred Critical-tier finding needs an executive acceptance with an expiry and a quarterly review, while a Low-tier one needs the practice lead
• [ ] Report deferred Critical-tier findings to the sponsor alongside remediated ones, because a practice that reports only closures presents a picture that flatters it
• [ ] Track findings that were deferred and later implicated in an incident as a distinct measure, since that is the only feedback available on whether deferral judgment is sound rather than merely authorised

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Level 2 Outcome Metrics

Metric	Baseline	Current	Target	Met?	Notes
Critical-tier assets re-reviewed within their cadence and on every material change	___	___	≥95%	☐
Critical- and High-tier assets under continuous drift detection	___	___	≥90% of Critical, ≥75% of High	☐
Time to detect configuration drift on a Critical-tier control	___	___	≤5 business days at the 90th percentile	☐
Findings remediated within the tier and severity service level	___	___	≥90% per tier	☐
Deferred Critical-tier findings carrying executive acceptance with expiry and quarterly review	___	___	100%, with none past expiry	☐

Where each number is read from:

• Critical-tier assets re-reviewed within their cadence and on every material change — Review records joined to change events and inventory tier
• Critical- and High-tier assets under continuous drift detection — Drift detection coverage registry
• Time to detect configuration drift on a Critical-tier control — Drift detection event log
• Findings remediated within the tier and severity service level — Issue tracker with service-level fields
• Deferred Critical-tier findings carrying executive acceptance with expiry and quarterly review — Finding disposition records

Level 2 Oversight Metrics

"A human approves it" is not a control unless you measure whether they refuse. A metric that cannot fail is not a metric.

Metric	Current	Oversight failure it detects	Met?
Reviewer finding rate per review, compared across reviewers on comparable assets	___	reviewer drift, where the same implementation would pass with one reviewer and fail with another, making sign-off depend on assignment rather than on the code	☐
Implementation reviews per reviewer per week against the published ceiling	___	reviewer overload, where depth falls as volume rises and the review continues to be recorded as complete	☐
Critical-tier findings deferred past their remediation window with a recorded acceptance authority	___	deferral becoming the default disposition under release pressure, which converts a blocking finding into an advisory one without a decision	☐

Level 2 30-second check (E0)

• ☐ Yes ☐ No — Would you know if a security control on an AI asset was turned off last week?
• ☐ Yes ☐ No — Does a high-risk AI asset get re-reviewed more often than a low-risk one?
• ☐ Yes ☐ No — Does deferring a critical finding require senior sign-off?

---

Maturity Level 3, Industry-Leading

Objective: Verify endpoints implementation continuously in the pipeline and at runtime, and measure whether review judgment was right rather than only whether review happened.

Prerequisite cells: IR-Endpoints-L2, SA-Endpoints-L3, ML-Endpoints-L2, IM-Endpoints-L2

Question L3.1: Assert controls continuously in pipeline and runtime

Q3.1: Do at least 80% of verifiable controls carry continuous assertions running both in the pipeline and against the running system, with the unassertable remainder listed explicitly and reviewed quarterly for reduction? Evidence this in the Endpoints domain against the go-live review record showing the allowlist in enforcing rather than audit mode.

Evidence Required:

• [ ] Express each verifiable control as an assertion that runs both in the endpoint policy release pipeline against the change and against the running system on a continuous schedule
• [ ] Pipeline assertions prevent a regression from merging
• [ ] Runtime assertions catch what was changed outside the pipeline, and both are needed because the second is where drift actually originates
• [ ] Give each assertion a tier-proportionate severity, blocks the policy rollout on Critical-tier failures, and route runtime failures to the backlog automatically
• [ ] Publish the list of controls that cannot be asserted and review it quarterly with a reduction target, because that list is the true boundary of the verification and is otherwise assumed to be empty
• [ ] Assertions in this domain read endpoint management state, extension inventories and permissions, device compliance enforcement and gateway limits, most of which sit in IT systems and need an agreed data path

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L3.2: Concentrate human review on the unassertable

Q3.2: Is review false-negative rate measured by tracing sign-offs forward to incidents and reported to the sponsor as the primary quality measure, rather than coverage and timeliness which a review examining nothing carefully can also achieve? For Endpoints, the assessor should be shown the go-live review record showing the allowlist in enforcing rather than audit mode.

Evidence Required:

• [ ] Reserve human implementation review for the controls automation cannot assert and for the judgment calls automation cannot make, such as whether a control's implementation is adequate to its intent rather than merely present
• [ ] Retain sampling on automatically verified assets so that the assertions themselves are checked, because an assertion that has silently stopped evaluating reports success indistinguishably from one that is passing
• [ ] Track the share of reviewer hours spent on unassertable controls, which should rise as automation extends, and use it as the measure that the automation is releasing capacity rather than merely adding a second process alongside the first
• [ ] What automation cannot assert here is what users actually place into an assistant, which input data-loss prevention samples rather than establishes

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L3.3: Measure review judgment by forward tracing

Q3.3: Is sampling maintained on automatically verified assets so that an assertion which has silently stopped evaluating is detectable, and does at least 60% of reviewer time go to controls automation cannot assert? In Endpoints, ask to see the go-live review record showing the allowlist in enforcing rather than audit mode.

Evidence Required:

• [ ] Trace every sign-off forward
• [ ] Where a production incident is attributable to something implementation review examined and passed, record it as a review false negative and investigate whether the checklist, the assertion or the judgment was at fault
• [ ] Report the false-negative rate to the sponsor alongside coverage and throughput, and treat it as the practice's primary quality measure, because coverage and timeliness are both achievable by a review function that examines nothing carefully
• [ ] Run blind double-review on a sample of Critical-tier implementations quarterly and report the disagreement rate, using disagreements to refine checklists and assertions rather than to rank reviewers
• [ ] Forward tracing in this domain most often implicates sign-offs that assessed a capability's declared permissions rather than the permissions it acquired at a later update

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Level 3 Outcome Metrics

Metric	Baseline	Current	Target	Met?	Notes
Verifiable controls covered by a continuous assertion in pipeline and runtime	___	___	≥80%, with unassertable controls listed explicitly	☐
Critical-tier assertion failures blocked at the policy release gate or routed to an owner within the day	___	___	≥95%	☐
Review false-negative rate: incidents attributable to something review examined and passed	___	___	tracked and falling year over year, reported to the sponsor	☐
Share of reviewer hours spent on controls automation cannot assert	___	___	≥60% and rising	☐
Blind double-review disagreement rate on Critical-tier implementations	___	___	≤15%, with disagreements used to refine checklists and assertions	☐

Where each number is read from:

• Verifiable controls covered by a continuous assertion in pipeline and runtime — Assertion registry joined to the control catalogue
• Critical-tier assertion failures blocked at the policy release gate or routed to an owner within the day — Pipeline and runtime assertion logs
• Review false-negative rate: incidents attributable to something review examined and passed — Incident root-cause analysis joined to review records
• Share of reviewer hours spent on controls automation cannot assert — Review time recording by control classification
• Blind double-review disagreement rate on Critical-tier implementations — Double-review calibration records

Level 3 Oversight Metrics

"A human approves it" is not a control unless you measure whether they refuse. A metric that cannot fail is not a metric.

Metric	Current	Oversight failure it detects	Met?
Automated verification failures overridden by a human, with the override sustained across builds	___	override becoming the routine path around a check that is inconvenient rather than wrong, which is how automated verification quietly becomes advisory	☐
Sign-offs later contradicted by a production incident attributable to what the review examined	___	review that is procedurally complete and substantively wrong, which only forward tracing to incidents can reveal	☐
Blind double-review disagreement rate on Critical-tier implementations	___	reviewer drift at the tier where it matters most, invisible unless deliberately sampled	☐

Level 3 30-second check (E0)

• ☐ Yes ☐ No — Is control verification continuous or tied to a review date?
• ☐ Yes ☐ No — Do you know which controls your automation cannot check?
• ☐ Yes ☐ No — Have you measured whether reviews you passed later caused incidents?

---

Summary Scorecard

Level	Q1	Q2	Q3	Level score	Weakest evidence grade	Achieved?
L1	___	___	___	___	___	☐
L2	___	___	___	___	___	☐
L3	___	___	___	___	___	☐

Implementation Review (IR) maturity in the Endpoints domain: ____________

---

Document Version: HAIAMM v3.5 (generated)