Monitoring & Logging (ML) - Data Domain

HAIAMM Assessment Questionnaire v3.5

Generated from cells/ML-Data-L*.cell.json. Do not edit by hand. Canonical model: HAIAMM-v3.5-Framing.md · Handbook: HAIAMM-v3.5-Handbook.md.

---

Practice: Monitoring & Logging (ML) Domain: Data Business Function: Operations Purpose: Establish a logging baseline per data archetype and a detection set covering the top threats, with retention set by the longest applicable obligation. Scoring Model: Evidence + Outcome Metrics (see Scoring Methodology below)

---

Instructions

• Answer from current, implemented practice, not plans or aspirations.
• Each question has two components: Evidence (what you did) and Outcome Metrics (how well it worked).
• Levels are derived, never self-assigned. Record evidence and metric values; the level falls out of the scoring below.
• Silence is not a pass. A metric you cannot produce is left blank and counts as not met. Do not estimate.
• Answer progressively. Complete all Level 1 questions before Level 2.

Evidence grade

Record how each answer is known, not only what is claimed (v3.5 framing §6). A level claim is graded at its weakest field.

Grade	Meaning
E0	Asserted by a practitioner, unevidenced
E1	Asserted, with a link to a named artifact
E2	Emitted by a system as telemetry
E3	Machine-attested and independently checkable

---

Scoring Methodology

Tier	Score	Criteria
Fully Mature	1.0	Evidence complete + ≥3 outcome metrics meet targets
Implemented	0.67	Evidence complete + 2 outcome metrics meet targets
Partial	0.33	Evidence partially complete + <2 outcome metrics meet targets
Not Implemented	0.0	No substantive evidence of practice

Practice maturity level achieved = the highest level where all 3 questions score ≥ 0.67.

---

Maturity Level 1, Foundational

Objective: Establish a logging baseline per data archetype and a detection set covering the top threats, with retention set by the longest applicable obligation.

Prerequisite cells: SM-Data-L1, PC-Data-L1, TA-Data-L1

Question L1.1: Publish and verify the logging baseline per archetype

Q1.1: Is a logging baseline defined per data archetype covering identity, asset, model and version context, and verified as actually emitting rather than merely configured, for at least 85% of archetypes?

Evidence Required:

• [ ] Define a logging baseline per archetype covering corpus access with requester identity and purpose, retrieval queries and the documents returned, training and fine-tuning run records with their input corpora, embedding generation and re-index events, erasure operations and their propagation, and canary access events
• [ ] Verify that each stream is actually emitting rather than merely configured, by asserting the presence of recent events per stream per asset, because a logging configuration that exists and emits nothing passes every configuration check and fails the one investigation that needs it
• [ ] Include the identity and context fields that make a log answerable: who initiated, what asset, which model and version, which session
• [ ] A log that records that something happened without recording which model version did it cannot answer the question that follows any output-integrity incident
• [ ] Publish the baseline alongside the hardening baseline so telemetry travels with configuration

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L1.2: Build the detection set from the threat library

Q1.2: Do detections cover at least 70% of the top-rated threats from the library with uncovered threats published as gaps, and does every detection route to the shared issue backlog with an IM severity rather than to a dashboard? For Data, the assessor should be shown the corpus access log and the canary-trigger detection.

Evidence Required:

• [ ] Build detections for the threats the TA library rates highest, covering in this domain a canary appearing outside its governed store, retrieval volume or breadth departing from a requester's baseline, an unregistered corpus entering a training run, erasure that did not propagate within its window, and retrieval cost per query departing from the store median
• [ ] Map each detection to the threat identifiers it addresses and publish the threats with no detection, because coverage is the claim that matters and detection count is not a proxy for it
• [ ] Route every detection to the shared issue backlog with a severity from the IM taxonomy rather than to a dashboard, since a detection whose only destination is a dashboard has no owner and no clock
• [ ] Include at least one consumption-anomaly detection from the outset, expressed against a denominated metric, because it is the cheapest detection available and the only one that covers denial-of-wallet

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L1.3: Set retention by obligation and make logs usable

Q1.3: Is retention per log class traced to a named obligation rather than to storage cost, with redaction applied at write, and are log streams verified as queryable within a working day at investigation volumes? In Data, ask to see the corpus access log and the canary-trigger detection.

Evidence Required:

• [ ] Set retention per log class by the longest applicable regulatory, contractual or investigative requirement, and record which obligation drives each period so the decision is reviewable
• [ ] Retention is routinely set on storage cost by people who do not hold the obligation, and the error surfaces during an investigation or a regulator request when it can no longer be corrected
• [ ] Apply redaction at write rather than at read so that sensitive content is not retained in the clear pending a query-time control
• [ ] Verify that logs are actually queryable at the volumes an investigation would need, because a retention period that cannot be searched within a working day is not meaningfully retained

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Level 1 Outcome Metrics

Metric	Baseline	Current	Target	Met?	Notes
Data archetypes with a logging baseline defined and verified as emitting	___	___	≥85% of archetypes within 9 months	☐
Top-rated threats from the library covered by at least one detection	___	___	≥70%, with uncovered threats published as gaps	☐
Log classes with a retention period traced to a named obligation	___	___	100%	☐
Detections routed to the shared issue backlog with an IM severity	___	___	100% of active detections	☐
Log streams verified as queryable within a working day at investigation volumes	___	___	100% of Critical-tier asset log classes	☐

Where each number is read from:

• Data archetypes with a logging baseline defined and verified as emitting — Log stream presence assertions per asset
• Top-rated threats from the library covered by at least one detection — Threat-to-detection coverage map
• Log classes with a retention period traced to a named obligation — Retention configuration with obligation references
• Detections routed to the shared issue backlog with an IM severity — Detection routing configuration
• Log streams verified as queryable within a working day at investigation volumes — Query exercise records

Level 1 Consumption Metrics

Absolute spend is a ledger input, never a metric. Every row is divided by a unit of business work. If AI spend is not allocated to an owning team, record this level as blocked on allocation rather than scoring it down (v3.5 framing §12).

Metric	Denominator	Current	Economic failure it detects	Met?
Consumption-anomaly detections active against a denominated metric	per archetype detection set	___	no coverage at all for denial-of-wallet, which produces no availability signal and is therefore invisible to every other detection in the set	☐
Detection threshold expressed against consumption per unit of work rather than absolute spend	per retrieval query	___	thresholds set on absolute spend, which fire on ordinary growth and are then widened until they never fire at all	☐
Logging and retention cost per retained record, by log class	per retained record	___	retention silently shortened for cost, removing the evidence that monitoring, investigation and deployer-duty obligations all depend on	☐

Level 1 Oversight Metrics

"A human approves it" is not a control unless you measure whether they refuse. A metric that cannot fail is not a metric.

Metric	Current	Oversight failure it detects	Met?
Detections acknowledged but closed without an investigation note	___	alert closure as queue management rather than as analysis, which is invisible in any alert-volume metric and is the normal response to a noisy detection set	☐
Detections muted or suppressed for more than one week, with the suppressing authority recorded	___	silent suppression, where a noisy detection is muted to clear a queue and never re-enabled, so coverage falls with no decision taken	☐

Level 1 30-second check (E0)

• ☐ Yes ☐ No — Are your AI systems actually emitting logs, or just configured to?
• ☐ Yes ☐ No — Do you know why your AI logs are kept for the period they are kept?
• ☐ Yes ☐ No — Is there a detection that would fire on unusual AI spend?

---

Maturity Level 2, Comprehensive

Objective: Calibrate data detection and logging depth to asset tier, tune for alert quality, and establish behavioural baselines so departure is detectable.

Prerequisite cells: ML-Data-L1, SM-Data-L2, TA-Data-L2, IM-Data-L2

Question L2.1: Calibrate logging and detection depth to tier

Q2.1: Are logging and detection depth published per tier and derived automatically from the inventory, with sampling and retention positions stated per tier and reductions at Critical tier treated as material changes? Evidence this in the Data domain against the corpus access log and the canary-trigger detection.

Evidence Required:

• [ ] Publish what each tier receives: Critical gets full logging, the complete detection set tuned for the asset, behavioural baselining and the shortest alerting path
• [ ] Low gets baseline logging and the shadow-AI detections
• [ ] Derive the applicable set from the inventory tier automatically
• [ ] Publish the retention and sampling position per tier explicitly, because sampling is where cost pressure lands first and a Critical-tier asset whose telemetry is sampled has monitoring that is statistical at exactly the tier where specific answers are needed
• [ ] Treat a sampling or retention reduction at Critical tier as a material change requiring approval rather than an operational tuning decision
• [ ] Behavioural baselining in this domain covers retrieval breadth and volume per requester, because extraction looks like enthusiastic legitimate use until compared against a baseline

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L2.2: Establish behavioural baselines per Critical-tier asset

Q2.2: Do at least 90% of Critical-tier assets carry a current behavioural baseline alerting on departure rather than only on signature, with baseline shifts recorded explicitly rather than refreshed silently? For Data, the assessor should be shown the corpus access log and the canary-trigger detection.

Evidence Required:

• [ ] Build a behavioural baseline per Critical-tier asset covering its normal operating envelope: request and tool-call volumes, the distribution of actions taken, typical session shapes, identity and access patterns, and consumption per unit of work
• [ ] Alert on departure rather than only on signature, because the AI failure modes that matter most produce traffic that is individually well-formed and collectively abnormal, and a signature-based set cannot see an agent competently doing something nobody asked for
• [ ] Refresh baselines on a defined cadence and after any material change to the asset, and record baseline shifts explicitly, since an unexplained baseline shift is itself the signal and is lost if baselines are refreshed silently
• [ ] Detection-as-code in this domain is fragile to retrieval and embedding schema changes, which silently break a rule that continues to report healthy

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L2.3: Tune for alert quality and report precision with recall

Q2.3: Is detection quality measured with precision and recall reported together using analyst disposition labels, with a false-positive budget enforced, and is a stopped detection or silent stream detected within 4 hours? In Data, ask to see the corpus access log and the canary-trigger detection.

Evidence Required:

• [ ] Measure detection quality using analyst disposition labels, and report precision and recall together
• [ ] Precision alone rewards a detector that fires twice a year and misses everything, which is exactly the detector a tuning programme under alert-fatigue pressure will produce
• [ ] Set a false-positive budget per detection and tune or retire detections that exceed it, because analysts who learn that a detection is usually wrong stop reading it and the coverage is then notional
• [ ] Assert continuously that each detection is still evaluating and each stream still emitting, alerting within hours, since a stopped detection and a clean environment are indistinguishable in every dashboard
• [ ] Data detection content contributes well as extraction and canary-trigger rules, which are expensive to develop and widely applicable

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Level 2 Outcome Metrics

Metric	Baseline	Current	Target	Met?	Notes
Assets receiving the logging and detection depth derived from their tier	___	___	100% of Critical and High tier assets	☐
Critical-tier assets with a current behavioural baseline	___	___	≥90%, refreshed on cadence and after material change	☐
Detection precision and recall both reported per detection	___	___	100% of active detections report both	☐
Detections exceeding their false-positive budget, tuned or retired	___	___	100% actioned within the quarter	☐
Time to detect a silent log stream or stopped detection	___	___	≤4 hours	☐
Sampling or retention reductions at Critical tier approved as material changes	___	___	100%	☐

Where each number is read from:

• Assets receiving the logging and detection depth derived from their tier — Monitoring configuration joined to inventory tier
• Critical-tier assets with a current behavioural baseline — Baseline registry with refresh dates
• Detection precision and recall both reported per detection — Detection quality reporting with analyst disposition labels
• Detections exceeding their false-positive budget, tuned or retired — Detection tuning records
• Time to detect a silent log stream or stopped detection — Telemetry health monitoring alerts
• Sampling or retention reductions at Critical tier approved as material changes — Change records filtered to telemetry configuration

Level 2 Consumption Metrics

Absolute spend is a ledger input, never a metric. Every row is divided by a unit of business work. If AI spend is not allocated to an owning team, record this level as blocked on allocation rather than scoring it down (v3.5 framing §12).

Metric	Denominator	Current	Economic failure it detects	Met?
Consumption-anomaly detections operating against a per-asset behavioural baseline	per retrieval query	___	consumption thresholds set globally rather than per asset, which either miss a small asset's drain entirely or drown a large asset's normal operation in alerts	☐
Time from a consumption anomaly beginning to its detection	per consumption anomaly	___	slow detection on the one incident class where the loss accrues continuously and nothing else in the monitoring set will notice	☐
Telemetry cost per retained record at Critical tier, tracked against sampling decisions	per retained record	___	sampling introduced at Critical tier for cost reasons, which makes monitoring statistical at exactly the tier where specific answers are required	☐

Level 2 Oversight Metrics

"A human approves it" is not a control unless you measure whether they refuse. A metric that cannot fail is not a metric.

Metric	Current	Oversight failure it detects	Met?
Alert disposition distribution per analyst, compared across analysts on comparable queues	___	disposition drift, where the same alert would be escalated by one analyst and closed by another, making detection outcome depend on shift rather than on evidence	☐
Alerts per analyst per shift against the published ceiling	___	analyst overload, where triage quality falls as volume rises while closure rates continue to look healthy	☐
Critical-tier detections closed as false positive and later implicated in an incident	___	dismissal under load of exactly the alerts that mattered, visible only by tracing closures forward	☐

Level 2 30-second check (E0)

• ☐ Yes ☐ No — Do you know what normal looks like for your highest-risk AI asset?
• ☐ Yes ☐ No — Do you report how often your detections miss, or only how often they are right?
• ☐ Yes ☐ No — Would you notice within a day if a detection stopped running?

---

Maturity Level 3, Industry-Leading

Objective: Engineer data detections as versioned, tested code, close the loop from incident to detection continuously, and contribute detection content back to the field.

Prerequisite cells: ML-Data-L2, TA-Data-L3, IM-Data-L3

Question L3.1: Engineer detections as versioned, tested code

Q3.1: Are at least 85% of active detections held in version control with an owner, documented intent and test cases running continuously against replayed or synthetic events, so a detection broken by an upstream log change is caught by its own test? Evidence this in the Data domain against the corpus access log and the canary-trigger detection.

Evidence Required:

• [ ] Move detection content into version control with the same discipline as application code: each detection carries an owner, a version, a documented intent, and test cases that assert it fires on known-positive events and does not fire on known-negative ones
• [ ] Run the tests continuously against replayed or synthetic events, because the most common silent failure in monitoring is a detection that still exists, still evaluates and no longer matches anything due to a log-format change upstream, and nothing in a conventional monitoring stack reports that
• [ ] Deploy through a pipeline with review, so a detection change is as traceable as a code change and can be reverted as easily
• [ ] Behavioural baselining in this domain covers retrieval breadth and volume per requester, because extraction looks like enthusiastic legitimate use until compared against a baseline
• [ ] Detection tests in this domain replay retrieval and canary-access events, and the known-negative cases matter most because legitimate heavy retrieval is common

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L3.2: Close the incident-to-detection loop continuously

Q3.2: Is every incident and adversarial finding assessed for detection coverage within 5 business days, with uncovered cases producing a deployed detection and its tests within 10, and detections that existed but did not fire treated as defects with owners? For Data, the assessor should be shown the corpus access log and the canary-trigger detection.

Evidence Required:

• [ ] Require every incident and every adversarial finding to be assessed for detection coverage within days: did a detection fire, should one have, and if so why did it not
• [ ] Where no detection existed, author one and deploy it within 10 business days, with its test cases
• [ ] Where one existed and did not fire, treat that as a detection defect with an owner rather than as an incident detail
• [ ] Track the share of incidents detected internally before external report as the practice's headline measure, since it is the one number that expresses whether monitoring is working, and report it alongside coverage rather than instead of it
• [ ] Detection-as-code in this domain is fragile to retrieval and embedding schema changes, which silently break a rule that continues to report healthy

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Question L3.3: Contribute detection content back

Q3.3: Is internal detection share reported as the headline outcome measure alongside coverage, and does the organisation contribute at least four accepted detection artifacts per year in deployable rather than descriptive form? In Data, ask to see the corpus access log and the canary-trigger detection.

Evidence Required:

• [ ] Contribute at least four substantive artifacts per year in deployable form: detection rules, behavioural baselining approaches, log schemas for AI telemetry and the test cases that verify a detection works
• [ ] Shared detection content is scarce in AI security while the failure modes are largely common across organisations, which makes this among the highest-leverage contributions available, since a peer can deploy a rule and cannot deploy a description of one
• [ ] Work through OWASP projects, MITRE ATLAS, sector information-sharing bodies and detection-engineering communities
• [ ] Anonymise, review legally, coordinate disclosure where content reveals a provider weakness, and prioritise contributing the detections that most often produce actioned findings internally
• [ ] Data detection content contributes well as extraction and canary-trigger rules, which are expensive to develop and widely applicable

Evidence grade for this question: ☐ E0 ☐ E1 ☐ E2 ☐ E3

Answer:

• ☐ Fully Mature (Evidence complete + ≥3 metrics meet targets)
• ☐ Implemented (Evidence complete + 2 metrics meet targets)
• ☐ Partial (Evidence partially complete + <2 metrics meet targets)
• ☐ Not Implemented (No evidence)

Evidence Location: _________________________________

Notes: ___________________________________________

Level 3 Outcome Metrics

Metric	Baseline	Current	Target	Met?	Notes
Detections held in version control with an owner, intent and test cases	___	___	≥85% of active detections	☐
Detections whose tests run continuously against replayed or synthetic events	___	___	≥85%, with failures alerting within the day	☐
Incidents and adversarial findings assessed for detection coverage within 5 business days	___	___	100%	☐
New detections authored and deployed within 10 business days of an uncovered incident	___	___	≥90%	☐
Incidents detected internally before external report	___	___	≥85% and rising, reported as the headline measure	☐
Accepted external detection-content contributions per year	___	___	≥4 accepted in deployable form	☐

Where each number is read from:

• Detections held in version control with an owner, intent and test cases — Detection repository
• Detections whose tests run continuously against replayed or synthetic events — Detection test pipeline results
• Incidents and adversarial findings assessed for detection coverage within 5 business days — Incident and finding records with coverage assessment
• New detections authored and deployed within 10 business days of an uncovered incident — Detection repository change log joined to incident records
• Incidents detected internally before external report — Incident records with detection source
• Accepted external detection-content contributions per year — Contribution log with acceptance references

Level 3 Consumption Metrics

Absolute spend is a ledger input, never a metric. Every row is divided by a unit of business work. If AI spend is not allocated to an owning team, record this level as blocked on allocation rather than scoring it down (v3.5 framing §12).

Metric	Denominator	Current	Economic failure it detects	Met?
Consumption-anomaly detections held as tested code with known-positive cases	per consumption detection	___	a consumption detection that silently stopped matching after a billing or telemetry schema change, leaving denial-of-wallet uncovered while the detection appears active	☐
Time from consumption anomaly onset to internal detection	per retrieval query	___	detection latency on the incident class where loss accrues continuously, which is the only class where detection time and loss are the same quantity	☐
Telemetry and detection platform cost per monitored asset	per monitored asset	___	monitoring cost growing faster than the estate, which eventually triggers a sampling or retention cut that removes coverage at the tiers that need it most	☐

Level 3 Oversight Metrics

"A human approves it" is not a control unless you measure whether they refuse. A metric that cannot fail is not a metric.

Metric	Current	Oversight failure it detects	Met?
Detection changes deployed without review, by severity of the detection	___	detection content changed directly in the console, which is untraceable, unrevertable and the most common way coverage is lost without a decision	☐
Detections whose tests have been failing for more than one week without action	___	test failure normalised, which restores exactly the silent-failure mode the tests were introduced to remove	☐
Incidents detected externally rather than internally, with a coverage assessment completed	___	external detection accepted without asking why internal detection missed it, which forfeits the single most informative signal monitoring produces	☐

Level 3 30-second check (E0)

• ☐ Yes ☐ No — Are your detections in version control with tests?
• ☐ Yes ☐ No — Would you know if a detection stopped matching because a log format changed?
• ☐ Yes ☐ No — Have you published a detection rule others can deploy?

---

Summary Scorecard

Level	Q1	Q2	Q3	Level score	Weakest evidence grade	Achieved?
L1	___	___	___	___	___	☐
L2	___	___	___	___	___	☐
L3	___	___	___	___	___	☐

Monitoring & Logging (ML) maturity in the Data domain: ____________

---

Document Version: HAIAMM v3.5 (generated)