Monitoring & Logging (ML)

Processes Domain - HAIAMM v3.0

Practice Overview

Objective: Establish the logging baseline per AI/HAI process archetype, operate a small high-signal detection set targeting the top TA-Processes threats, and produce the evidence trail that proves EU AI Act deployer duties, GDPR processor obligations, and ISO/IEC 42001 AIMS requirements for business workflows embedding AI/HAI, on demand, inside a published SLA.

Description: ML-Processes captures the signals produced by every AI/HAI-embedded business workflow the organization operates, decision pipelines, customer-facing flows, human-AI collaboration chains, back-office augmentation workflows, approval/review workflows, content-generation workflows, and knowledge-management workflows. For each archetype it specifies the exact events to capture (decision events, HITL review events, disclosure events, override events, admin-audit events, and identity events), the retention window required to satisfy the longest applicable regulation, and the export path that supports auditor review and DSAR fulfillment within a published SLA. On top of the logging baseline it operates a bounded, purposeful detection set, each detection tied to a TA-Processes archetype threat, with a named owner, a defined query, and an active tuning record. The full corpus produced by ML-Processes is the primary evidence artifact for PC-Processes's compliance map: EU AI Act Art. 12 deployer-duty logs, GDPR Art. 22 contestation evidence, GDPR Art. 30 records of processing, and ISO/IEC 42001 AIMS operational evidence.

Context: Logging AI/HAI-embedded workflows is not the same as logging classic business process events. A decision-pipeline event must carry the request-id, principal, AI output and confidence score, decision threshold, final decision, model and version, and override flag, not merely a workflow step completion status. A HITL review event must record the reviewer's authenticated identity, the AI suggestion, the reviewer's decision, time-spent, and rationale, not only the queue-item closure. An Art. 50 disclosure event must capture the template version rendered and the execution-id to support an individual's contestation claim. None of this exists by default in standard BPM audit logs or SIEM tooling unless the archetype's event schema has been explicitly defined and instrumented. ML-Processes makes that schema explicit, per archetype, from day one, so the organization is not reconstructing an evidence trail from incomplete telemetry at the moment a regulator, DSAR, or adverse-action contestation demands it.

Maturity Level 1

Objective: Establish the per-archetype logging baseline, operate a small high-signal detection set targeting the top TA-Processes threats, and produce an on-demand evidence trail that satisfies EU AI Act Art. 12, GDPR Arts. 22 and 30, and ISO/IEC 42001 AIMS requirements within a published SLA

At this level, the program makes every production AI/HAI-embedded workflow observable with a defined minimum event schema, closes the most dangerous detection gaps (rubber-stamp HITL, reviewer-capacity saturation, override-audit anomaly, disclosure suppression, shadow-AI-in-process, and workflow-config drift), and demonstrates that the resulting log corpus can produce deployer-duty evidence for regulators and auditors on demand.

Dependencies

• SM-Processes L1 (required): the workflow inventory and archetype taxonomy define what must be instrumented; ML-Processes L1 cannot baseline what SM-Processes has not yet catalogued.
• SA-Processes L1 (required): reference patterns specify where logs originate (BPM audit log, review-UI event stream, classification-routing gateway, disclosure rendering layer); the patterns specify what to log.
• EH-Processes L1 (required): hardened identity controls (SSO + MFA on review UIs, immutable decision logs, tamper-evident override audit trail) produce the identity and audit events that ML-Processes captures; without EH-Processes instrumentation there are no usable per-reviewer identity events or tamper-evident audit sources.
• TA-Processes L1 (required): archetype threat library drives detection priority; the high-signal detection set is selected from TA-Processes's top threat list, not invented independently.
• Supports / unblocks: IM-Processes L1 (detections become the primary runtime input to the workflow issue backlog); PC-Processes L1 (ML-Processes log retention and export path constitutes the compliance evidence trail the PC-Processes priority compliance map requires).

Desired Outcomes

• Every production AI/HAI-embedded workflow emits the per-archetype minimum event schema; no workflow produces logs architecturally insufficient for deployer-duty evidence or GDPR Art. 22 contestation support.
• Retention meets or exceeds the longest applicable compliance window; export path demonstrated for each archetype at least annually.
• A small, bounded detection set is live, each detection tied to a named TA-Processes archetype threat and at least one HAI TTP tag (EA / AGH / TM / RA); each with an owner, a query, an SLA, and a tuning record; false-positive rate tracked per detection.
• EU AI Act Art. 12 deployer-duty logs, GDPR Art. 22 contestation evidence, GDPR Art. 30 records-of-processing entries, and ISO/IEC 42001 AIMS operational evidence can be satisfied from ML-Processes log stores within the published SLA (on-demand pull ≤24 hours).
• Shadow-AI-in-process (new AI step detected in a workflow not in inventory) and workflow-config drift (workflow definition changed without a corresponding DR record) are detectable from ML-Processes signals alone.

Activities

A) Establish the per-archetype logging baseline

Define and instrument the minimum event schema for each archetype in the SM-Processes inventory. Each event record includes: event-id / correlation-id, principal (user or service account), timestamp, archetype tag, workflow-id linked to the SM-Processes inventory, and the archetype-specific fields below. PII scrubbing applied per SR-Processes data-boundary requirements before logging where logging the raw field would create a regulated-data exposure.

Decision pipeline:

• Decision event: request-id, principal, AI output (or hash where regulated data may be present), AI confidence score, decision threshold, final decision, model and version, override flag (yes/no), timestamp.
• Override event: reviewer identity (SSO-resolved), override rationale, AI recommendation before, decision after, timestamp, workflow-id.
• Decision-distribution metric event: rolling distribution of decision outcomes by class (for protected-attribute intersection monitoring, security-relevant only, not fairness-standalone), model version, time window.

Customer-facing flow:

• Interaction event: session-id, PII-redacted content hash, AI step identifier, disclosure-shown flag, template version, timestamp.
• Escalation event: session-id, escalation trigger, destination (human agent, supervisor), timestamp.
• Brand-safety filter event: filter triggered (name), action taken (block / redact / reroute), session-id, timestamp.

Human-AI collaboration chain (HITL):

• Review event: reviewer identity (SSO-resolved), item-id, AI suggestion, reviewer decision, time-spent (seconds), rationale (mandatory for Critical/High-tier), timestamp.
• Reviewer-capacity event: queue depth, estimated-SLA-breach time, SLA-at-risk flag, timestamp.

Back-office augmentation:

• Assistant-session event: session-id, principal, workflow step, timestamp.
• Tool-call event: tool name, arguments (or hash for sensitive parameters), return value (or hash), principal, success/fail, timestamp.
• Output-review-gate event: gate triggered (yes/no), reviewer identity, decision, timestamp.

Approval/review workflow:

• Screen event: item-id, AI screening result, threshold applied, tier-routing decision, timestamp.
• Threshold event: threshold value, model version, workflow version, timestamp.
• Tier-routing event: item-id, tier assigned, routing rule applied, timestamp.
• Class-shift-monitor event: decision-distribution delta vs. baseline, protected-attribute class shift flag (security-intersection only), timestamp.

Content-generation workflow:

• Generation event: request-id, principal, content type, model and version, timestamp.
• Output-review event: reviewer identity, item-id, review decision (approve / revise / reject), timestamp.
• Copyright-filter event: filter triggered (name), action taken, request-id, timestamp.
• Downstream-emission event: where the generated content was published or routed, request-id, timestamp.

Knowledge-management workflow:

• Query event: query-id, principal, query text (or hash), workflow-id, timestamp.
• Retrieval event: document IDs retrieved, classification labels, provenance references, query-id, timestamp.
• Provenance event: source attribution confirmed (yes/no), provenance chain, query-id, timestamp.
• Role-based-policy event: access policy checked, role, document-id, decision (allowed / denied), timestamp.

Admin-audit events (all archetypes):

• Workflow-definition changes (version, author, change type, AI step add/modify/remove vs. other).
• Threshold changes to AI decision thresholds or screening rules.
• HITL-policy changes (reviewer-pool membership, rotation rules, capacity thresholds).
• Disclosure-template changes (template version, author, change description).
• Reviewer-pool membership changes (added, removed, role changes).
• Any configuration change to a workflow step's AI model binding or version.

Identity events (cross-archetype):

• SSO sign-ins to BPM tools (Camunda, Temporal, Argo admin consoles), review UIs, and workflow management consoles.
• Reviewer-session start and end events (session-id, reviewer identity, workflow assignment).

Retention and exportability:

• Retention window meets or exceeds the longest applicable requirement: EU AI Act Art. 12 high-risk-system logs ≥6 months; GDPR Art. 22 contestation evidence (retain for the applicable contestation window per jurisdiction); sector-specific windows (FCRA adverse-action records 25 months; FINRA 6 years; HIPAA 6 years); where multiple windows apply to a single workflow, the longest governs.
• Export path: JSON or structured CSV from the log store tested at least annually; on-demand pull SLA ≤24 hours for evidence requests from auditors, regulators, or legal hold; DSAR-capable export ≤72 hours for individual rights requests touching a decision or contestation record.
• Log integrity: write-once or append-only storage for admin-audit and decision-log tiers; access-control separation between workflow application teams and log store administrators.

B) Operate a small high-signal detection set

L1 target: ≤12 detections, each tied to a TA-Processes archetype threat and at least one HAI TTP tag or ATLAS tactic. Each detection has: owner, detection query, SLA (time-to-IM-ticket), and last-tuned date. False-positive rate tracked per detection; monthly tuning review.

Core detection set:

• Rubber-stamp HITL detection (EA TTP), reviewer decision matches AI recommendation ≥98% over a rolling 100-item window for a given reviewer on Critical/High-tier workflows; fires per-reviewer and per-workflow.
• Reviewer-capacity saturation detection (EA TTP), SLA breach imminent: estimated time to SLA breach for a HITL queue falls below the configured warning threshold; fires per HITL step.
• Decision-distribution drift detection (security-intersection only, bias-driven gate bypass / fairness manipulation), class-shift on protected-attribute decision distribution beyond a defined sigma from baseline; fires per decision pipeline where this monitoring is in scope per SR-Processes requirements.
• Override-audit anomaly detection (ATLAS TA0008 Defense Evasion), override event present in the workflow log without a corresponding rationale field for a Critical/High-tier workflow; fires immediately on detection.
• Disclosure-suppression detection (ATLAS TA0008 Defense Evasion), Art. 50 disclosure UI not rendered in a customer-facing flow execution where the workflow definition requires it; fires on any execution where the disclosure-shown flag is absent.
• Affected-persons-rights-response SLA breach detection, contestation response window has elapsed without a logged response event for a GDPR Art. 22 workflow; fires at SLA expiry.
• Shadow-AI-in-process detection (ATLAS TA0001 Reconnaissance / EA TTP), new AI step detected in a workflow definition version that does not appear in the SM-Processes inventory; fires on definition version comparison.
• Workflow-config drift without DR record detection (ATLAS TA0008 Defense Evasion), a workflow definition version was promoted to production without a corresponding DR-Processes decision record in the last 5 business days for material changes.

Each detection routes to the IM-Processes backlog on fire; median detection-to-ticket time target ≤1 hour for Critical-tier workflows.

C) Produce and drill the deployer-duty evidence trail

ML-Processes is the primary evidence source for PC-Processes's priority compliance map (§10.2). At L1, wire the log store to the compliance requirements:

• EU AI Act Art. 12 (high-risk-system deployer-duty logs): for every workflow assessed as involving an Annex III high-risk AI system use case, confirm that decision events, override events, disclosure-shown events, and admin-audit events are captured and retained at the required window; produce a deployer-duty evidence view (log record + retention attestation + export test result) for each such workflow.
• GDPR Art. 22 (automated decision-making contestation evidence): for every decision-pipeline workflow subject to Art. 22, confirm that the decision log contains the AI output, threshold, final decision, and override flag for each individual decision; confirm the export path can produce this record for a named individual within the DSAR response window; link the log-store retention policy to the Art. 22 contestation evidence requirement.
• GDPR Art. 30 (records of processing): for every workflow processing personal data, the decision events, interaction events, and reviewer events with principal identity, data-class tag, and purpose label constitute the records-of-processing operational entries.
• ISO/IEC 42001 AIMS (operational evidence): workflow-definition version events, admin-audit events, threshold change events, and reviewer-pool change events constitute the AIMS operational records.

Quarterly deployer-duty drill: pull the deployer-duty evidence package for one randomly selected production workflow per archetype within the published SLA (≤24 hours from request to assembled package). Record drill results; gaps route to IM-Processes.

Outcome Metrics (L1)

Process Metrics (leading)

• Archetype-baseline gap list maintained, every production workflow scored against its archetype baseline; gaps on IM-Processes backlog with named owner.
• Detection tuning cadence, monthly review per detection; stale or superseded detections retired; new detection candidates from IM post-incident reviews queued.
• Retention / export test cadence, at least annually per archetype, with documented evidence.
• Deployer-duty drill cadence honored, quarterly, covering all archetypes in rotation.

Effectiveness Metrics (business value)

• Workflow incidents detected internally before external notification, trend over quarters.
• Regulator / auditor / legal evidence requests turned around inside the published SLA (≤24 hours).
• HITL rubber-stamp incidents caught by rubber-stamp detection before a regulatory audit, documented cases per year.

Success Criteria

• Per-archetype logging baseline published and instrumented for ≥90% of production AI/HAI-embedded workflows.
• ≤12-detection high-signal set live, each with owner, detection query, SLA, and monthly tuning record; false-positive rate tracked per detection.
• Retention meets the longest applicable regulatory window for every production workflow; export path tested at least annually; DSAR-capable export ≤72 hours.
• EU AI Act Art. 12, GDPR Arts. 22 and 30, and ISO/IEC 42001 AIMS evidence-trail wiring documented; quarterly deployer-duty drill executed inside the ≤24-hour SLA.

Maturity Level 2

Objective: Calibrate logging depth and detection set to the SM-Processes L2 risk-tier rubric; integrate with SIEM for cross-workflow correlation; and feed incident-driven and ST-driven detection updates into a continuous tuning loop

At this level, monitoring intensity matches risk tier. Critical-tier decision pipelines and HITL workflows receive full event corpora retained for the longest regulatory window; Low-tier back-office augmentation workflows receive the baseline only. The SIEM ingests ML-Processes log feeds and executes cross-workflow correlation rules. The detection set evolves continuously from IM-Processes post-incident reviews and ST-Processes findings. Anomaly detection on reviewer behavior and decision distributions supplements the rule-based set for Critical and High-tier workflows.

Dependencies

• ML-Processes L1 (required): per-archetype logging baseline, detection set, and deployer-duty evidence trail must be operational before tier calibration is meaningful.
• SM-Processes L2 (required): risk-tier rubric (Critical / High / Medium / Low) and tier-treatment matrix drive per-tier logging depth and detection priority.
• EH-Processes L2 (required): hardened HITL controls (dedicated reviewer pools, JIT access, log integrity verification) at L2 produce richer reviewer-behavior and override signals that ML-Processes L2 consumes for anomaly baselining.
• IM-Processes L1+ (required): post-incident reviews from IM feed detection-tuning and new-detection requests into ML's review cycle.
• ST-Processes L1+ (required): security test findings identify gaps in the detection set and drive new detection candidates.
• Supports / unblocks: PC-Processes L2 (tier-calibrated compliance evidence bundles auto-assembly requires ML L2 log completeness signals); IM-Processes L2 (richer detections feed the tiered incident playbook with higher-fidelity severity signals).

Desired Outcomes

• Logging intensity visibly differentiates by tier: Critical-tier workflows retain full event corpora (decision events with full AI output, HITL review events with rationale, disclosure-completion events) for the longest regulatory window; Low-tier workflows produce only the baseline schema.
• SIEM integration enables cross-workflow correlation, a single actor interacting anomalously with multiple AI/HAI-embedded workflows surfaces as a correlated detection rather than isolated per-workflow alerts.
• The detection set evolves quarterly from a defined, governed feedback loop (IM post-incident reviews + ST findings + external advisory updates).
• Anomaly detection establishes behavioral baselines for Critical and High-tier reviewer populations and decision distributions; rule-based detections that fire frequently without incident are refined or retired.

Activities

A) Tier-calibrated logging depth

Apply the SM-Processes L2 tier-treatment matrix to logging configuration:

• Critical tier: full decision event content (AI output text or hash where regulated; full confidence scores; override rationale text), full HITL review event content (reviewer identity, time-spent, full rationale), full disclosure-completion events, full admin-audit events at maximum fidelity; retained for the longest regulatory window; per-workflow log partitioning (Critical-tier workflow logs partitioned from other tier logs).
• High tier: full decision and HITL events retained; standard admin-audit and identity events; core detections active.
• Medium tier: decision and HITL events with hashed content retained for regulatory window; standard admin-audit; baseline detections active.
• Low tier: baseline logging schema only; workflow-config drift and shadow-AI detections only.

For every Critical-tier workflow, the ML-Processes log store is the primary source for PC-Processes's compliance evidence bundle (ML logging-baseline validation ≤30 days for Critical).

B) SIEM integration and cross-workflow correlation

• Ingest all tier-appropriate ML-Processes log feeds into the SIEM.
• Author and maintain at least three cross-workflow correlation rules at L2:
• Multi-workflow rubber-stamp correlation: the same reviewer exhibits rubber-stamp behavior (matches-AI ≥98%) on two or more Critical/High-tier workflows in the same rolling window, fires a unified incident.
• Disclosure-suppression plus decision-outcome-shift: disclosure-suppression detection on a customer-facing flow correlates to a shift in that flow's decision distribution in the same time window, escalates to Critical regardless of workflow tier.
• Shadow-AI-in-process plus admin-audit-gap: a shadow-AI-in-process detection correlates to a missing admin-audit event for a workflow definition change in the same time window, signals a covert workflow modification.
• Cross-workflow correlation alerts route to IM-Processes at the tier of the highest-tier workflow involved.

C) Detection tuning loop: IM post-incident and ST feedback

Operate a quarterly detection review cycle:

• IM-Processes post-incident reviews that touch a logging or detection gap generate a detection-update request (new detection, tuned query, or retired false-positive rule).
• ST-Processes findings (HITL bypass test, disclosure rendering test, workflow-isolation test) that are not caught by the current detection set generate a detection-gap finding routed to ML-Processes.
• External advisory updates (EU AI Act enforcement decisions, NYC LL 144 audit findings, EEOC bias enforcement relevant to decision pipelines, OECD AI incident disclosures) are assessed quarterly; each applicable update either adds a new detection candidate or updates an existing detection's query.
• Monthly anomaly-baseline refresh for Critical and High-tier workflows: normal reviewer behavior baseline (decision-match rate, time-spent distribution, rationale length distribution) and decision-distribution baseline refreshed from the previous 30-day window; anomaly threshold auto-tunes to maintain target false-positive rate.
• Each detection has a last-tuned date and false-positive rate; detections that have not fired a true positive in 90 days or that exceed a 20% false-positive rate are reviewed for retirement at the quarterly cycle.

Outcome Metrics (L2)

Process Metrics (leading)

• Anomaly baseline refresh cadence honored (monthly for Critical; quarterly for High).
• SIEM correlation rule health monitored (rules producing alerts within expected frequency; no rule silent for >90 days without investigation).
• Detection-gap review calendar on schedule; IM and ST feedback queues reviewed at each quarterly cycle.
• Retention-tier calibration reconciled with SM-Processes inventory tier changes, when a workflow is re-tiered, logging depth updated within 14 days (Critical re-tier) or 30 days (other tiers).

Effectiveness Metrics (business value)

• True-positive rate improvement quarter-over-quarter as detection tuning loop matures.
• Cross-workflow correlation incidents that would have appeared as isolated per-workflow findings at L1 now unified, reduces IM mean-time-to-understand for complex workflow incidents.
• Compliance evidence bundle ML validation element completing inside PC-Processes staleness threshold with no manual intervention.

Success Criteria

• Tier-calibrated logging depth applied to 100% of SM-Processes inventory with current tier assignments; Critical-tier full event corpus retention confirmed.
• SIEM integration live; ≥3 cross-workflow correlation rules active.
• Quarterly detection tuning loop operating with IM-Processes and ST-Processes feedback; ≥1 net change per cycle.
• ≥90% of Critical/High-tier workflows with anomaly-detection baselines; FP rate tracked and trending down.
• ML logging-baseline validation element fresh (≤30 days) for all Critical-tier workflows in PC-Processes compliance evidence bundles.

Maturity Level 3

Objective: Express detections as code with automated deployment; apply behavioral anomaly detection to reviewer and decision-distribution corpora; and contribute anonymized detection signatures and telemetry schemas to OECD AI, ISO/IEC 42005, and sector ISACs

At this level, detections are version-controlled software artifacts deployed through CI/CD. Behavioral anomaly detection on reviewer corpora and decision distributions surfaces novel manipulation patterns that rule-based detections miss. The detection library and telemetry schemas are contributed back to the AI-assurance ecosystem, OECD AI, ISO/IEC 42005, CSA AI Safety Initiative, and sector ISACs.

Dependencies

• ML-Processes L2 (required): tier-calibrated logging, SIEM integration, and detection tuning loop must be mature before automation is trustworthy.
• PC-Processes L3 (required): continuous compliance attestation pipeline consumes ML-Processes log signals; attestation SLO depends on ML L3 log freshness guarantees.
• SM-Processes L3 (required): automated inventory and tier-maintenance events trigger automated detection-set updates (new workflow at Critical tier auto-provisions the full detection set for that archetype).

Desired Outcomes

• Detection-as-code: every detection in the set is a version-controlled artifact deployed via the same CI/CD pipeline as the workflow configuration it monitors.
• Behavioral anomaly detection on reviewer corpora and decision-distribution time series identifies reviewer-manipulation patterns, unusual decision-distribution shifts, and novel disclosure-suppression techniques that elude rule-based detections.
• The organization is a net contributor to AI workflow governance telemetry standards, OECD AI, ISO/IEC 42005, CSA, and sector ISAC detection-sharing feeds.
• Industry peers can adopt contributed detection schemas and telemetry standards without significant adaptation.

Activities

A) Detection-as-code

• Every detection in the detection set expressed as a versioned, tested artifact in source control (detection query + metadata: owner, SLA, archetype tag, HAI-TTP tag, false-positive threshold, last-test-result).
• Detection CI/CD pipeline: changes to detection code trigger a test suite (unit tests over synthetic workflow log data, integration tests against a log replay environment) before production deployment.
• Detection deployment via the same change-management pipeline as workflow configuration; detection changes are reviewed, not applied ad hoc in the SIEM console.
• Detection coverage automatically checked on SM-Processes inventory change events: when a new archetype is registered or a workflow is re-tiered to Critical, automation verifies the required detection set is active for that workflow and opens a gap finding if not.

B) Behavioral anomaly detection on workflow corpora

Apply unsupervised and semi-supervised anomaly models to the reviewer-behavior and decision-distribution corpora for Critical and High-tier workflows:

• Reviewer-behavior anomaly: reviewer sessions whose decision-pattern sequence (match rate, time-spent distribution, rationale frequency, override rate) is a statistical outlier from that reviewer's normal baseline and from the reviewer-pool baseline, signals reviewer fatigue, coercion, or systematic override manipulation.
• Decision-distribution anomaly: rolling decision-outcome distribution that shifts beyond a defined threshold from the established baseline, for security-intersection cases (where a bias-driven shift could constitute a gate-bypass or fairness-manipulation attack), fires a detection even without a specific prohibited-attribute signal.
• Disclosure-completion anomaly: per-execution disclosure completion rate dropping below baseline for a customer-facing flow on a rolling time window, may indicate a rendering failure, a covert A/B-test suppression, or a workflow modification that removed the disclosure step.
• Knowledge-management RAG-behavior anomaly: retrieval patterns (document-id sequences, provenance chain distributions) that deviate from normal query behavior for a given role-class, potential RAG-poisoning signal or unauthorized knowledge-base manipulation.

Anomaly model outputs feed the same detection-to-IM-ticket pipeline as rule-based detections. Anomaly models retrained monthly; model retraining produces a new version tracked in the model registry.

C) Contribute detection signatures and telemetry schemas

• OECD AI governance, contribute AI/HAI workflow monitoring telemetry schema (decision event, HITL review event, disclosure-completion event, override audit event) as a candidate schema for cross-jurisdictional AI deployer-duty evidence standards.
• ISO/IEC 42005 AI incident management, contribute detection pattern examples for rubber-stamp HITL, disclosure suppression, and shadow-AI-in-process from production telemetry; target at least one detection pattern per cycle.
• CSA AI Safety Initiative, contribute anonymized detection signatures for workflow-specific AI risks (decision-distribution manipulation, HITL bypass, disclosure suppression).
• Sector ISACs (FS-ISAC, H-ISAC, IT-ISAC AI working groups), share anonymized, generalized detection signatures; target ≥12 signatures per year; signatures implementable by partner organizations without significant adaptation.

Target: ≥2 telemetry-standard contributions per year and ≥12 ISAC detection signatures per year; all contributions anonymized, legally vetted, and maintained.

Outcome Metrics (L3)

Process Metrics (leading)

• Detection CI/CD pipeline health monitored; failed detection-deployment builds are on-call paged the same as production workflow failures.
• Anomaly model drift monitored (model performance against labeled incident records); retraining SLO met monthly.
• Contribution pipeline active, ≥2 telemetry-standard items in-flight (draft, in-review, or submitted) at any time.
• ISAC submission cadence, at least one anonymized detection signature submitted per month.

Effectiveness Metrics (business value)

• True-positive incidents surfaced first by behavioral anomaly detection (vs. rule-based or external notification), trending up quarter-over-quarter.
• Detection deployment lead time (merge to production) measured in hours, not weeks.
• Industry adoption of contributed workflow monitoring schemas cited by OECD AI, ISO/IEC 42005, CSA, or sector ISAC publications.

Success Criteria

• ≥90% of the detection set expressed as version-controlled, CI/CD-deployed artifacts; detection changes reviewed and deployed through the same change pipeline as workflow configuration.
• Detection coverage auto-verified for 100% of new or re-tiered SM-Processes inventory entries within 24 hours.
• ≥90% of Critical/High-tier workflows with behavioral anomaly detection active; anomaly models retrained monthly on schedule.
• ≥2 telemetry-standard contributions per year to OECD AI or equivalent; ≥12 anonymized detection signatures per year to sector ISACs.

Key Success Indicators

Level 1: - Per-archetype logging baseline published and instrumented for ≥90% of production AI/HAI-embedded workflows, covering decision events, HITL review events, disclosure events, override events, admin-audit events, and identity events per archetype with the fields specified in Activity A. - ≤12-detection high-signal set live, each tied to a TA-Processes archetype threat and HAI TTP tag, with owner, detection query, SLA, and monthly tuning record; false-positive rate tracked per detection; detection set includes rubber-stamp HITL, reviewer-capacity saturation, override-audit anomaly, disclosure suppression, shadow-AI-in-process, and workflow-config drift. - Retention meets the longest applicable regulatory window (including FCRA 25 months for adverse-action records, FINRA 6 years, HIPAA 6 years, EU AI Act Art. 12 ≥6 months where applicable); export path tested annually; DSAR-capable export ≤72 hours. - Quarterly deployer-duty drill executed and inside the ≤24-hour SLA; gaps routed to IM-Processes.

Level 2: - Tier-calibrated logging depth applied per SM-Processes L2 tier-treatment matrix; Critical-tier workflows retain full event corpora at the longest regulatory window. - SIEM integration live with ≥3 cross-workflow correlation rules; ≥90% of Critical/High-tier workflows with behavioral anomaly baselines and FP rate trending down. - Quarterly detection tuning loop operating from IM-Processes post-incident and ST-Processes finding inputs; ≥1 net detection change per cycle. - ML logging-baseline validation element fresh (≤30 days) for all Critical-tier workflows in PC-Processes compliance evidence bundles.

Level 3: - ≥90% of the detection set expressed as version-controlled, CI/CD-deployed artifacts; detection coverage auto-verified on SM-Processes inventory changes within 24 hours. - ≥90% of Critical/High-tier workflows with behavioral anomaly detection active; anomaly models retrained monthly. - ≥2 telemetry-standard contributions per year to OECD AI or equivalent; ≥12 anonymized detection signatures per year to sector ISACs.

Common Pitfalls

Level 1: - ❌ Logging baseline defined at the archetype level but actual production workflows never audited against it, gaps accumulate inside the SM-Processes inventory without appearing in any backlog. - ❌ HITL review events are logged at the queue-item-close level (item-id and timestamp only) rather than at the review-action level, rubber-stamp detection is architecturally impossible without the reviewer identity, AI suggestion, and time-spent fields. - ❌ Decision events capture the final decision but not the AI output and threshold, Art. 22 contestation evidence cannot show what the AI recommended vs. what the final decision was; individual rights requests cannot be fulfilled. - ❌ Detection set grows without governance because new detections are added at every incident but none are ever retired, the team spends more time triaging false positives than investigating real signals. - ❌ Deployer-duty evidence view exists as a template document but is never populated for specific workflows, the quarterly drill is skipped because "we know the logs exist." - ❌ Retention meets GDPR Art. 30 but not FCRA 25-month adverse-action record requirements for decision pipelines used in credit or employment contexts, evidence requests for those workflows cannot be satisfied.

Level 2: - ❌ Tier-calibrated logging configured at deployment time but not maintained, when a workflow is re-tiered from Medium to Critical, logging depth is not updated; full event corpora are absent for the first Critical-tier incident on that workflow. - ❌ SIEM correlation rules are built once and never validated, a rule that has not fired in 90 days may be broken rather than evidence that no correlatable events occurred. - ❌ Anomaly baselines established at onboarding and never refreshed, natural reviewer population turnover makes the baseline stale; FP rates spike as new reviewers' behavior patterns differ from the original cohort. - ❌ Detection tuning loop exists on paper but IM and ST feedback never actually feeds into the review cycle, the same false-positive rubber-stamp detection remains in the set for years because the quarterly process has no dedicated owner.

Level 3: - ❌ Detection-as-code pipeline deployed but detection tests use synthetic data that does not resemble production workflow log patterns, tests pass in CI and detections fail silently in production. - ❌ Behavioral anomaly models retrained on full reviewer log corpus, including incident-period logs where reviewers were operating under audit pressure, poisoned baseline; the model learns to treat incident-period behavior patterns as normal. - ❌ Contributed telemetry schemas are published as point-in-time artifacts and then diverge from internal practice, external adopters build against v1.0 while the org operates v1.3 internally. - ❌ ISAC detection signatures generalized to the point of uselessness, partner organizations cannot implement them without reconstructing the context removed for anonymization.

Practice Maturity Questions

Level 1: 1. Has a per-archetype logging baseline been published specifying the minimum event schema, fields, retention window, and export path for each AI/HAI process archetype in the SM-Processes inventory (decision pipeline, customer-facing flow, HITL chain, back-office augmentation, approval/review workflow, content-generation workflow, knowledge-management workflow), and has compliance of each production workflow been measured against it within the last quarter? 2. Is a high-signal detection set of ≤12 detections active, each with a named owner, detection query, SLA, archetype tag, and last-tuned date, including rubber-stamp HITL detection, reviewer-capacity saturation, decision-distribution drift (security-intersection only), override-audit anomaly, disclosure-suppression, affected-persons-rights-response SLA breach, shadow-AI-in-process, and workflow-config drift, with false-positive rates tracked per detection and monthly tuning reviews occurring? 3. Has the evidence trail for EU AI Act Art. 12, GDPR Arts. 22 and 30, and ISO/IEC 42001 AIMS been wired to the ML-Processes log store with retention meeting the longest applicable regulation (including FCRA 25 months, FINRA 6 years, HIPAA 6 years where applicable), and has a quarterly deployer-duty drill confirmed that the evidence package for a randomly selected production workflow can be assembled within the ≤24-hour SLA?

Level 2: 1. Is tier-calibrated logging depth applied per the SM-Processes L2 tier-treatment matrix, Critical-tier workflows retaining full decision and HITL event corpora at the longest regulatory window, Low-tier workflows receiving baseline only, and is this calibration automatically updated when a workflow is re-tiered? 2. Is the SIEM ingesting ML-Processes log feeds with ≥3 cross-workflow correlation rules active (covering at minimum multi-workflow rubber-stamp correlation, disclosure-suppression plus decision-outcome-shift, and shadow-AI-in-process plus admin-audit-gap), and is a quarterly detection tuning cycle operating from IM-Processes post-incident and ST-Processes finding inputs? 3. Are ≥90% of Critical/High-tier workflows running behavioral anomaly-detection baselines with reviewer-behavior and decision-distribution profiles refreshed monthly and FP rates tracked and trending down, and is the ML logging-baseline validation element completing inside the ≤30-day staleness threshold for all Critical-tier workflows in PC-Processes compliance evidence bundles?

Level 3: 1. Are ≥90% of detections expressed as version-controlled, CI/CD-deployed code artifacts with automated test coverage against realistic synthetic workflow log data, and is detection coverage auto-verified for 100% of new or re-tiered SM-Processes inventory entries within 24 hours of the inventory change event? 2. Are ≥90% of Critical/High-tier workflows running behavioral anomaly detection on reviewer and decision-distribution corpora, with anomaly models retrained monthly, model versions tracked in the model registry, and anomaly-model alerts feeding the IM-Processes incident backlog through the same detection-to-ticket pipeline as rule-based detections? 3. Has the program contributed ≥2 telemetry-standard artifacts per year to OECD AI, ISO/IEC 42005, or equivalent and ≥12 anonymized detection signatures per year to sector ISACs, with contributions maintained current and external adoption tracked?

Document Version: HAIAMM v3.0 Practice: Monitoring & Logging (ML) Domain: Processes Last Updated: 2026-05-14 Author: Verifhai