Monitoring & Logging (ML)

Vendors Domain - HAIAMM v3.0

Practice Overview

Objective: Provide the detection and evidence foundation for the Vendor AI Assurance program, the logs that prove EU AI Act deployer duties and GDPR processor obligations, and the detections that surface shadow AI, data-egress anomalies, agent-tool abuse, and vendor-side behavior changes.

Description: ML-Vendors captures the signals produced by sanctioned AI vendor integrations (via the reference pattern's proxy, admin-audit feeds, endpoint/egress telemetry, SaaS admin logs) and turns them into a small, high-signal detection set at L1. It also owns the evidence trail, the log retention and exportability that proves human oversight, logging, and disclosure obligations were met.

Context: AI-vendor logging is famously patchy, vendor-side logs are often short-retention and not export-friendly, and org-side logs miss AI-specific events unless someone has explicitly configured them. L1 ML-Vendors installs a minimum logging baseline and a small set of high-value detections, rather than a sprawling detection catalog that nobody tunes.

Maturity Level 1

Objective: Establish the AI-vendor logging baseline (per archetype), operate a small high-signal detection set including shadow-AI detections, and ensure evidence retention meets deployer-duty requirements

At this level, the program has visibility into what sanctioned AI vendors are doing and what shadow AI is attempting, enough signal to feed IM and prove deployer duties, without chasing detection coverage for its own sake.

Dependencies

• SM-Vendors L1 (required): inventory defines what to instrument.
• SA-Vendors L1 (required): the reference pattern defines where logs come from (proxy, admin audit, SSO, egress).
• EH-Vendors L1 (required): the hardened controls are the source of most shadow-AI signal.
• TA-Vendors L1 (required): archetype threats prioritize which detections to build first.
• Supports / unblocks: IM-Vendors L1 (detections become issues/incidents), ST-Vendors L1 (shadow-AI tests validate detections), PC-Vendors L1 (evidence trail is the compliance deliverable).

Desired Outcomes

• Every sanctioned AI vendor integration produces the logs required to prove human oversight, logging, and disclosure obligations.
• A small, high-signal detection set is active, targeted at the threats most likely to materialize, not at comprehensive coverage.
• Shadow AI detections exist as a named set, validated quarterly by the ST-Vendors discovery exercise.
• Evidence retention meets or exceeds the longest applicable compliance requirement; exports are demonstrable on request.
• Detection tuning is a practice, not a project, false positives are actively suppressed, new detections are added from incident lessons.

Activities

A) Establish the AI-vendor logging baseline per archetype

Minimum events and fields per archetype (L1): - Common across archetypes, user identity (SSO subject), timestamp, vendor, integration identifier, action class (access, data-flow, config-change, admin-audit), data-class tag where applicable, trace/correlation ID. - Consumer GenAI, org-tenant session events, admin-audit events, content-filter actions. - AI-embedded SaaS, AI-feature activation/deactivation, per-workspace usage, admin-audit for AI-feature toggles, parent-vendor audit feed ingested. - AI coding assistant, IDE policy-match events, prohibited-path/data-marker blocks, license-usage attribution. - AI API / model, prompt/response logging at the internal proxy (with PII scrubbing per SR), model-version, region, latency, error; admin-level key-use attribution. - AI agent / automation platform, session start/end, tool-call events (tool, parameters, outcome), HITL gate invocations, permission-denials, session outcome.

Baseline retention: meets or exceeds the longest applicable requirement (e.g., EU AI Act high-risk logs, GDPR records-of-processing), per data-class. Export path is tested at least annually.

B) Operate a small high-signal detection set

L1 target: ≤12 detections, each tied to a TA archetype threat, each with owner, query, SLA, and last-tuned date.

Core detections: - Shadow-AI egress, traffic to unsanctioned AI vendor domains from managed endpoints/networks. - Shadow-AI SaaS sign-in, SSO or IdP activity against an AI SaaS that is not in the sanctioned catalog. - Consumer AI personal-account sign-in from org endpoints, domain/email pattern anomaly. - Bulk content paste/upload to AI vendor domains, DLP rule-set match (volume and data-class). - API-proxy anomalies, prompt/response volume spikes, model-version unexpected change, PII scrubbing failures. - Agent tool-call violations, calls outside allowlist or outside scoped parameters. - Agent HITL bypass attempt, HITL gate invoked and declined, followed by retry patterns. - Parent-SaaS AI-feature toggle change, unexpected enablement of an AI feature in an approved parent SaaS. - Admin-key anomaly, AI-vendor admin-key use outside known pattern (geo, time, volume). - No-train setting change, vendor admin-audit event where the training-toggle state changes. - AI-vendor breach / advisory, external-intel feed match against inventory. - Egress to new AI vendor domain first-seen, discovery signal for SM inventory.

Each detection: false-positive rate tracked; monthly tuning review.

C) Prove the evidence trail for deployer duties and processor obligations

• A single "deployer-duty evidence view" per high-risk AI vendor integration, pulls: intake approval, REM, DR decision, IR config records, ST pass records, ML logs, incident records, AUP coverage for associated users.
• GDPR Art. 30 records-of-processing entries reference ML retention evidence where relevant.
• ISO/IEC 42001 AIMS evidence assets linked (or identified as gaps) from ML storage.
• At least one quarterly drill: pull deployer-duty evidence for a random high-risk AI vendor within 2 business days.

Outcome Metrics (L1)

Process Metrics (leading)

• Detection tuning cadence, monthly review per detection; stale detections retired.
• Retention / export test cadence, at least annual, with evidence.
• Archetype-baseline gap list maintained, every integration scored vs. baseline; gaps on the backlog.

Effectiveness Metrics (business value)

• Incidents detected internally before external notification, trend over quarters.
• Regulator / auditor / customer evidence requests turned around inside published SLA.
• Detection-to-prevention conversion, detections that led to a preemptive control change (SA/SR/EH update), not only an incident.

Success Criteria

• Per-archetype logging baseline published and operated.
• ≤12-detection high-signal set live, each with owner and tuning record.
• Deployer-duty evidence view produced for every high-risk AI vendor integration.
• Quarterly deployer-duty drill executed and inside target SLA.
• Retention and export path tested at least annually.

Maturity Level 2

Objective: Add anomaly detection on AI-vendor behavior, correlate across vendors, and automate deployer-duty evidence generation

At this level, monitoring moves from rule-based detections to anomaly detection with a baseline of normal AI-vendor behavior. Multi-vendor integration chains are analyzed as a graph. Deployer-duty evidence (for EU AI Act Art. 26, GDPR Art. 30) is assembled automatically from logs.

Dependencies

• ML-Vendors L1 (required): per-archetype logging baseline and high-signal detection set.
• SM-Vendors L2 (required): tiers drive detection priority.
• EH-Vendors L2 (required): hardened control surfaces produce reliable signal.

Desired Outcomes

• Anomaly-based detections catch novel behavior that rule-based detections miss.
• Cross-vendor chains (e.g., coding assistant → code review AI → deploy pipeline) analyzed as a graph; anomalies on the graph surface.
• Deployer-duty evidence for Critical-tier integrations is machine-generated, not manually assembled.

Activities

A) Anomaly detection on AI-vendor behavior

• Baseline normal per-integration behavior: prompt volume, tool-call patterns, egress volume, time-of-day patterns.
• Detect anomalies; tune for false-positive rate.
• Escalate to IM with context (baseline vs. observed).

B) Cross-vendor correlation

• Multi-vendor integration chains mapped as a graph.
• Graph anomalies: unexpected edges, changing centralities, new intermediate vendors.
• Shadow-AI detection graph signals: unexpected AI-vendor egress from previously non-AI endpoints.

C) Automated deployer-duty evidence

• For Critical-tier integrations, the evidence view (logs, human-oversight assignments, disclosures, Art. 26 checklist attestations) auto-assembles on a schedule.
• Regulator inquiry turnaround SLA ≤3 business days.

Outcome Metrics (L2)

Process Metrics (leading)

• Baseline refresh cadence, monthly per integration.
• Graph refresh, weekly.
• Evidence-generation pipeline health monitored.

Effectiveness Metrics (business value)

• Incidents detected internally before external notification, trending up.
• Regulator / customer evidence turnaround visibly faster.

Success Criteria

• ≥90% Critical with anomaly baselines.
• Cross-vendor graph active.
• Automated deployer-duty evidence for 100% Critical.
• Regulator inquiry ≤3 business days.

Maturity Level 3

Objective: Real-time AI-vendor attestation; contribute to industry AI-vendor telemetry standards; share anonymized detection signatures

At this level, attestation is continuous. The regulator / customer / auditor can query a live posture for any Critical-tier AI-vendor integration. The program contributes to AI-vendor telemetry standards (OpenTelemetry AI workgroup, CSA AI Safety Initiative) and shares anonymized detection signatures with ISACs.

Dependencies

• ML-Vendors L2 (required): anomaly detection, cross-vendor graph, deployer-duty evidence.
• PC-Vendors L3 (required): compliance-evidence automation substrate.

Desired Outcomes

• Live deployer-duty posture for any Critical AI vendor is queryable.
• Program contributes to AI-vendor telemetry standards that reduce fragmentation.
• Industry benefits from shared detection signatures.

Activities

A) Real-time attestation

• Live-queryable evidence view for Critical integrations; cryptographic signing where applicable.

B) Contribute telemetry standards

• OpenTelemetry AI workgroup, CSA AI Safety Initiative, OpenSSF AI telemetry efforts.
• Schemas, semantic conventions, required fields.

C) Shared detection signatures

• Anonymized detection signatures contributed to ISAC AI-vendor feeds, OpenSSF AI.
• Target ≥12 signatures/year.

Outcome Metrics (L3)

Process Metrics (leading)

• Live-attestation pipeline health, % Critical integrations producing a fresh attestation artifact within the last 1 hour.
• Telemetry-standard contribution pipeline, at least one artifact (schema, semantic convention, field spec) in-draft, in-review, or published at any time.
• Detection-signature release cadence, at least one anonymized signature submitted to ISAC AI-vendor feeds per month.
• Cryptographic signing coverage, % of Critical attestation artifacts machine-signed and verifiable by regulators or auditors.

Effectiveness Metrics (business value)

• Regulator or auditor evidence requests resolved from the live attestation view within 1 business day, no manual log assembly.
• Industry adoption of contributed telemetry schemas reduces cross-org log-format fragmentation and lowers mutual-aid investigation cost.
• ISAC partners cite shared detection signatures in their own detection catalogs, externally recognized contribution.
• ML-reviewer hours per integration trending toward zero for Critical tier as continuous attestation replaces periodic log reviews.

Success Criteria

• ≥90% Critical with live attestation.
• ≥2 telemetry-standard contributions/year.
• ≥12 shared detection signatures/year.

Key Success Indicators

Level 1: - Per-archetype logging baseline published and instrumented: prompt/completion events at the proxy, tool-call events, admin-audit events, identity events, with retention meeting the longest applicable compliance requirement and export path tested annually. - High-signal detection set of ≤12 detections active, each tied to a TA archetype threat, with owner, query, SLA, and monthly tuning record; false-positive rate tracked per detection. - Deployer-duty evidence view produced for every high-risk AI vendor integration; quarterly drill shows evidence assembled within 2 business days. - Shadow-AI detections cover egress to unsanctioned AI vendor domains, SSO/IdP activity against unlisted AI SaaS, bulk content uploads to AI vendor domains, and no-train toggle changes.

Level 2: - ≥90% of Critical-tier AI vendor integrations have anomaly-detection baselines established from normal behavior; FP rate actively tuned downward. - Cross-vendor integration graph refreshed weekly; anomalous edges, changing centralities, and unexpected intermediate vendors surface automatically. - Automated deployer-duty evidence assembled for 100% of Critical-tier integrations; regulator inquiry turnaround ≤3 business days. - EU AI Act Art. 26 and GDPR Art. 30 obligations traceable to machine-generated log evidence, no manual compilation required for Critical tier.

Level 3: - ≥90% of Critical-tier integrations expose a live-queryable deployer-duty posture, including logs, human-oversight assignments, disclosures, and compliance attestations. - ≥2 telemetry-standard contributions per year to OpenTelemetry AI workgroup, CSA AI Safety Initiative, or equivalent, schemas, semantic conventions, or required fields. - ≥12 anonymized detection signatures per year contributed to ISAC AI-vendor feeds or OpenSSF AI, with external adoption tracked.

Common Pitfalls

Level 1: - ❌ Logging baseline defined per archetype but actual integrations never audited against it, gaps accumulate without appearing in any backlog. - ❌ Prompt/completion logging configured at the application tier but not at the proxy, PII scrubbing is bypassed and retention policy is unenforceable. - ❌ Detection set grows without bounds because no-one retires stale detections; the team spends more time on false positives than on real signals. - ❌ Deployer-duty evidence view exists as a document template but is never populated for specific integrations, the quarterly drill is skipped. - ❌ Shadow-AI detections cover egress domains but miss the SaaS-admin-audit path, AI features silently enabled in approved parent SaaS go undetected for months.

Level 2: - ❌ Anomaly baselines established at onboarding and never refreshed, behavioral drift in normal usage makes the baseline stale and FP rates spike. - ❌ Cross-vendor graph built but not acted on, anomalous edges identified and never routed to IM; the graph is a visualization, not a detection surface. - ❌ Automated deployer-duty evidence assembles the right log citations but the underlying logs don't actually meet the retention window, evidence is incomplete for historical queries. - ❌ Regulator inquiry turnaround target is set but never drilled, the first real regulatory request reveals the automation doesn't work end-to-end.

Level 3: - ❌ Live attestation view exists for Critical integrations but the underlying log pipeline has gaps, the green dashboard is not representative of actual log completeness. - ❌ Telemetry-standard contributions made but not maintained, schemas published as point-in-time artifacts diverge from internal practice within a year. - ❌ Detection signatures shared with ISACs are generalized to the point of uselessness, partner organizations cannot implement them without significant adaptation. - ❌ Cryptographic signing of attestation artifacts planned but not deployed, regulators cannot independently verify the attestation chain.

Practice Maturity Questions

Level 1: 1. Is a per-archetype logging baseline published specifying the minimum events, fields, retention period, and export path for each of the five AI vendor archetypes, and has compliance of each active integration been measured against it within the last quarter? 2. Is a high-signal detection set of ≤12 detections active, each with a named owner, alert query, SLA, and last-tuned date, including shadow-AI egress, no-train toggle change, agent tool-call violations, and bulk data upload, with false-positive rates tracked and monthly tuning reviews happening? 3. Has a deployer-duty evidence view been produced for every high-risk AI vendor integration, and has a quarterly drill confirmed it can be assembled within 2 business days on request?

Level 2: 1. Are ≥90% of Critical-tier AI vendor integrations running anomaly detection with established behavioral baselines, a tuned FP rate trending downward, and escalation to IM with baseline-vs.-observed context when anomalies fire? 2. Is the cross-vendor integration graph refreshed at least weekly, with anomalous edges and unexpected vendor intermediaries automatically surfacing as detections, not requiring manual graph inspection? 3. Is automated deployer-duty evidence assembled for 100% of Critical-tier integrations, covering EU AI Act Art. 26 and GDPR Art. 30 obligations, and can the program turn around a regulator inquiry within 3 business days?

Level 3: 1. Are ≥90% of Critical-tier AI vendor integrations covered by a live-queryable deployer-duty posture, logs, oversight assignments, disclosure records, compliance attestations, that a regulator or auditor can query directly without manual log assembly? 2. Has the program contributed ≥2 telemetry-standard artifacts per year (schemas, semantic conventions, required-field definitions) to OpenTelemetry AI workgroup, CSA AI Safety Initiative, or an equivalent industry body, with adoption tracked? 3. Has the program contributed ≥12 anonymized detection signatures per year to ISAC AI-vendor feeds or OpenSSF AI, with at least one ISAC partner citing adoption?

Document Version: HAIAMM v3.0 Practice: Monitoring & Logging (ML) Domain: Vendors Last Updated: 2026-05-12 Author: Verifhai