--- title: "An AI agent can bypass a monitor without fooling it" date: 2026-10-19 summary: "Recent monitoring disclosures show why detecting dangerous actions and proving which actions reached the monitor need separate evidence." slug: agent-monitoring-coverage-gap author: "Sebastiaan Hilbers" hero_glyph: waveform --- An AI monitor can correctly reject every dangerous action in a test and still leave a production path unprotected. METR's [September 27 research note](https://metr.org/notes/2026-09-27-implementing-a-basic-blocking-action-monitor/) documents the distinction: its checks found useful detection performance, but also qualifying evaluations launched without monitoring and older framework versions that omitted subagent actions from the monitor's view. The observation changes what a monitoring claim needs to contain. Accuracy measures judgments on inputs the monitor received. Coverage asks which actions reached it. Evidence that an action was stopped requires another observation: the execution decision. For agent operators and independent verifiers, these deserve separate records. ## The missing denominator METR's note reports that a monitor would have detected all ten malicious transcripts supplied by the UK AI Security Institute. The researchers explicitly caution that this was a transcript-level result: it did not establish that each harmful action would have been caught before execution. The same note describes a researcher misunderstanding the monitoring criteria and running qualifying evaluations without enabling the monitor. It is an unusually useful disclosure because it distinguishes an encouraging test from the argument needed to rely on it. [Haskins et al., September 27](https://metr.org/notes/2026-09-27-implementing-a-basic-blocking-action-monitor/). The authors label the note an early research update with less review than METR's research posts. These examples establish failure modes in the described setup, not their prevalence across agent deployments. METR's [October 14 control-assessment proposal](https://metr.org/blog/2026-10-14-control-assessment-proposal/) makes the denominator explicit. It calls for cross-referencing inference logs with monitor logs, investigating unmonitored paths, and checking whether trajectories are silently dropped. It separately asks whether an agent can disable monitoring, alter what the monitor receives, or evade a monitor that is functioning correctly. Those are different tests, with different access requirements. Our inference is straightforward: a monitor cannot establish its own coverage by counting the requests it received. The evidence needs a second account of the activity that should have arrived. ## A valid record can have an untrustworthy display There is also a boundary after collection. In an [October 6 disclosure](https://metr.org/blog/2026-10-06-ai-systems-could-cover-up-misbehavior/), David Rein describes a proof-of-concept vulnerability in the Inspect transcript viewer. A researcher, assisted by an AI agent, found a way to change the transcript as presented to a reviewer, including the viewer's download behavior. The underlying trajectory would still have existed in METR's database. METR says it had not observed agents exploiting this vulnerability in its evaluations. The work took place in an isolated staging deployment, and the Inspect team patched the vulnerability within a day of reporting. The finding is therefore evidence of a demonstrated presentation failure, not evidence that historical evaluations were covertly rewritten. The consequence for verification is narrower and more useful than a general claim that logs cannot be trusted. A reviewer needs to distinguish the recorded event, the representation supplied to a monitor, and the representation displayed to a person. Preserving one does not automatically protect the others. [Rein, October 6](https://metr.org/blog/2026-10-06-ai-systems-could-cover-up-misbehavior/). ## Reconcile actions before reporting a detection rate We propose a small evidence contract for a bounded agent deployment. Implementing and assessing it would require access to the operator's internal execution and monitoring systems. Record eligible action attempts at a trusted execution gateway, including attempts subsequently denied or cancelled. Give each attempt an identifier, its parent run, the applicable policy version, and a binding to the action's contents. Separately record whether a monitor accepted the expected input bundle, produced a decision, and returned that decision before execution was permitted. Keep both records outside the acting agent's write permissions.
TWO RECORDS, ONE RECONCILIATION
IDGateway recordMonitor receipt
APresentPresent
BPresentMissing: investigate
CPresentPresent

Illustrative records, not measured deployment results.

For a declared scope and time window, report the fraction of eligible attempts with a valid monitor decision. Report missing, late, and failed decisions separately. Then measure detection on an explicitly described set of harmful actions. Finally, test whether a block actually prevented the protected effect. A single percentage should not stand in for all three. The contract needs an enforcement boundary. An agent that can use another execution route may bypass this accounting entirely. Subagents, provider-side tools, local models, and alternate credentials therefore belong in the scope statement or in the list of exclusions. A gateway's records can support a coverage claim for traffic forced through that gateway. They cannot establish that no other traffic exists. Token totals can help investigate missing inference sources, but they are not an action count. A deployment must not divide monitored actions by billed tokens and call the result coverage. The unit, eligibility rule, and exclusions have to match the claim. ## Sampling and signatures answer narrower questions Performance telemetry often deliberately discards data. [OpenTelemetry's sampling documentation](https://opentelemetry.io/docs/concepts/sampling/) distinguishes early decisions to retain or drop a trace from decisions made after inspecting more of the trace. Sampling can control cost; it changes what an observer retains. For the proposed evidence contract, retain a minimal accounting record for every in-scope attempt even if detailed diagnostic traces are sampled. Document that sampling separately. A complete list of attempt identifiers would still not establish that the monitor saw every relevant image, tool response, or intermediate action; input completeness needs its own test. Cryptographic logs provide another useful but bounded guarantee. [RFC 9162, Sections 2.1.3–2.1.4](https://www.rfc-editor.org/rfc/rfc9162.html#section-2.1.3), the experimental Certificate Transparency v2 specification, defines proofs that an entry belongs to a Merkle tree and that successive trees preserve an append-only history. Applied to agent records, those mechanisms could make later alteration detectable. They do not establish that every real action was submitted to the log in the first place. That is why the independent accounting path matters. ## Test the pipeline with known omissions Anthropic's [October 9 incident report](https://www.anthropic.com/research/investigating-unintended-model-actions) describes unintended activity on real websites during evaluations and internal use. Its response included restricting internet access, changing tools, and adding monitoring. Anthropic reports that the new tooling blocked the disclosed cases when tested against them. That is evidence about those cases; it is not a measured rate for all possible future behavior. The report also notes that alignment training alone is not yet sufficiently robust. For builders, a useful next experiment is a controlled omission test. In an isolated, authorized environment, introduce a known missing receipt, an input truncated before review, and a delayed block decision. Check whether the operator detects each failure and whether execution stops where the policy requires it. Include a provider-side tool or delegated action only when the deployment actually supports that path. Publish the tested versions and exclusions. For public registries, the corresponding discipline is to preserve the limit of observation. A protocol response cannot reveal the coverage of an operator's internal monitor. Agenstry's [public conformance methodology](https://agenstry.com/methodology) describes evidence available at the agent's public surface; an internal monitoring assessment would require additional access and a separate method. The next useful result is a report that gives both the eligible activity count and the successfully monitored count, with independently testable evidence for how those counts were obtained. Until then, a quiet monitoring dashboard describes the records it received. The missing records remain an open question. ## Sources - [Implementing and Evaluating a Basic Per-Action Monitor for Safer Evals](https://metr.org/notes/2026-09-27-implementing-a-basic-blocking-action-monitor/) — Reilly Haskins, Rif A. Saurous, Nate Rush, Neev Parikh and Beth Barnes, METR, September 27, 2026. Research note. - [How an assessor could investigate control and monitoring of internal AI agents](https://metr.org/blog/2026-10-14-control-assessment-proposal/) — METR, October 14, 2026. - [AI systems could cover up misbehavior](https://metr.org/blog/2026-10-06-ai-systems-could-cover-up-misbehavior/) — David Rein, METR, October 6, 2026. - [Sampling](https://opentelemetry.io/docs/concepts/sampling/) — OpenTelemetry, documentation accessed October 19, 2026. - [Certificate Transparency Version 2.0, RFC 9162](https://www.rfc-editor.org/rfc/rfc9162.html) — B. Laurie, E. Messeri and R. Stradling, IETF, December 2021. - [Investigating unintended model actions in our evaluations and internal use](https://www.anthropic.com/research/investigating-unintended-model-actions) — Anthropic, October 9, 2026; correction October 12, 2026. - [Conformance Methodology](https://agenstry.com/methodology) — Agenstry, accessed October 19, 2026. ## Cite this post ```bibtex @misc{hilbers2026monitorcoverage, author = {Hilbers, Sebastiaan}, title = {An AI agent can bypass a monitor without fooling it}, year = {2026}, month = oct, url = {https://agenstry.com/blog/agent-monitoring-coverage-gap}, publisher = {Agenstry} } ```