Watching it run

Pre-deployment testing establishes a baseline; it cannot anticipate an environment that changes under the agent daily. Monitoring is the other half of the release, and the framework's guidance compresses to three questions: what do you log, what fires an alert, and what happens when one fires.

Log with intent. Record decisions and actions across layers — the user-agent conversation, the tool invocations, the model's stated reasoning — prioritised by risk: the database write and the payment matter more than the status lookup. Keep the problem traces immutable; the trajectory that embarrasses someone is the one most worth preserving, and the one most likely to quietly disappear.

• Programmatic thresholds: unauthorised-access attempts, repeated identical calls, call volume, empty answers. Deterministic, instant, cheap enough to run on everything.
• Anomaly detection: statistical or learned models over trajectories, for the failures nobody wrote a rule for yet.
• Agents monitoring agents: a second model reading trajectories for semantic trouble — actions inconsistent with the request, over-broad scopes, dishonest summaries.
• Interventions, decided in advance per alert type: log it, flag it for review, pause the agent until a human looks, or kill it. An alert with no owner and no intervention is a log line with self-esteem.

The layers are ordered by what they can see and how fast they see it. Rules fire in microseconds but only on patterns you predicted. The model reviewer reads intent but is itself probabilistic — its misses and false alarms need auditing like any other sensor, which is the same lesson the judge lessons in the evals course taught, wearing a pager.

The lab puts both detector layers over one trace. Replay the canned incident — the injected afternoon from lesson 3, preserved — and watch six threshold rules and a model reviewer read the same steps. Then run the offboarding ticket live and ungated: if the model behaves, the rules stay quiet and the reviewer earns its keep; if it does not, you get to watch the rules catch it in real time.