Lesson 05 · Human accountability · 14 min

Oversight that stays meaningful

"Human in the loop" is the most abused phrase in agent governance, because a loop with a human who clicks approve in two seconds is automation with extra latency. The framework's version is more demanding: put checkpoints where they count, make each approval evaluable, and then audit the oversight itself.

Where to put checkpointsHigh-stakes decisions, irreversible actions — deletions, external communications, payments — outlier behaviour outside the agent's normal scope, and thresholds users set for themselves. Not everywhere: a checkpoint on everything is a checkpoint on nothing, because attention is the scarce resource.

What an approvable request looks likeShort, contextual, honest about risk: the action, its plain-language effect, whether it can be undone. For complex work, approving a plan before execution beats approving a wall of diffs after. For the riskiest actions, require a written justification — friction there is a feature.

The checkpoint is the visible half. The audit underneath — override rates, review times — is what tells you whether the human is still reviewing or has become a very slow rubber stamp.

Then the uncomfortable half: automation bias. A system that is right most of the time trains its reviewers to stop reviewing — the framework is specific about the failure and the fix. Track the override rate: near zero for weeks is not evidence of a perfect agent, it is the signature of rubber-stamping. Track decision time: approvals that get steadily faster mean attention is decaying. And train reviewers on the agent's actual failure modes, because a reviewer who has never seen the agent be confidently wrong has no reference for what wrong looks like.

Complement people with machines: alerts for unauthorised-access attempts and repeated failures, anomaly detection on trajectories, and a deny-by-default stance when the approval path is down.
Deny-by-default is the detail that dates a design: if the approver is unreachable and the agent proceeds anyway, the checkpoint was decorative.
One coding-assistant case study in the framework maps approvals to action risk — reads free, edits approved per session, shell commands approved per project, suspicious commands re-approved even when previously allowed. Approval effort tracks risk, not habit.

In the lab, you are the checkpoint. The agent proposes a batch of actions for four tickets, each with a plain-language explanation. One proposal is subtly wrong. Every decision you make is timed. The audit at the end shows your override rate, your median decision time, and whether the flaw got through — the same three numbers the framework says to collect about real reviewers.

Checkpoint

If you approved the planted flaw: what information was missing from the request that would have caught it? If you caught it: how long did that batch take, and would you sustain that pace for forty batches a day?