AssayAssay
04 / 09

Lesson 04 · Human accountability · 12 min

Who answers for the agent

When an agent misfires in production, the first question is not technical: it is whose job this was. If that question has no crisp answer, the organisation discovers it during the incident, which is the most expensive possible time. The framework's second dimension is making the answer crisp in advance.

The agentic value chainModel & toolmakers, MCP hostsPlatformprovidersApp / systemdevelopersDeployeryour organisationEnd usersinteract · integrateaccountability concentrates hereone organisation often plays several of these roles at once
Accountability does not stop at the organisation that deploys — but it concentrates there. Contracts and transparency requirements are how a deployer manages the links it cannot see inside.

The value chain runs from model and tooling makers through platform providers and app developers to the deployer and, finally, end users. Two things are true at once: accountability is distributed along the chain, and it concentrates at the deployer — the organisation that chose to put the agent in front of its work. Deploying on a vendor's platform outsources the software; it does not outsource the answerability.

Inside the organisation, the framework sketches four owners, and the sketch survives contact with most org charts. Decision makers choose which use cases run at which autonomy and set the risk appetite. The product or AI team designs the agent, its tools, its rollout and its user education. The security team owns threat modelling, trust boundaries and monitoring. Users own what they approve and how they use the output. The exercise that matters is not memorising the list — it is being unable to name a failure that lacks an owner.

Third-party opacityA vendor agent you cannot observe is a risk you are accepting on faith. The framework's remedies are contractual and technical: demand disclosure of capabilities and data handling, scoped keys and per-agent identity, tool-call logs you can read. Where a vendor cannot provide them, shrink the use case until the opacity is tolerable — or decline it.

Accountability diffuses in exactly the places where several parties each did something reasonable. The recruitment and report-drafting case studies in the framework both draw the same line: the checkpoint that makes a decision consequential — the shortlist, the final draft — stays with a named human role, whatever the agents around it do.

The lab replays an incident — the firewall change from lesson 3, six contributing failures — and asks you to assign each one to the role best placed to have prevented it. Then a model applies the same role definitions. Where you and it disagree is usually where two roles are both a little bit responsible, which in production means neither is.

Checkpoint

For one agent you know of, name the human who would stand in front of leadership after an incident. If it took more than ten seconds, that is the finding.

The rubric — you and the model both apply this

An incident: Relay, the IT-helpdesk agent, read a knowledge-base article that had been tampered with via a synced partner portal. The article contained an embedded instruction; the agent followed it and staged an allow-all firewall rule. The approving engineer clicked approve in four seconds without reading. The change ran for six hours before monitoring flagged unusual traffic. Assign each contributing failure to the single role best placed to have prevented it: key decision makers set risk appetite and decide which use cases run at which autonomy; the product / AI team designs the agent, its prompts, tools and rollout; the cybersecurity team owns threat modelling, input trust boundaries and monitoring; the platform vendor owns the integrity of what their sync writes into your systems; the end user owns reviewing what they approve.

Assign all six failures, then compare. The interesting rows are the ones where the model's assignment is defensible and different from yours — say out loud why yours is better, and notice whether you can.

Who owned preventing this failure? Decide for all 6 before running — going first is the point.

Untrusted content reached the agent

The KB accepted writes from an external sync with no integrity check, so tampered content flowed straight into the agent's context.

external write pathno signing or reviewknown injection vector

The agent could stage an allow-all rule

update_firewall_rule accepted any rule text, including one that opened everything to everything.

no input validationtool accepts arbitrary scope

The approval was a rubber stamp

The engineer approved the staged change in four seconds, without reading the rule.

4-second reviewno justification recorded

Nobody was watching the approvers

Override rates and review times were not tracked, so months of reflexive approvals went unnoticed.

no oversight metricsautomation bias unmeasured

Detection took six hours

The allow-all rule ran for six hours before traffic monitoring noticed.

no alert on rule scopetraffic anomaly caught it eventually

Firewall changes ran at too much autonomy

Leadership had approved firewall changes as a lightly-governed use case to hit an automation target.

risk appetite set top-downautomation KPI pressure