AssayAssay
09 / 09

Lesson 09 · Defence in depth · 14 min

Adversarial testing and incident response

Security you cannot measure is a hope. The final discipline is adversarial testing — deliberately attacking your own agent, repeatedly, as a standing part of the build rather than a one-time audit before launch. An agent's defences age the moment the model, the prompt, or a tool changes, so a penetration test from last quarter tells you about an agent that no longer exists.

The scalable version is a maintained battery of injection techniques — override, authority, obfuscation, cross-tenant exfil — run automatically and escalated each round, with a model attacker layered on to mutate them into fresh variants. There is a catch worth knowing: aligned models often refuse to play attacker, returning a polite refusal instead of payloads. That is a genuine property, not a bug, and it is why the durable core is a battery you maintain rather than a model you hope will cooperate. Either way it runs on every commit, and that cadence catches the regressions a quarterly audit sleeps through — wire it into the release gate from the operations course and every change faces it automatically.

When an attack does land — and over enough changes, one will — the trace is the start of incident response. The questions are always the same, and they are answerable only if you logged enough to answer them: what did the agent reach, what actually left the system, which control should have stopped it, and what is the blast radius given the credential it held. This is why the monitoring and immutable-logging from the operations course are security infrastructure, not just reliability infrastructure.

  • Contain: revoke the agent's credential and disable the affected tool or path before investigating — a scoped credential makes this a narrow action.
  • Investigate: replay the trace to find the entry point and the reached systems; the immutable log is the evidence chain.
  • Fix the class, not the instance: the landed payload is one of a family. Add the structural control that defeats the family, then add the payload to the red-team battery so a regression is caught.
  • The blast radius you bounded in lesson 7 is what makes response survivable: it is the difference between rotating one scoped key and notifying every customer.

The healthy end state is that security stops being a phase and becomes a fixture: injection payloads in the eval set, an egress allowlist in the config, a red-team loop in the release gate, and a rehearsed response for when one gets through. The attacker only has to be right once; this is how you make once survivable.

The lab is that red-team loop: each round runs a different family of injection technique against the defence you choose, with a model attacker mutating them into fresh variants when it will cooperate — and telling you when it declines. Run it under a hardened prompt and then structural, and watch the same techniques go from occasionally landing to reaching nothing worth taking.

Checkpoint

Write the first three lines of your agent's incident runbook: how do you revoke its access, find what it reached, and prove what left? If you can't answer the third, you are under-logged.

Defencethe system prompt says don't obey documents, don't leak secrets. Words only.

Run the loop under 'hardened prompt', then 'structural'. Note whether the attacker model cooperated or declined — both are real — and watch structural defence take the same techniques to zero.

Each round runs a different family of injection technique — override and authority, obfuscation, cross-tenant exfil — against the chosen defence. A model attacker mutates each family into fresh variants when it will cooperate; when it refuses, the maintained battery runs. Vertex defends.