AssayAssay
08 / 08

Lesson 08 · Practice · 14 min

Designing a harness

Everything in this course converges on one exercise: take an agent, decide what must govern it, and write the governance down in the order it should be built. This lesson is that exercise, against the support agent every lab on the site has been using.

Work through it in the order the harness lesson gave, and for each part name which target it serves.

  • Maintainability sensors, all code. The reply parses; it has a category, an urgency and a reply; it contains no promise, date or amount. Four checks, zero model calls.
  • Guides for what those keep catching. If the promise gate fires often, the system prompt gets a line — and the gate stays, because the prompt is a request.
  • Architecture. issue_refund requires read_policy. lookup_order must precede any refund discussion. No tool called twice with identical arguments. Three trace assertions and one precondition.
  • One behaviour sensor where it matters most. A checker with a rubric, on replies that mention money — not on status lookups. Validated against twenty of your own labels first.
  • Outcome checks on the cases you know. The order that is still in transit must not be refunded. The fourth contact must be escalated. Assert them.
  • The dataset. Every failure any of the above catches becomes a case. Start with the fifty tickets; add to them weekly.

Then the question that decides whether it is a harness or a pile of checks: who owns it, where does it live, and what runs when it changes. If the answer to any of those is nobody, nowhere, or nothing, the work above will decay within a quarter.

target          guides                     sensors
maintainability schema in the prompt       schema_valid · promise gate · length
architecture    refund precondition        order(lookup, refund) · no repeat calls
behaviour       skill: refund procedure    rubric checker on money replies · outcome cases

owner: the support-agent team     lives: harness/ in the repo     runs: on every change, nightly on prod sample

Notice how much of that table already exists on this site. The graders are the evals course. The precondition is the agent-loops course. The route assertions are the industry blueprints. The checker is loop engineering. The dataset is the fifty tickets. A harness is not a new thing to build; it is the name for assembling what you already have, with an owner.

Checkpoint

Write the full harness table for the support agent: every check, its target, and what it was added to catch.

Order support at volume

High-volume order support, where the engineering problem is cost per resolution rather than difficulty. Most contacts are simple; a few are not, and the two must be handled differently.

loop — The tasks are individually simple and highly varied, so the flexibility of a loop is worth more than the guarantees of a graph — with a hard stop before anything touching money.

“Where is order 41902?”

Good handling: One lookup, then an answer citing the carrier and expected date. One tool call is the whole job.

The retail blueprint is a partial harness for this agent. Read its constraints, classify each by target, and list what is missing from the table in the lesson.

Rules this agent must obey

Looked the order up

An invented delivery date is a promise the business then has to keep.

Resolved within 5 iterations

At this volume, cost per resolution is the constraint that decides whether the system is viable.

Promised nothing unsupported

Support language becomes a commitment the operation is measured against.

What goes wrong here

  • Optimising quality until the cost per contact exceeds the human cost it replaced.
  • Inventing delivery dates, which converts a support answer into an operational promise.
  • Handling the difficult 5% with the same path as the easy 95%.