AssayAssay
02 / 06

Lesson 02 · The stack · 14 min

The verification loop

Loop two is small: something produces work, something else grades it, and a failure comes back as a critique for the next attempt. The mechanism takes an afternoon. Getting value from it depends on details that look cosmetic and are not.

The first is that the checker must be a separate call. A model reviewing its own draft in the same conversation is anchored on the reasoning that produced it — it reliably catches omissions and reliably misses its own misconceptions. Fresh context is what makes the check worth running.

The second is that a different model is usually better than the same one twice. Different training produces different blind spots, and the whole point of a checker is to have blind spots that do not overlap with the maker's.

The third is what comes back. A verdict alone gives the maker nothing to act on: told only that it failed, a model will usually produce a near-identical draft. The critique has to say what to change.

The verification loopmakerdraftscheckergrades vs rubricacceptedshipsdraftpassfail · with a critique saying what to changenothing checked it if this loop is off
The checker is a separate call, ideally a different model. A verdict alone is not enough — the critique is what the next attempt has to work from.
attempt 1  "...a refund will be with you shortly."
  checker   FAIL - includes a timeframe, which the rubric forbids
  fed back  remove the timeframe; say someone will be in touch

attempt 2  "...someone will be in touch to discuss next steps."
  checker   PASS

That trace is a real run from this lab. The maker did not see its own rejected draft on the second attempt — only the critique. Rewriting from the note rather than editing the failed text avoids anchoring on the wording that failed.

One failure mode is worth designing against explicitly: a checker whose output you cannot parse. It is tempting to treat that as a failure and retry. Do not. An unreadable checker has not checked anything, and feeding an empty critique forward produces a loop that burns its whole budget rewriting the same draft. Stop instead, and say which model to replace.

Checkpoint

Show a case where a different checker model catches something the same model missed.

maker
checker

“My order 88213 arrived with a cracked base. I'd like a refund please.”

The natural, friendly reply promises the refund — which the agent cannot authorise.

Set the maker and checker to the same model, run it, then set them to different models and run again. Watch whether the checker catches things when it shares the maker's blind spots.

Run once with the verification loop off, then again with it on, and compare the replies.