Catching regressions

An evaluation you run by hand gets run when you remember, which is usually after something has already gone wrong. Wiring it into your pipeline turns it into a gate: every prompt change, model change or dependency bump gets measured before it ships.

Set thresholds per grader rather than one overall number. A single average hides the case where category accuracy improved four points while order extraction collapsed. Thresholds also want a small tolerance, because model-graded checks vary slightly between runs and a gate that fails on noise gets switched off within a fortnight.

• Record the score for every run, so you can see the trend rather than one number.
• Fail on a drop from the last known good score, not on an absolute value.
• Keep the fast, code-based graders in the pull-request check; save the model-graded ones for merges.
• When the gate fails, print the failing rows — a number alone does not tell anyone what to fix.

The failure modes you named in lesson 5 are useful here too. Tracking counts per mode over time shows whether a change fixed the problem you targeted or simply moved it.