Writing a grader

A grader is a function. It takes what the system produced and what you expected, and returns a score. Nothing about that requires a model, and the graders that need no model are the ones to write first: they are fast, they cost nothing, and they return the same answer every time.

export const exactMatch = (output, expected) => ({
  score: output.category === expected.category ? 1 : 0,
  passed: output.category === expected.category,
});

The triage agent returns three things, and each needs a different treatment. The category is drawn from a closed set of five values, so a string comparison settles it. The order number either matches the one in the message or it does not. The customer-facing reply has no single correct wording, so no amount of code can score it — that one waits until lesson 6.

Before any of those, check the shape. A reply that cannot be parsed is a failure regardless of how sensible it reads, and separating shape errors from content errors saves a lot of confusion later. A model that returns prose around its JSON is failing differently from one that returns well-formed JSON with the wrong category.