Curriculum
Eleven lessons, about 2.1 hours in total. The order follows how the work is done in practice: look at output first, name what goes wrong, and write the measurement afterwards.
Foundations
- 01What an evaluation isThe vocabulary, and why a single test run tells you less than it appears to.8 min
- 02Building a datasetWhere goldens come from, how many you need, and what makes a set worth measuring against.10 min
- 03Writing a graderScoring output with ordinary code, and knowing which parts of an answer that works for.12 min
- 04Reading failuresWhat to do with a score once you have one, and why the first move is not to change the prompt.14 min
- 05Grouping failures into modesTurning a pile of individual notes into a small set of named, countable problems.12 min
Judgement
- 06Writing a judge rubricScoring open-ended output by asking a model, and writing the instructions that make it reliable.14 min
- 07Checking your judgeMeasuring whether the judge agrees with you, before you trust anything it says.12 min
- 08Comparing modelsRunning one evaluation across many models, and reading the table that comes back.12 min
Production
- 09Evaluating tool useScoring what an agent did, not only what it said.12 min
- 10Catching regressionsTurning an evaluation into something that runs on every change and tells you when a score drops.10 min
- 11Deploying with confidenceShipping the agent, and keeping the evaluation useful once real traffic arrives.10 min