Course 01 of 10
Evaluating AI agents
Everything else on this platform assumes you can tell whether a change made things better. This course builds that ability from nothing: a dataset of your own, graders that need no model, error analysis on real failures, and a judge you have checked against your own labels.
11 lessons·about 2.1 hours
Start lesson 01Foundations
- 01What an evaluation isThe vocabulary, and why a single test run tells you less than it appears to.8 min
- 02Building a datasetWhere goldens come from, how many you need, and what makes a set worth measuring against.10 min
- 03Writing a graderScoring output with ordinary code, and knowing which parts of an answer that works for.12 min
- 04Reading failuresWhat to do with a score once you have one, and why the first move is not to change the prompt.14 min
- 05Grouping failures into modesTurning a pile of individual notes into a small set of named, countable problems.12 min
Judgement
- 06Writing a judge rubricScoring open-ended output by asking a model, and writing the instructions that make it reliable.14 min
- 07Checking your judgeMeasuring whether the judge agrees with you, before you trust anything it says.12 min
- 08Comparing modelsRunning one evaluation across many models, and reading the table that comes back.12 min
Production
- 09Evaluating tool useScoring what an agent did, not only what it said.12 min
- 10Catching regressionsTurning an evaluation into something that runs on every change and tells you when a score drops.10 min
- 11Deploying with confidenceShipping the agent, and keeping the evaluation useful once real traffic arrives.10 min