Lesson 01 · Foundations · 8 min

What an evaluation is

An evaluation is a repeatable measurement of how well a system does a task. For ordinary software the measurement is usually a test suite: the same input produces the same output, so a passing test is a fact about the program. Language models complicate this in one specific way — the same input can produce different outputs on different runs.

That single difference is what makes evaluation a separate skill. You are no longer asking whether the program is correct. You are asking how often it is correct, on which kinds of input, and whether that rate is going up or down as you change things.

EvalA dataset of inputs, a set of expected results, and one or more graders that score the system's output against them.

GraderA function that takes what the system produced and what you expected, and returns a score. It may be plain code or another model.

GoldenA single reference example — one input paired with the answer you consider correct.

Below, the same message is sent to a model several times. If the replies differ, that variation is the reason a single spot-check cannot tell you whether a change helped. If they are identical, that is worth knowing too: at temperature 0 many models are close to deterministic, and the variation you care about comes from differences between inputs rather than between runs.

Throughout this course the worked example is a support-ticket triage assistant. It returns a category, an order number, an urgency and a short reply — three kinds of output that need three different kinds of grader.

Checkpoint

Run the same five cases twice. Note whether any answer changed between runs.