Lesson 01 · Foundations · 8 min

What an evaluation is

After this lesson you will be able to explain why one successful run is not evidence of reliability, and name the three things an evaluation is made of.

An evaluation is a repeatable measurement of how well a system does a task. For ordinary software the measurement is usually a test suite: the same input produces the same output, so a passing test is a fact about the program. Language models complicate this in one specific way — the same input can produce different outputs on different runs.

That single difference is what makes evaluation a separate skill. You are no longer asking whether the program is correct. You are asking how often it is correct, on which kinds of input, and whether that rate is going up or down as you change things.

Three parts: inputs with the answers you expect, the system under test, and a grader that compares what came out with what should have. The score is the grader's average over the dataset.

EvalA dataset of inputs, a set of expected results, and one or more graders that score the system's output against them.

GraderA function that takes what the system produced and what you expected, and returns a score. It may be plain code or another model.

GoldenA single reference example — one input paired with the answer you consider correct.

Below, the same message is sent to a model several times. If the replies differ, that variation is the reason a single spot-check cannot tell you whether a change helped. If they are identical, that is worth knowing too: at temperature 0 many models are close to deterministic, and the variation you care about comes from differences between inputs rather than between runs.

Throughout this course the worked example is a support-ticket triage assistant. It returns a category, an order number, an urgency and a short reply — three kinds of output that need three different kinds of grader.

Checkpoint

Run the same five cases twice and note whether any answer changed. If one did, you have just seen why a single run proves nothing — the system is not deterministic, and one success could have been luck. If none did, the model is close to deterministic at temperature 0, and the variation that matters will come from different inputs rather than repeated ones. Either way, the lesson is the same: reliability is a rate, measured over many cases, not a fact established by one.