Deep dive · 18 min

What is your score actually worth?

Your eval says 0.84. Is that better than last week's 0.80, or is it the same number wearing a different hat?

Every lesson on this site so far has taught you to compare two numbers. Run the eval, change something, run it again, see whether the score went up. None of them have told you when a difference is real — and most of the time, at the dataset sizes people actually use, it is not.

An eval score is an estimate. You ran fifty cases drawn from the space of things your users might send, and 84% of them passed. Run a different fifty and you would get a different number. The question is how different.

Point estimateThe score you measured. 0.84.

Confidence intervalThe range the true rate plausibly sits in, given how many cases you ran. For 42 of 50 passing, roughly 0.71 to 0.92.

Statistical powerThe chance your eval would notice a real improvement of a given size. Most small evals have far less than people assume.

That interval is the uncomfortable part. Forty-two passes out of fifty gives a 95% interval spanning about twenty points. A change from 0.80 to 0.84 sits entirely inside it — which means the honest reading is not "slightly better" but "indistinguishable".

This is not a reason to stop measuring. It is a reason to know which differences your measurement can see, so you stop shipping changes on noise and stop discarding changes that were real but too small for fifty cases to detect.

Two sources of variation get confused here, and they need separating.

Sampling variationDifferent cases would give a different score. Fixed by more cases — this is what the interval above measures.

Run-to-run variationThe same cases give a different score on a second run, because the model is not deterministic. Fixed by more repetitions, or by temperature 0 — though even then some models vary.

The lab measures both. It runs the same cases several times so you can see run-to-run spread directly, and computes an interval on the pooled result so you can see sampling spread. Then it answers the practical question: given the score you just measured, how many cases would you need to reliably detect an improvement of five points?

42 / 50 passed      score 0.84
95% interval        0.71 - 0.92        (Wilson)
run-to-run spread   0.82 - 0.86        (5 repetitions)

to detect +5 points:    733 cases per variant, so 1,466 labelled
to detect +10 points:   153 cases per variant

Those sample sizes are the finding, and they are why almost nobody reports intervals: they are inconvenient. Fifteen hundred labelled cases is a serious undertaking, and most teams have fifty.

The response is not despair. It is to stop pretending small differences are signal, and to change what you measure so that differences get larger:

Measure per failure mode, not overall. A change that fixes one mode moves that mode's rate a lot, even when the overall score barely shifts.
Use paired comparisons. Running both variants on the same cases removes case difficulty from the comparison entirely and needs far fewer examples.
Weight the cases you care about. A score across everything hides a large improvement on the ten cases that actually matter.
Report the interval. A number without one invites a decision the number cannot support.

Paired comparison is the highest-value change here, and the least known. Run both variants over identical cases and you compare per case, which removes case difficulty from the comparison entirely. How much it saves depends on how often the two variants actually disagree: at 20% disagreement the same detection takes 626 cases against 1,466 unpaired, and at 6% — which is typical for a prompt tweak — it takes 186. The rarer the disagreement, the bigger the saving, which is the opposite of most intuitions.