Building a dataset

A dataset is the part of an evaluation you own. Models change, prompts change, but a good set of examples keeps its value for years, so it is usually the highest-return thing to invest in early.

Fifty examples is a useful working size, as long as you are clear about what it is for. It is small enough to read in one sitting and label by hand, and large enough to show which kinds of case fail and to catch a large regression. What fifty cannot do is confirm a small improvement. At fifty cases the 95% confidence interval on your score spans roughly twenty points, so an apparent move from 0.80 to 0.84 sits inside the noise — the honest reading is "no detectable change", not "four points better". That noise is sampling variation: the luck of which examples you happened to include, and the only thing that shrinks it is more examples, often many hundreds to see a few points reliably. Much above a few hundred you stop reading individual rows, and reading rows is where most of the learning happens.

Where the examples come from matters more than how many there are. Real production traffic is best. Support tickets, search queries, customer emails: whatever your system will actually see. When you have none yet, write examples from the situations you already know are hard, then replace them with real ones as traffic arrives.

A dataset should include cases you expect to fail. A set where everything is easy produces a high score that means nothing, because it cannot distinguish a good system from a mediocre one. The fifty tickets in this course are balanced ten per category, and thirteen of them are marked ambiguous — cases where two labels could each be defended.