Building a dataset

A dataset is the part of an evaluation you own. Models change, prompts change, but a good set of examples keeps its value for years, so it is usually the highest-return thing to invest in early.

Fifty examples is a useful starting size. It is small enough to read in one sitting and label by hand, and large enough that a change of a few percentage points is visible. Below about twenty, ordinary run-to-run variation swamps any real difference. Much above a few hundred, you stop reading individual rows — and reading rows is where most of the learning happens.

Where the examples come from matters more than how many there are. Real production traffic is best. Support tickets, search queries, customer emails: whatever your system will actually see. When you have none yet, write examples from the situations you already know are hard, then replace them with real ones as traffic arrives.

A dataset should include cases you expect to fail. A set where everything is easy produces a high score that means nothing, because it cannot distinguish a good system from a mediocre one. The fifty tickets in this course are balanced ten per category, and thirteen of them are marked ambiguous — cases where two labels could each be defended.