Reading failures

A score of 0.84 is a starting point, not a finding. It tells you eight rows out of fifty went wrong; it tells you nothing about why, and nothing about whether the eight have anything in common. The next step is to open them.

This is called error analysis, and it is the highest-value activity in evaluation work. Read each failing case and write one plain sentence about what went wrong. Not a category yet — just a description. "Chose billing for a damaged-item refund." "Returned the invoice number as the order id." "Answered in prose instead of JSON."

Do this for every failure before changing anything. The temptation is to see a failure, adjust the prompt, and re-run — but a prompt change made after reading one example usually fixes that example and moves the problem somewhere else. Reading all of them first tells you whether you have eight separate problems or one problem eight times.

Some failures are not the model's. In this dataset, refund requests after a damaged delivery are labelled returns, but a case can be made for billing, since the customer is asking for money back. When a model gets those wrong it is disagreeing with a debatable label rather than making a mistake. Cases like that are marked in the results table.