Comparing models

Once an evaluation exists, pointing it at a different model costs one string change. The endpoint speaks the OpenAI format, so the same code runs against Nemotron, Llama, DeepSeek, MiniMax and the rest without modification.

One thing to know before sweeping: a model catalogue is not a list of what you can run. The catalogue endpoint needs no API key, so it returns every model the provider hosts rather than the ones your key is entitled to. When this course was last checked, 23 of the 102 listed models answered — most of the rest returned a 404, and seven accepted the connection and then never replied at all.

That matters more than it sounds. A model that returns an error is easy to handle; a model that goes quiet will hang your evaluation until something times out. The picker beside this lesson marks what answered and how fast, and every run here is preflighted with a single cheap call before the whole dataset is committed to a model.

A sweep runs the same dataset and the same graders across several models and puts the scores side by side, broken down by category. Read it in two directions. Down a column tells you which models handle that category well. Across a row tells you what a single model is good and bad at.

The most useful thing a sweep shows is a column where every model scores badly. Models trained by different organisations on different data rarely share a specific weakness — so when they all dip on the same category, the more likely explanation is that the category is defined badly or labelled inconsistently. That is a finding about your dataset, discovered by looking at models.

Accuracy is not the only column that matters. A model two points behind but four times faster and cheaper may well be the right choice, and a sweep that reports only a score hides that trade-off.