Evaluating tool use

An agent that can call tools has a second kind of output: the sequence of actions it took. A reply can read perfectly while the actions behind it were wrong, so the two need scoring separately.

Start with the single-step version, because it catches most real problems. For each case, record which tool should have been called and with what arguments, then compare. Unnecessary calls matter as much as missing ones — an agent that looks up an order on every message is wasteful and slow even when its answers are right.

Grading a whole trajectory strictly is usually a mistake, because there is often more than one reasonable route. Score the properties you actually care about instead: that a required step happened, that a forbidden one did not, that the number of steps stayed within a bound.