What should an agent emit?

An ordinary request produces one log line. An agent produces a tree: a model call, the tools it asked for, another model call reading their results, and so on until it stops. Logging the final answer alone is like logging only the return value of a function that made forty network calls — it tells you what came out and nothing about why.

Tracing is the shape that fits. A trace is a tree of spans, each with a start, a duration, a status, and attributes. OpenTelemetry is the vendor-neutral standard for producing them, and since 2024 it has had semantic conventions specifically for generative AI — agreed attribute names, so that a span from your agent means the same thing in Honeycomb, Datadog, Jaeger, or a file.

The GenAI conventions define an operation name per span kind. The ones an agent uses most:

invoke_agent support-agent          the whole run
  chat meta/llama-3.1-8b-instruct   one model call
    execute_tool read_policy        one tool execution
    execute_tool lookup_order
  chat meta/llama-3.1-8b-instruct
    execute_tool issue_refund
  chat meta/llama-3.1-8b-instruct   the final answer

span name = {gen_ai.operation.name} {gen_ai.request.model}

An inference span requires gen_ai.operation.name and gen_ai.provider.name, and recommends a handful more: the request and response model, input and output token counts, finish reasons, temperature. A tool-execution span requires only the operation name and gen_ai.tool.name — it is not a call to a provider. The lab beside this converts a real run into that structure, with the token counts the API actually returned, and you can click any span to see what it carries.

The most important design decision is also the easiest to get wrong by default: what to record. The conventions split attributes into the recommended set — models, tokens, durations, finish reasons — and an opt-in set containing content: the prompt, the messages, the tool arguments and results. The opt-ins are where the customer's data lives.

• Record the recommended set on every span. It carries no content and answers most questions: how slow, how expensive, which model, which tool, did it fail.
• Treat content attributes as a decision per deployment, not a default. Tool arguments may contain an order number and an address; messages may contain anything.
• If you do record content, sample it, redact it, and set retention. A trace store full of customer conversations is a liability with a dashboard.
• Add your own attributes under your own prefix for what the conventions do not cover — iteration number, stop reason, whether a tool call had a side effect. The lab uses assay.*

Metrics come from the same spans and are what you alert on. For inference the conventions define gen_ai.client.operation.duration (in seconds) and gen_ai.client.token.usage, both as histograms; for agents they add gen_ai.invoke_agent.inference_calls, gen_ai.invoke_agent.tool_calls and gen_ai.invoke_agent.duration. Add two of your own on top — tool error rate and side-effect count — because those are the numbers that move first when something upstream changes.

Read a trace the way you would read a profiler. The waterfall shows where time went — almost always in the model calls, which is why the lab's tool spans are a few milliseconds wide. Then the attributes on the slow or failed span show what it was doing. Then the content, if you recorded it, shows what it said.

• A run that hit its iteration limit: look for a repeated execute_tool span with identical arguments. That is a stall, and the repeat flag on the span makes it visible without reading content.
• A run that was slow: sort chat spans by duration. One model call at four seconds among six at four hundred milliseconds is a cold start or a rate limit, not your code.
• A run that did something it should not: find the execute_tool span with side_effect true, and walk up to the chat span that requested it. Its input messages are the evidence.