Why this matters
If you cannot see the run, you cannot fix it
An agent run is a chain: the model plans, calls a tool, reads the result, calls another tool, then replies. When the final answer is wrong or slow, the cause is usually one link in that chain. A trace records every link as a span with a start time, a duration, a status and a few attributes such as token counts.
Traces are also where cost and speed become visible. Each model span has input and output tokens, which turn into money. Each span has a duration, which adds up to the wait your user feels. Without a trace those are guesses.
This lesson replays one recorded support conversation through a simulated agent. Nothing calls a real model. The latencies and token counts are fixed, so the same settings always give the same trace and you can compare changes fairly.
- PlanModel reads the message
- lookup_orderRead tool, safe to retry
- DecideModel proposes a refund
- ApprovalA person, above the limit
- issue_refundWrite tool, needs an idempotency key
- ReplyModel writes to the customer
Blue edge: model · orange: tool · green: person · grey: data · plain: your code
Build step
Four settings that decide how a run fails
Tool timeout: how long you wait for a tool before giving up on that attempt. Too short and slow but healthy calls fail. Too long and a dead service holds the user hostage.
Retries: how many more attempts after a timeout. Reads like lookup_order are safe to retry. Writes like issue_refund are only safe to retry with an idempotency key: a unique id sent with the request so the refund service runs it once even if it receives it twice.
Approval limit: the amount above which a refund waits for a person. This is enforced in code at the policy step, so the model cannot talk its way past it. Our demo company policy says anything over 250.00 needs a person.
Model routing: send easy turns to a fast, cheap model and hard ones to a slower, careful model. Routing is how you meet a cost budget without failing the cases that need more care.
export async function runTool(tool: Tool, args: unknown, cfg: RunConfig, span: Span) { const key = tool.writes ? idempotencyKey(span.traceId, tool.name, args) : undefined; for (let attempt = 0; attempt <= cfg.retries; attempt++) { if (tool.writes && attempt > 0 && !key) throw new Error("refusing to retry a write without a key"); try { return await withTimeout(tool.call(args, { idempotencyKey: key }), cfg.timeoutMs); } catch (err) { span.event("attempt_failed", { attempt, reason: String(err) }); await sleep(200 * 2 ** attempt); // backoff: 200ms, 400ms, 800ms } } return fallback(tool, args); // tell the customer clearly, hand over to a person}Try it
Replay the run with your settings
Change the settings and the scenario, then read the trace. Each bar is a span. Retries appear as extra attempts on the same tool. A run held for approval stops at the approval span. All timings and token counts are from a fixed recording, labelled illustrative.
customer → "Order ord_1042 arrived damaged. Can I get a refund?"
Span notes and log lines
- plan: all steps on fast model (ok)
- lookup_order #1: read tool (ok)
- decide: propose refund (ok)
- policy: 64.00 within limit (ok)
- issue_refund #1: write tool (ok)
- reply: confirm to the customer (ok)
Latency and cost
Turn the trace into money and waiting time
Cost per run is input tokens times the input price plus output tokens times the output price, for every model span. Tool calls usually cost little in money but a lot in time. Prices in this lesson are illustrative round numbers, not any provider's real price list. Plug in your provider's current prices before making decisions.
Latency adds up along the chain: spans that run one after another add their durations, spans that run in parallel cost only the slowest one. Retries add a timeout plus a backoff each time, which is why a generous timeout with many retries can make a dead service feel like a frozen app.
- per action
- 0.00252
- per month
- 151.2
- model wait
- about 2.2 s per action, before tools
- budget check
- under the lesson budget of 0.010
Cached input is priced at a tenth of normal input here, a common pattern but not a promise. Use your provider's real prices.
Test your result
Make six production cases pass
Each case replays a different recording: a slow but healthy lookup, a large refund, a refund whose response was lost, a lookup service that is down, a cost budget, and a hard case the fast model gets wrong. One configuration should pass all six. Finding it is the lesson: every setting trades speed, cost and safety against each other.
What broke
What broke for other builders
- Double refunds. A refund response was lost to a timeout and the retry ran the refund again. Writes need an idempotency key before they are ever retried.
- Frozen app. Timeout of 10 seconds with 3 retries meant a dead service made users wait over 40 seconds for an error.
- Approval in the prompt. The prompt said to ask before large refunds. A long conversation pushed that instruction out of focus and a large refund ran.
- Logs without ids. Every log line was a sentence with no request id, so no one could connect a customer complaint to the run that caused it.
- Recording private text. Full customer messages were stored in traces forever. Record shapes, ids and counts by default and redact free text.
Next commit
Next commit: put it on a dashboard
You now have the pieces of a production AI feature: contracts on tool calls, retrieval that cites or refuses, and traced runs with safe retries and an approval boundary. The Production track turns these into a dashboard, alerts and a regression suite that runs on every change.
Quick check
Ship the demo
- ○ Test your result: every case passing at least once
- ○ Quick check answered
Pass the tests and answer the quick check to unlock the merge.