Build my path

0c5a8b3lesson(ship-and-see-it):free lesson · 35 min · works in your browser

Trace an agent run, end to end

Run a recorded support agent with your own timeouts, retries, approval limit and model routing. Read the trace, price the run, and make six production test cases pass.

Part of Ship It and See It. Sandboxes are deterministic simulations: no code runs on a server and no model is called.

Why this matters

If you cannot see the run, you cannot fix it

An agent run is a chain: the model plans, calls a tool, reads the result, calls another tool, then replies. When the final answer is wrong or slow, the cause is usually one link in that chain. A trace records every link as a span with a start time, a duration, a status and a few attributes such as token counts.

Traces are also where cost and speed become visible. Each model span has input and output tokens, which turn into money. Each span has a duration, which adds up to the wait your user feels. Without a trace those are guesses.

This lesson replays one recorded support conversation through a simulated agent. Nothing calls a real model. The latencies and token counts are fixed, so the same settings always give the same trace and you can compare changes fairly.

Architecture · The run you will trace
  1. PlanModel reads the message
  2. lookup_orderRead tool, safe to retry
  3. DecideModel proposes a refund
  4. ApprovalA person, above the limit
  5. issue_refundWrite tool, needs an idempotency key
  6. ReplyModel writes to the customer

Blue edge: model · orange: tool · green: person · grey: data · plain: your code

Build step

Four settings that decide how a run fails

Tool timeout: how long you wait for a tool before giving up on that attempt. Too short and slow but healthy calls fail. Too long and a dead service holds the user hostage.

Retries: how many more attempts after a timeout. Reads like lookup_order are safe to retry. Writes like issue_refund are only safe to retry with an idempotency key: a unique id sent with the request so the refund service runs it once even if it receives it twice.

Approval limit: the amount above which a refund waits for a person. This is enforced in code at the policy step, so the model cannot talk its way past it. Our demo company policy says anything over 250.00 needs a person.

Model routing: send easy turns to a fast, cheap model and hard ones to a slower, careful model. Routing is how you meet a cost budget without failing the cases that need more care.

runTool.ts ts
export async function runTool(tool: Tool, args: unknown, cfg: RunConfig, span: Span) {  const key = tool.writes ? idempotencyKey(span.traceId, tool.name, args) : undefined;  for (let attempt = 0; attempt <= cfg.retries; attempt++) {    if (tool.writes && attempt > 0 && !key) throw new Error("refusing to retry a write without a key");    try {      return await withTimeout(tool.call(args, { idempotencyKey: key }), cfg.timeoutMs);    } catch (err) {      span.event("attempt_failed", { attempt, reason: String(err) });      await sleep(200 * 2 ** attempt); // backoff: 200ms, 400ms, 800ms    }  }  return fallback(tool, args); // tell the customer clearly, hand over to a person}

Try it

Replay the run with your settings

Change the settings and the scenario, then read the trace. Each bar is a span. Retries appear as extra attempts on the same tool. A run held for approval stops at the approval span. All timings and token counts are from a fixed recording, labelled illustrative.

sandbox/agent-replayrecorded run · illustrative timings and prices

customer → "Order ord_1042 arrived damaged. Can I get a refund?"

5000 ms
2 more attempts
refunds over 500.00 wait for a person
Model
Refunded 64.00 once and confirmed to the customer.
total 5.11 s · 3700 in / 370 out tokens · cost 0.0026 (illustrative)
plan700 ms
lookup_order #12400 ms
decide600 ms
policy5 ms
issue_refund #1500 ms
reply900 ms
Span notes and log lines
  • plan: all steps on fast model (ok)
  • lookup_order #1: read tool (ok)
  • decide: propose refund (ok)
  • policy: 64.00 within limit (ok)
  • issue_refund #1: write tool (ok)
  • reply: confirm to the customer (ok)
+ 0ms plan on fast model, 900 in / 120 out tokens + 700ms lookup_order attempt 1 ok in 2400 ms + 3100ms decide on fast model, 1300 in / 90 out tokens + 3705ms issue_refund attempt 1: refund issued + 4205ms reply on fast model, 1500 in / 160 out tokens

Latency and cost

Turn the trace into money and waiting time

Cost per run is input tokens times the input price plus output tokens times the output price, for every model span. Tool calls usually cost little in money but a lot in time. Prices in this lesson are illustrative round numbers, not any provider's real price list. Plug in your provider's current prices before making decisions.

Latency adds up along the chain: spans that run one after another add their durations, spans that run in parallel cost only the slowest one. Retries add a timeout plus a backoff each time, which is why a generous timeout with many retries can make a dead service feel like a frozen app.

worksheet/cost-and-latencyillustrative prices
2,000
3
1,200
120
0 %
Model
per action
0.00252
per month
151.2
model wait
about 2.2 s per action, before tools
budget check
under the lesson budget of 0.010

Cached input is priced at a tenth of normal input here, a common pattern but not a promise. Use your provider's real prices.

Test your result

Make six production cases pass

Each case replays a different recording: a slow but healthy lookup, a large refund, a refund whose response was lost, a lookup service that is down, a cost budget, and a hard case the fast model gets wrong. One configuration should pass all six. Finding it is the lesson: every setting trades speed, cost and safety against each other.

tests/agent-production.specnot run yet
5000 ms
2 more attempts
refunds over 500.00 wait for a person
Model

What broke

What broke for other builders

  • Double refunds. A refund response was lost to a timeout and the retry ran the refund again. Writes need an idempotency key before they are ever retried.
  • Frozen app. Timeout of 10 seconds with 3 retries meant a dead service made users wait over 40 seconds for an error.
  • Approval in the prompt. The prompt said to ask before large refunds. A long conversation pushed that instruction out of focus and a large refund ran.
  • Logs without ids. Every log line was a sentence with no request id, so no one could connect a customer complaint to the run that caused it.
  • Recording private text. Full customer messages were stored in traces forever. Record shapes, ids and counts by default and redact free text.

Next commit

Next commit: put it on a dashboard

You now have the pieces of a production AI feature: contracts on tool calls, retrieval that cites or refuses, and traced runs with safe retries and an approval boundary. The Production track turns these into a dashboard, alerts and a regression suite that runs on every change.

Quick check

1. Which tool is safe to retry without an idempotency key?
2. Timeout 4000 ms, 2 retries, service is down. Roughly how long until fallback?
3. Why route easy turns to a smaller model?

Ship the demo

  • ○ Test your result: every case passing at least once
  • ○ Quick check answered
Trace and cost dashboard: DraftSee it in Builds

Pass the tests and answer the quick check to unlock the merge.