Build my path

6b1f9c8course(production):curriculum preview · demo

Ship It and See It

Tracing, latency budgets, cost per request and alerts for features that call models and tools.

track
Production
level
Advanced
length
5 modules · about 12 hours

Your progress dot moves to this branch

foundationsbuildautomatedeploylead

Prerequisites

  • A working AI feature
  • Comfortable reading logs

6b1f011docs:outcomes and build

You will be able to

  • Trace every request as spans
  • Set and enforce a latency budget
  • Turn tokens into cost per action
  • Alert on the few signals that matter

Hands-on build

A health dashboard

Traces, cost, latency and refusal rates for one feature on one page.

6b1f022feat:5 modules

Modules.

  1. module 1

    Traces, not just logs

    Outcome: Every request is one trace with nested spans.

    Lessons: Spans and attributes · What to record from a model call · Privacy: what never to record

    free Trace an agent run, end to end covers this module interactively.

  2. module 2

    Latency budgets

    Outcome: A budget per step and a view of who breaks it.

    Lessons: Where time goes · Parallel vs sequential tools · Caching honestly

    free Trace an agent run, end to end covers this module interactively.

  3. module 3

    Fallbacks

    Outcome: Each failure has a degraded but useful response.

    Lessons: Timeouts per dependency · Smaller model fallback · Search-only fallback

  4. module 4

    Cost and alerts

    Outcome: Cost per action and three alerts worth waking up for.

    Lessons: Tokens to money · Spike alerts · Refusal rate as a signal

    free Trace an agent run, end to end covers this module interactively.

  5. module 5

    Incidents

    Outcome: A runbook and a practice incident.

    Lessons: Finding the bad span · Rolling back a prompt · Writing the note after

6b1f033docs:sample lesson

Sample lesson · Privacy: what never to record

Traces are only useful if people can read them, which means they must be safe to read. Record token counts, timings, tool names and ids. Redact email addresses, card numbers and free text the user typed unless you have a clear reason and consent.

A good default is to record the shape of the data, not the data: that a field was present and how long it was.

6b1f044test:assessment rubric

Ready to merge when

CriterionReady to mergeNot yet
VisibilityAny bad answer can be traced to the step that caused it.Only the final answer is logged.
BudgetsLatency and cost budgets exist and are checked.Nobody knows what a request costs.
ResponseA runbook was used in a practice incident.No plan for a bad day.