6b1f9c8course(production):curriculum preview · demo
Ship It and See It
Tracing, latency budgets, cost per request and alerts for features that call models and tools.
- track
- Production
- level
- Advanced
- length
- 5 modules · about 12 hours
Your progress dot moves to this branch
Prerequisites
- A working AI feature
- Comfortable reading logs
6b1f011docs:outcomes and build
You will be able to
- Trace every request as spans
- Set and enforce a latency budget
- Turn tokens into cost per action
- Alert on the few signals that matter
Hands-on build
A health dashboard
Traces, cost, latency and refusal rates for one feature on one page.
6b1f022feat:5 modules
Modules.
- module 1
Traces, not just logs
Outcome: Every request is one trace with nested spans.
Lessons: Spans and attributes · What to record from a model call · Privacy: what never to record
free Trace an agent run, end to end covers this module interactively.
- module 2
Latency budgets
Outcome: A budget per step and a view of who breaks it.
Lessons: Where time goes · Parallel vs sequential tools · Caching honestly
free Trace an agent run, end to end covers this module interactively.
- module 3
Fallbacks
Outcome: Each failure has a degraded but useful response.
Lessons: Timeouts per dependency · Smaller model fallback · Search-only fallback
- module 4
Cost and alerts
Outcome: Cost per action and three alerts worth waking up for.
Lessons: Tokens to money · Spike alerts · Refusal rate as a signal
free Trace an agent run, end to end covers this module interactively.
- module 5
Incidents
Outcome: A runbook and a practice incident.
Lessons: Finding the bad span · Rolling back a prompt · Writing the note after
6b1f033docs:sample lesson
Sample lesson · Privacy: what never to record
Traces are only useful if people can read them, which means they must be safe to read. Record token counts, timings, tool names and ids. Redact email addresses, card numbers and free text the user typed unless you have a clear reason and consent.
A good default is to record the shape of the data, not the data: that a field was present and how long it was.
6b1f044test:assessment rubric
Ready to merge when
| Criterion | Ready to merge | Not yet |
|---|---|---|
| Visibility | Any bad answer can be traced to the step that caused it. | Only the final answer is logged. |
| Budgets | Latency and cost budgets exist and are checked. | Nobody knows what a request costs. |
| Response | A runbook was used in a practice incident. | No plan for a bad day. |