e9a5f13branch(deploy):git switch production
Production
See every request, know what it cost, and catch regressions before users do.
Tracing, budgets and regression tests for model features. You instrument a working app so a bad answer can be traced to the prompt, the retrieval or the tool that caused it.
- for
- Builders with something working who need it to stay working.
- time
- 4 weeks at 3 hours a week
- not for
- Projects that are still ideas. Build first, then come here.
You will ship
A monitored assistant with traces, a cost dashboard, alerts and a test suite that runs on every change.
Prerequisites
- A working AI feature, or the App Engineer project
- Comfortable reading logs
- Access to a place to deploy
e9a5e01plan:six commits to ship
The branch, commit by commit.
0/6 commits done · saved in this browser only
- commit 1One trace per request
Every model call, retrieval and tool call becomes a span with timing and token counts.
Where this happens: /lessons/agent-trace#try - commit 2Cost per request
Turn token counts into money per request and per user action.
Where this happens: /lessons/agent-trace#cost - commit 3Latency budget
Set a budget per step and find which step breaks it.
Where this happens: /courses/ship-and-see-it#m2 - commit 4Regression suite in CI
Your test set runs on every prompt or model change and blocks merges that score worse.
Where this happens: /courses/evals-before-vibes#m4 - commit 5Alerts that mean something
Alert on error rate, cost spikes and refusal spikes, not on every slow call.
Where this happens: /courses/ship-and-see-it#m4 - commit 6Ship the dashboard
One page anyone on the team can read to know if the feature is healthy.
Where this happens: /projects#trace-dashboard
e9a5c02docs:courses on this branch
Courses.
e9a5d03ship:capstone
Capstone: Trace and cost dashboard
One page showing traces, cost per run, latency per step and refusal rate.
Tests it must pass
- every request has one trace id
- cost per run under budget
- no raw user text stored
span.set({ model: m, tokens_in, tokens_out, cost_usd: price(m, tokens_in, tokens_out) });span.set({ user_text_len: text.length }); // shape, not content