7e5011cdocs(resources):library
Resources.
Short, usable reference pages. No email gate: read them, copy them, use them at work.
10 of 10 resources
Checklist · production, evals, newsletterThe Ship Checklist
Twelve checks before an AI feature reaches real users. The newsletter's welcome resource.
Read it
- Model output is validated by code before it is used.
- Every tool call is checked against a schema.
- Write actions above a limit need a person.
- Writes are retried only with an idempotency key.
- Each dependency has a timeout and a fallback.
- There is a test set of at least twenty real cases.
- The test set includes cases that must be refused.
- The score is checked before every merge.
- Each request has one trace id.
- Cost per request is known and has a budget.
- Free text from users is redacted in traces.
- Someone named can switch the feature off.
Template · tool calling, agentsTool schema template
A fill-in schema for any tool: name, description, arguments with types, enums, ranges and the approval rule.
Read it
- name: verb_noun, for example issue_refund.
- description: one sentence on when to use it and one on when not to.
- arguments: each with type, required or optional, and allowed values or range. Put units in names (amount_cents, timeout_ms).
- writes: true or false. If true, idempotency key required.
- approval: the condition above which a person must approve.
- errors: the messages your validator returns, written so a model can fix the call.
Cheat sheet · retrievalChunking cheat sheet
When to split by heading, by paragraph or by size, and what each costs.
Read it
- By heading: best default for help centres and docs. One idea per chunk, readable citations.
- By paragraph: for long sections. Repeat the heading at the top of each piece.
- By fixed size: only for text with no structure. Cuts sentences and mixes topics.
- Overlap: a sentence or two of overlap helps fixed-size chunks; it adds index size and duplicate hits.
- Always: keep the source id and anchor with every chunk so citations link somewhere real.
Worksheet · production, latencyLatency budget worksheet
Split the wait your user accepts into budgets per step, and find the step that breaks it.
Read it
- Write the total wait you will accept, for example 6 seconds.
- List every step in order: retrieval, model calls, tool calls, rendering.
- Give each step a budget. Steps that run in parallel share one budget.
- Worst case per tool = attempts x timeout + backoff. Check it fits.
- Compare with real traces weekly. Fix the step that is most often over budget.
Worksheet · production, costCost per request worksheet
Turn token counts into money per request, per user action and per month.
Read it
- For each model call: input tokens x input price + output tokens x output price.
- Add calls per user action (an agent turn is often three or more calls).
- Multiply by actions per day, then by 30.
- Apply your cache hit rate only to calls that are really cached.
- Set a budget per action and alert when the daily average goes over it.
Template · evalsEval case template
How to write one test case: input, expected result, check type, and why it exists.
Read it
- id: easy-01, hard-07, nasty-03. The prefix says what kind of case it is.
- input: the real message or document, with private details removed.
- expect: the tool and arguments, the source document, or refused: true.
- check: exact, contains, rule in code, or graded with a written rubric.
- why: one line on what failure this case protects against.
Glossary · basicsPlain-words glossary
The terms used across SHIPSHIFT, each in one sentence.
Read it
- Token: a piece of text a model reads or writes; prices are per token.
- Tool call: a model's request for your code to run a function with given arguments.
- Schema: the written shape data must have; code checks it.
- Retrieval (RAG): searching your own documents first and answering only from what was found.
- Eval: a test case for model behaviour with an expected result.
- Trace and span: a record of one request, made of timed steps.
- Idempotency key: an id that makes a repeated write run only once.
- Approval boundary: the rule, in code, for which actions wait for a person.
Checklist · production, incidentsRunbook for a bad AI day
What to do when answers go wrong, costs spike or a tool misbehaves.
Read it
- Find one bad example and open its trace.
- Name the step: retrieval, prompt, model, tool or policy.
- If a prompt or model changed recently, roll it back first and investigate after.
- If a tool is misbehaving, switch it to read-only or turn the feature off.
- Add the bad example to the test set before fixing it.
- Write a short note: what happened, what fixed it, what test now covers it.
Template · decisions, leadsBuild or buy one-pager
The page to fill in before choosing a vendor or starting a build.
Read it
- Job: who has the problem and what they need done.
- Cost of a wrong answer: money, trust, safety.
- Options: build, buy, wait. One paragraph each.
- Evidence: scores on the same test set.
- Running cost: per month at today's traffic and at three times today's.
- Exit: who can switch it off, and when you look again.
Checklist · evals, promptsChanging a prompt safely
Five steps so a prompt tweak does not quietly break last month's fixes.
Read it
- Keep prompts in version control next to the code.
- Run the test set before the change and save the score.
- Make one change at a time.
- Run the test set again and compare case by case, not just the total.
- Merge only if no previously passing nasty case now fails.