Build my path

7e5011cdocs(resources):library

Resources.

Short, usable reference pages. No email gate: read them, copy them, use them at work.

10 of 10 resources

Checklist · production, evals, newsletter

The Ship Checklist

Twelve checks before an AI feature reaches real users. The newsletter's welcome resource.

Read it
  1. Model output is validated by code before it is used.
  2. Every tool call is checked against a schema.
  3. Write actions above a limit need a person.
  4. Writes are retried only with an idempotency key.
  5. Each dependency has a timeout and a fallback.
  6. There is a test set of at least twenty real cases.
  7. The test set includes cases that must be refused.
  8. The score is checked before every merge.
  9. Each request has one trace id.
  10. Cost per request is known and has a budget.
  11. Free text from users is redacted in traces.
  12. Someone named can switch the feature off.
Template · tool calling, agents

Tool schema template

A fill-in schema for any tool: name, description, arguments with types, enums, ranges and the approval rule.

Read it
  1. name: verb_noun, for example issue_refund.
  2. description: one sentence on when to use it and one on when not to.
  3. arguments: each with type, required or optional, and allowed values or range. Put units in names (amount_cents, timeout_ms).
  4. writes: true or false. If true, idempotency key required.
  5. approval: the condition above which a person must approve.
  6. errors: the messages your validator returns, written so a model can fix the call.
Cheat sheet · retrieval

Chunking cheat sheet

When to split by heading, by paragraph or by size, and what each costs.

Read it
  1. By heading: best default for help centres and docs. One idea per chunk, readable citations.
  2. By paragraph: for long sections. Repeat the heading at the top of each piece.
  3. By fixed size: only for text with no structure. Cuts sentences and mixes topics.
  4. Overlap: a sentence or two of overlap helps fixed-size chunks; it adds index size and duplicate hits.
  5. Always: keep the source id and anchor with every chunk so citations link somewhere real.
Worksheet · production, latency

Latency budget worksheet

Split the wait your user accepts into budgets per step, and find the step that breaks it.

Read it
  1. Write the total wait you will accept, for example 6 seconds.
  2. List every step in order: retrieval, model calls, tool calls, rendering.
  3. Give each step a budget. Steps that run in parallel share one budget.
  4. Worst case per tool = attempts x timeout + backoff. Check it fits.
  5. Compare with real traces weekly. Fix the step that is most often over budget.
Worksheet · production, cost

Cost per request worksheet

Turn token counts into money per request, per user action and per month.

Read it
  1. For each model call: input tokens x input price + output tokens x output price.
  2. Add calls per user action (an agent turn is often three or more calls).
  3. Multiply by actions per day, then by 30.
  4. Apply your cache hit rate only to calls that are really cached.
  5. Set a budget per action and alert when the daily average goes over it.
Template · evals

Eval case template

How to write one test case: input, expected result, check type, and why it exists.

Read it
  1. id: easy-01, hard-07, nasty-03. The prefix says what kind of case it is.
  2. input: the real message or document, with private details removed.
  3. expect: the tool and arguments, the source document, or refused: true.
  4. check: exact, contains, rule in code, or graded with a written rubric.
  5. why: one line on what failure this case protects against.
Glossary · basics

Plain-words glossary

The terms used across SHIPSHIFT, each in one sentence.

Read it
  1. Token: a piece of text a model reads or writes; prices are per token.
  2. Tool call: a model's request for your code to run a function with given arguments.
  3. Schema: the written shape data must have; code checks it.
  4. Retrieval (RAG): searching your own documents first and answering only from what was found.
  5. Eval: a test case for model behaviour with an expected result.
  6. Trace and span: a record of one request, made of timed steps.
  7. Idempotency key: an id that makes a repeated write run only once.
  8. Approval boundary: the rule, in code, for which actions wait for a person.
Checklist · production, incidents

Runbook for a bad AI day

What to do when answers go wrong, costs spike or a tool misbehaves.

Read it
  1. Find one bad example and open its trace.
  2. Name the step: retrieval, prompt, model, tool or policy.
  3. If a prompt or model changed recently, roll it back first and investigate after.
  4. If a tool is misbehaving, switch it to read-only or turn the feature off.
  5. Add the bad example to the test set before fixing it.
  6. Write a short note: what happened, what fixed it, what test now covers it.
Template · decisions, leads

Build or buy one-pager

The page to fill in before choosing a vendor or starting a build.

Read it
  1. Job: who has the problem and what they need done.
  2. Cost of a wrong answer: money, trust, safety.
  3. Options: build, buy, wait. One paragraph each.
  4. Evidence: scores on the same test set.
  5. Running cost: per month at today's traffic and at three times today's.
  6. Exit: who can switch it off, and when you look again.
Checklist · evals, prompts

Changing a prompt safely

Five steps so a prompt tweak does not quietly break last month's fixes.

Read it
  1. Keep prompts in version control next to the code.
  2. Run the test set before the change and save the score.
  3. Make one change at a time.
  4. Run the test set again and compare case by case, not just the total.
  5. Merge only if no previously passing nasty case now fails.