Build my path

9b4f7a2lesson(retrieval-that-cites):free lesson · 30 min · works in your browser

Retrieval that cites or refuses

Search your own documents first, then answer only from what you found. You will chunk a small help centre, rank the chunks, set a refusal threshold, and measure it on six test questions.

Part of Retrieval That Cites Its Sources. Sandboxes are deterministic simulations: no code runs on a server and no model is called.

Why this matters

The model does not know your documents

A model knows a lot about the world and nothing about your returns policy as of last Tuesday. Asked about it, it will produce a fluent, confident answer that may be wrong. Retrieval fixes this by searching your documents first and giving the model only the passages it found.

This pattern is often called RAG, retrieval-augmented generation. In plain words: look it up, then answer from what you looked up. The answer is only as good as the search, so most of the work is in the search, not the prompt.

The second rule matters as much as the first. If the search finds nothing relevant, the right answer is to say so. A system that refuses when it has no basis is more useful than one that is always confident.

Architecture · One question, four steps
  1. QuestionWhat the user typed
  2. ChunksYour docs, split into passages
  3. RankerScores every chunk against the question
  4. ThresholdTop score too low? Refuse
  5. ModelAnswers from the top chunks only
  6. CitationsEach claim links to its chunk

Blue edge: model · orange: tool · green: person · grey: data · plain: your code

Build step

Chunk, rank, then cite or refuse

Chunking splits documents into passages small enough to rank. Splitting at headings keeps each chunk about one topic. Splitting every forty words is simpler but cuts sentences and mixes topics, which hurts ranking and makes citations hard to read.

Ranking scores each chunk against the question. This lesson uses BM25, a keyword scoring method that rewards chunks containing the question's rarer words. Production systems often mix keyword scores with embeddings, which match meaning rather than exact words. Start with keywords: you can see exactly why a chunk ranked where it did.

The threshold is the refusal rule. If the best chunk scores below it, the system answers that it does not have enough material instead of guessing. Set it too high and you refuse good questions. Set it too low and you answer from irrelevant text.

Citations close the loop. Every answer names the chunk it came from, so a reader can check it in one click and you can debug a bad answer by looking at what was retrieved.

answer.ts ts
export function answer(question: string, chunks: Chunk[], k = 3, threshold = 2.0) {  const ranked = rankBM25(question, chunks).slice(0, k);  if (!ranked.length || ranked[0].score < threshold) {    return { answer: "I don't have enough material to answer that.", citations: [] };  }  return {    answer: compose(question, ranked),          // the model sees only these chunks    citations: ranked.map((r) => ({ id: r.chunk.id, title: r.chunk.title })),  };}

Try it

Ask the demo help centre

The documents below belong to Parcelo, a made-up courier used for this lesson. Ask a question, then change the chunking method, the number of chunks kept (k) and the refusal threshold. Every score is computed in your browser so you can see why each chunk ranked where it did.

sandbox/parcelo-help-centre7 demo documents · BM25
Chunking
k = 1
top score must be ≥ 1.0
Answer. working days after the sender receives the item. If your parcel arrived damaged, take photos of the box and the contents before you throw anything away. [1]
[1] Chunk 2 (starts in Returns)

Ranked chunks (top 1). Highlighted words matched your question.

  1. #1 Chunk 2 (starts in Returns)score 3.12
    working days after the sender receives the item. If your parcel arrived damaged, take photos of the box and the contents before you throw anything away. Report the damage within 7 days from the order page and attach the photos.

Test your result

Measure it: recall at k on six questions

Recall at k asks one question per test case: did the right document appear in the top k chunks? It is the simplest retrieval measure and the most useful one to start with, because if the right passage never reaches the model, no prompt can save the answer.

The sixth test question has no answer in the documents. It passes only if the system refuses. Find settings that pass all six, then switch to fixed-size chunks with the same settings and watch the score drop. The ranked list shows why: fixed chunks mix topics and dilute the scores.

tests/recall-at-k.specnot run yet
Chunking
k = 1
top score must be ≥ 1.0

What broke

What broke for other builders

  • No refusal path. Every question got an answer, including ones the documents never covered. Users trusted the confident wrong ones.
  • Chunks too big. Whole pages as chunks meant the right sentence was retrieved inside two thousand irrelevant words and the model missed it.
  • Stale index. Documents changed but the index did not. Rebuild or update the index whenever a source document changes.
  • Measured by feel. A prompt change felt better in three manual tries and lowered recall on the real test set. Measure before and after.

Next commit

Next commit: trace every answer

Once retrieval is measured, record it. Store which chunks were retrieved for each answer and how long each step took, so a bad answer can be traced to the search or the model. The agent trace lesson shows how.

Quick check

1. The right passage is not in the top k. What do you fix first?
2. What should happen when the best chunk scores below the threshold?
3. Recall at 3 means:

Ship the demo

  • ○ Test your result: every case passing at least once
  • ○ Quick check answered
Cited answer box: DraftSee it in Builds

Pass the tests and answer the quick check to unlock the merge.