# Fix retrieval before you blame the model Express course on RAG and grounding. 22 concepts — knowledge cutoff, RAG, grounding, chunking, metadata, embeddings, vector search, hybrid search, reranking, precision and recall, citations, honest refusal, faithfulness, golden sets, index freshness and more — each with its English name, a diagram, an example and a card. The core idea: in RAG the model is rarely the problem — your retrieval is. A one-page cheat sheet of every concept at the end. - **Author:** Fedor Chishchin - **Level:** intermediate - **Reading time:** ~35 minute read - **Published:** 2026-06-13 - **Source:** https://fedorthinks.com/learn/rag-and-grounding-express --- ## 1. Why RAG exists A model doesn't know your data, or anything after its training cutoff. RAG is how you give it the facts to answer from — so understand the problem it solves before the mechanics. ### The map: 22 concepts in seven groups > A metro map: first you see the whole network, then you ride it station by station. First, three **basics** — why the model's memory isn't enough, what RAG is, and why grounding matters. Then six groups, one per chapter: **chunking**, **embeddings and search**, **retrieval quality**, **grounding**, **evaluation**, and **building it well**. Every concept after this follows the same shape: the **English name** (remember it — it's the term), a real-life picture, a **diagram**, and a **card** with fields: _what it is_ or _problem_, _how it works_ or _idea_, _example_, _catch_ or _cost_, and _related_. _Diagram: The basics on top, six groups below — chapters 2–7 follow them._ ### Knowledge Cutoff — the model's memory is frozen > A brilliant graduate who read the whole library years ago — but not your company's files, and nothing printed since they graduated. An LLM only knows what was in its training data, frozen at a cutoff date — its **knowledge cutoff**. It doesn't know your documents, your product, your customer's record, or yesterday's news. Ask it anyway and it will **answer from fuzzy memory or invent something plausible.** RAG exists to fix exactly this: stop relying on what the model memorised, and hand it the relevant facts at question time. _Diagram: Anything after the cutoff, or inside your company, the model has never seen._ #### Knowledge Cutoff (training cutoff) - **What it is:** The model only knows its training data, frozen at one date. - **Example:** It doesn't know your docs, a customer's record, or yesterday's news. - **Catch:** The model doesn't go quiet when it doesn't know — it answers from fuzzy memory or invents. - **Related:** **RAG** closes this gap: the facts arrive at question time. ### RAG — retrieve first, then answer > An open-book exam — instead of trusting memory, you look up the relevant pages first, then write your answer from what's in front of you. This is the **Retrieval-Augmented Generation (RAG)** pattern. Three steps: **retrieve** — take the user's question and **fetch the most relevant chunks** of your documents; **augment** — put them in the context; **generate** — ask the model to answer from those. Now the answer comes from your actual data, not the model's recollection. This is how an assistant answers about your docs, your policies, your knowledge base — accurately and up to date. _Diagram: The three steps of RAG: retrieve chunks, add them to the context, generate the answer._ #### Retrieval-Augmented Generation (RAG) - **Problem:** The model doesn't know your data or anything after its cutoff. - **Idea:** Find the relevant pieces of your documents and have the model answer from them. - **Example:** A support assistant answers about the refund policy from the current knowledge-base article. - **Cost:** A whole search system: chunking, an index, freshness, evaluation. - **When to use:** The knowledge is large, changes often, or depends on the user. - **Related:** Exists for **grounding**; built on **chunking**, **embeddings** and **vector search**. ### Grounding — the real prize > A journalist who must cite a source for every claim writes far fewer fabrications than one writing from memory. The deep reason RAG matters is **grounding**: because the answer is drawn from retrieved text, you can demand citations and check that each claim is actually supported by a source. It doesn't fully erase hallucination, but it slashes it and makes what slips through catchable. An answer you can trace to a document is one you can trust — and defend. _Diagram: Every claim has a source. A claim without one stands out immediately._ #### Grounding - **What it is:** The answer rests on retrieved text, not on the model's memory. - **How it works:** Each claim can be traced to a specific chunk and checked. - **Example:** "Refunds take 14 days [1]", where [1] is a paragraph of the refund policy. - **Catch:** It slashes hallucination but doesn't take it to zero. - **Related:** Done through **context-only answering**, **citations** and **refusal**; measured by **faithfulness**. ### Retrieval-first — a search system with a model on the end > A research assistant is only as good as the filing system they search — give them a great library and a bad index, and they still hand you the wrong folder. The mental model that saves you months: RAG is mostly a **search problem**, with a language model on the end to phrase the answer. The model is the easy, mostly-solved part. The hard, quality-determining part is everything that decides _which text reaches the model_ — and that's the rest of this course. _Diagram: Chunking, embedding, search and ranking are the hard part. The model only phrases the answer._ > **The framing question:** "Is this answer coming from facts I retrieved and can cite — or from the model's frozen, fuzzy memory?" > Don't ask the model what it knows. Retrieve the relevant facts and ask it to answer from those — grounded, current, and citable. ## 2. Chunking Before you can retrieve, you split your documents into pieces. How you cut them quietly decides retrieval quality — it's the least glamorous step and one of the most important. ### Chunk — the unit of quality > A reference book is only useful cut into findable entries — one giant unindexed scroll, or a thousand torn scraps, are both useless to look things up in. You don't retrieve whole documents; you retrieve **chunks** — passages a few hundred words long. The process of cutting them is **chunking**. That chunk is the unit the search sees and the model reads, so its boundaries matter enormously. Good chunks are self-contained and on one topic. Get chunking wrong and no amount of clever search or smart model recovers it. _Diagram: A document is cut into chunks. Search sees chunks, the model reads chunks — not documents._ #### Chunk (passage) - **What it is:** A passage a few hundred words long — the unit of search and reading. - **How it works:** Documents are split ahead of time; search returns chunks, the model answers from them. - **Size:** Too big buries the answer, too small loses the context. You find the number by measuring. - **Catch:** Bad chunking can't be rescued by search or by the model. - **Related:** Cut with **structure-aware chunking**, tag with **metadata**, turn into an **embedding**. ### Chunk Size — too big buries the answer, too small loses the context > Hand someone an entire chapter to answer a one-line question and they wade through noise; hand them a single sentence with no surrounding context and they can't tell what it means. Chunk too **large** and each one mixes the relevant fact with paragraphs of unrelated text — search gets fuzzy and the model's attention dilutes. Chunk too **small** and you sever the context that made the fact meaningful. The sweet spot holds one coherent idea with enough surroundings to stand alone. There's no universal **chunk size**; it depends on your documents, and you tune it by measuring. _Diagram: The coloured bar is the fact you need. On the left it drowns in noise; on the right it has lost its context._ ### Structure-Aware Chunking — cut on meaning, not on character count > Splitting a book by counting characters can cut a sentence — or a recipe — clean in half. Splitting on chapters and sections keeps each piece whole. The naive approach — **fixed-size chunking** — chops every N characters, which slices through sentences, tables, and ideas. Better is to split on the document's **natural structure** — headings, sections, paragraphs — so each chunk is a complete thought. A little **overlap** between adjacent chunks keeps a fact from falling into the crack between two of them. Respect the shape of the document and retrieval gets easier for free. _Diagram: Top: cuts every N characters. Bottom: cuts on sections, with a small overlap at the seams._ #### Structure-Aware Chunking (semantic chunking, overlap) - **Problem:** Splitting by character count cuts sentences and tables in half. - **Idea:** Split on headings, sections and paragraphs, with a small overlap. - **Example:** The whole "Refund timelines" section becomes one chunk. - **Cost:** You have to parse the document format: Markdown, HTML, PDF. - **Related:** Improves every **chunk** and everything downstream. ### Metadata — attach labels to every chunk > A library card doesn't just hold the text — it records the author, date, and section, so you can filter to exactly the right shelf before you even search. Store each chunk with **metadata**: source document, title, date, section, permissions. This lets you **filter** before or alongside the semantic search — only this product's docs, only current policies, only what this user is allowed to see — and it gives you the source for citations. Metadata is cheap to keep and turns a flat pile of text into something you can target. _Diagram: Metadata gives you two things: a filter before search, and a source for the citation._ #### Metadata (chunk metadata) - **Problem:** A flat pile of text: you can't narrow the search and have nothing to cite. - **Idea:** Store source, title, date, section and permissions with each chunk. - **Example:** Search only the current policies this user is allowed to see. - **Cost:** Almost nothing — a few fields per chunk. - **Related:** Gives **vector search** a filter and **citations** a source. > **The chunking question:** "Does each chunk stand on its own as one coherent idea — with enough context to be understood, and enough metadata to be filtered and cited?" > You retrieve chunks, not documents. Cut on meaning, keep each one self-contained, and tag it — chunking quality caps everything downstream. ## 3. Embeddings and vector search To find the chunks that match a question, you search by meaning, not by matching words. Embeddings are the trick that makes "find me things like this" possible. ### Embedding — meaning becomes coordinates > A map where every idea has a location, and things that mean similar things sit close together — "dog" near "puppy," both far from "tax return." An **embedding** is a list of numbers — a **vector** — that represents the meaning of a piece of text, placed in a high-dimensional space so that similar meanings land near each other. You embed every chunk once and store the vectors. Now "meaning" has coordinates, and "find related text" becomes "find nearby points" — a problem a computer can solve fast. _Diagram: Chunk → embedding model → vector. Once per chunk, then into the store._ #### Embedding (vector) - **What it is:** A list of numbers that represents the meaning of a text. - **How it works:** An embedding model outputs coordinates; similar meaning means nearby points. - **Example:** "Dog" and "puppy" sit close; both are far from "tax return." - **Related:** Makes **vector search** possible; the vectors live in a **vector database**. ### Vector Search — find by similarity, not keywords > A librarian who finds books by what they're about — "things like this" — instead of only matching the exact title you said. At question time you embed the **question** the same way, then search the store for the chunks whose vectors are closest to it. This is **vector search**, also called **semantic search**: it finds a chunk about "refund policy" even when the user asked about "getting my money back," because the meanings sit close. Keyword search would miss that; vector search is built for it. _Diagram: The question and the refund chunk share no words — but in meaning space they sit side by side._ #### Vector Search (semantic search) - **What it is:** Finding the chunks whose vectors are closest to the question's vector. - **How it works:** The question is embedded with the same model, then nearest points are found. - **Example:** "Get my money back" finds the "Refund policy" section. - **Catch:** It fumbles exact codes and names; near isn't always useful. - **Related:** Runs on a **vector database**; completed by **hybrid search** and **reranking**. ### Vector Database — the retrieval engine > A warehouse designed so that, given any item, it can instantly hand you the hundred most similar items — not by reading every shelf, but by how it's organised. A **vector database** stores your embeddings and answers "nearest to this" queries in milliseconds, even over millions of chunks, using an approximate index (**ANN**, approximate nearest neighbour) instead of scanning everything. It's the retrieval engine under RAG. You don't need to build it — but you do need to know it's where your chunks live and how fast and accurate its search is. _Diagram: A vector database doesn't scan everything: the approximate index goes straight to the nearest._ #### Vector Database (vector store, ANN index) - **What it is:** A store of vectors with fast nearest-neighbour search. - **How it works:** An approximate index (ANN) instead of a full scan. - **Example:** Millions of chunks — and the nearest to the question in milliseconds. - **Catch:** You don't build it, but you must know how fast and accurate its search is. - **Related:** Stores **embeddings**; **vector search** runs on it. ### Similar ≠ Relevant — similarity is not the same as relevance > Two passages can be on the same topic yet answer different questions — "how to cancel" and "why people cancel" sit close, but only one is what was asked. Vector search returns what's **semantically near (similar)**, which is usually **relevant** but not always. A chunk can be about the right subject and still not contain the answer. This gap — near in meaning versus actually useful — is why raw vector search is a strong first pass but not the final word, and why the next section exists. _Diagram: Both chunks are close to the question. Only one holds the answer._ #### Similar ≠ Relevant (similarity vs relevance) - **What it is:** A chunk close in meaning doesn't necessarily contain the answer. - **Example:** "How to cancel" and "why people cancel" sit close; only the first answers. - **Catch:** Raw vector search mistakes closeness for usefulness. - **Related:** Fixed by **reranking**; measured by **retrieval eval**. > **The search question:** "Am I matching the meaning of the question — or just hoping the user typed the same keywords my documents used?" > Embeddings give meaning coordinates; vector search finds chunks by similarity. It's powerful — and "near in meaning" isn't quite "answers the question." ## 4. Retrieval is the whole game Here's the lesson that fixes most broken RAG systems: when answers are bad, the retrieval is usually the cause. Get the right chunks in, and a decent model does the rest. ### Garbage In, Confident Garbage Out > Hand someone the wrong file and ask them to summarise it — they'll give you a flawless summary of entirely the wrong thing, sounding completely sure. A RAG answer is only as good as the chunks it was given. Retrieve the wrong passages and the model **faithfully answers from the wrong context**, with all its usual confidence. The bug looks like "the model is wrong," but the real failure happened upstream, in retrieval. So when answers are bad, **inspect what was retrieved before you touch the prompt or the model.** _Diagram: The error shows up in the answer but was born in retrieval. Look at the chunks first._ ### Hybrid Search — combine keywords and meaning > One searcher who knows exactly what the words mean, and another who matches exact names and codes — together they catch what either alone would miss. Vector search understands meaning but can fumble exact terms — product codes, names, rare jargon. **Keyword search** nails those but misses paraphrases. **Hybrid search** runs both and merges the results, covering each other's blind spots. For most real corpora, hybrid beats either alone, because real questions mix concepts with specific names. _Diagram: Two searches in parallel: one catches exact codes, the other catches rephrasings. The results are merged._ #### Hybrid Search (keyword + vector) - **Problem:** Vector search fumbles exact codes and names; keyword search misses paraphrases. - **Idea:** Run both searches and merge the results. - **Example:** "Error SKU-4471 on return" — keyword search finds the code, vector search the meaning. - **Cost:** Two indexes and the merge logic. - **When to use:** Almost always, for real corpora. - **Related:** The wide first pass before **reranking**. ### Reranking — rerank the shortlist > A hiring process: a cheap first pass pulls fifty plausible résumés, then a careful reviewer reads them closely and ranks the real top five. First-pass search is fast but rough. A **reranker** takes the top, say, fifty candidates and scores each one against the question more carefully, pushing the genuinely relevant chunks to the top. Retrieve broadly, then rerank to a precise few. This two-stage shape — wide cheap recall, then sharp ranking — is one of the highest-leverage upgrades to a mediocre RAG system. _Diagram: Stage 1 — quickly gather 50 candidates. Stage 2 — carefully pick the best 5._ #### Reranking (two-stage retrieval) - **Problem:** The first pass is fast but rough: the right chunk may sit at rank 30. - **Idea:** Take the top 50 and carefully score each candidate against the question. - **Example:** Of 50 retrieved chunks, the reranker keeps the 5 most useful. - **Cost:** An extra step — a little latency and money on every question. - **When to use:** When RAG is mediocre — one of the strongest upgrades. - **Related:** Fixes **similar ≠ relevant**; comes after **hybrid search**. ### Precision & Recall — they pull against each other > A net with wide holes catches only the big fish but lets many through; a fine net catches everything, including weeds. You tune the mesh to the catch you need. Retrieve too few chunks and you might **miss the one with the answer** (low **recall**); retrieve too many and you bury the answer in irrelevant text that dilutes the model and costs tokens (low **precision**). The right number of chunks (often called **k**) balances the two, and it's specific to your data and question type. You don't guess it — you measure it, which is the next section. _Diagram: Too few chunks — the answer is missed. Too many — it drowns. You find the middle by measuring._ #### Precision & Recall (top-k) - **What it is:** Recall — did the right chunk come back; precision — how much junk came with it. - **How it works:** Bigger k raises recall and lowers precision, and vice versa. - **Example:** k = 2 misses the answer; k = 40 buries it and burns tokens. - **Catch:** You can't guess the right k — you measure it on your own data. - **Related:** Measured by **retrieval eval**; drives **cost & latency**. > **The retrieval question:** "When an answer is wrong, did I actually look at which chunks were retrieved — or did I assume it was the model and start tweaking the prompt?" > Most bad RAG answers are bad retrieval. Combine keyword and semantic, rerank the shortlist, and check the chunks before you blame the model. ## 5. Grounding and citations Once the right chunks are in the context, the job is to make the model answer strictly from them — and to prove that it did. This is what turns retrieval into a trustworthy answer. ### Context-Only Answering — answer only from the retrieved text > A witness instructed to testify only to what they personally saw — not what they assume, remember vaguely, or heard secondhand. Instruct the model to answer **only from the provided chunks**, and to say it doesn't know when the answer isn't there. This is the core of grounding: the retrieved text is the allowed **source of truth**, and the model's own memory is off the table. It won't be perfect — the model can still drift — but the instruction plus the right context does most of the work. _Diagram: The chunks are the allowed source. The model's memory is not._ #### Context-Only Answering (answer from context) - **Problem:** The model blends retrieved facts with its own memory. - **Idea:** An instruction: answer only from the provided chunks, otherwise say "I don't know." - **Example:** "Answer only from the excerpts below. If the answer isn't there, say so." - **Cost:** Not perfect: the model can still drift. - **Related:** The core of **grounding**; checked by **citations** and **faithfulness**. ### Citations — make claims checkable > A research paper with footnotes lets any reader trace a claim back to its source and verify it — or catch where it doesn't hold. Have the model **cite which chunk** each part of its answer came from. **Citations** do two things: they let the user (and you) verify a claim against its source, and the very act of grounding each statement in a retrieved passage discourages the model from wandering off the facts. An answer with traceable sources is one you can audit; an unsourced answer is just a confident guess. _Diagram: Every sentence of the answer points to its source — you can open it and check._ #### Citations (source attribution) - **Problem:** An unsourced answer can't be checked — it's a confident guess. - **Idea:** The model says which chunk each claim came from. - **Example:** "Refunds take 14 days [1]", where [1] is refund-policy.md, section "Timelines." - **Cost:** Every chunk needs metadata with its source. - **Related:** Built on **metadata**; make **grounding** checkable. ### Refusal — teach it to say I don't know > The expert you trust most is the one who says "that's not in what I have" instead of confidently filling the gap with a guess. The most dangerous RAG failure is answering anyway when retrieval came back empty or off-target. So make **refusal a valid, expected output**: if the chunks don't contain the answer, the model should say so, not improvise from memory. A system that admits the gap is far more trustworthy than one that papers over it — and it surfaces where your retrieval needs work. _Diagram: Both branches are correct outputs. Only one thing is wrong: inventing the answer from memory._ #### Refusal (an honest “I don't know”) - **Problem:** Retrieval came back off-target, and the model answers anyway. - **Idea:** "That's not in my sources" is a normal, expected answer. - **Example:** A question outside the knowledge base → "The docs don't cover this." - **Cost:** Sometimes the user gets no answer — but never a made-up one. - **Related:** Part of **grounding**; refusals reveal gaps in retrieval. ### Grounding reduces hallucination — it doesn't end it > Seatbelts cut deaths dramatically, but you still drive carefully. A strong safeguard is not a reason to stop watching the road. Even with perfect chunks, the model can misread, over-generalise, or blend a retrieved fact with its own memory. Grounding **slashes** **hallucination** and makes it catchable; it doesn't remove it. So you still verify — with citations the user can check, and with evals that measure faithfulness. Treat grounding as a powerful control, not a guarantee. > **The grounding question:** "Can every claim in this answer be traced to a retrieved chunk — and does the system say 'I don't know' when the facts aren't there?" > Grounding means the answer comes only from retrieved text, with citations to prove it — and an honest "I don't know" when the facts aren't there. ## 6. Evaluating RAG RAG has two stages that fail differently, so you measure them separately. Lump them together and you'll tune the wrong half for weeks. (The Evals course goes deeper; here's the RAG-specific shape.) ### Two stages — measure retrieval and generation apart > A restaurant with a bad dish has two possible culprits — the ingredients delivered, or the cook. You taste the ingredients first, or you'll retrain the wrong person. A RAG answer can fail because **retrieval** brought the wrong chunks, or because **generation** answered badly from the right chunks. These need different fixes, so measure them separately: did the right chunks come back? and, given those chunks, was the answer faithful and complete? Most teams who "can't improve their RAG" are scoring the final answer only and guessing which half to blame. _Diagram: Each stage has its own meter — and its own fix._ ### Retrieval Eval — did the right chunks come back? > Before judging the essay, check the student was even handed the right reference pages — if not, nothing they wrote could have been right. Evaluate retrieval on its own — this is a **retrieval eval**: for a set of questions with known relevant documents, did the search return them in the top results? The share of such hits is often called **recall@k**. This catches bad chunking, weak search, and missing reranking directly — the upstream causes of most bad answers. **Fix retrieval first**, because no improvement to the prompt or model can rescue an answer built from the wrong chunks. _Diagram: The right document for the question is known. If it lands in the top k, the test passes._ #### Retrieval Eval (recall@k) - **Problem:** The final answer doesn't tell you whether search was to blame. - **Idea:** Questions with known right documents: did they come back in the top k? - **Example:** Of 50 real questions, the right document is in the top 5 for 41. - **What it catches:** Bad chunking, weak search, missing reranking. - **Related:** Measures **precision & recall**; fix it before **faithfulness**. ### Faithfulness — was the answer true to the chunks? > A fact-checker reads the answer against the cited sources and flags any sentence the sources don't actually support. Once retrieval is good, evaluate the answer for **faithfulness**: is every claim supported by the retrieved chunks, with nothing invented and nothing contradicted? Also check it actually used what was retrieved and answered the question. This is where you catch the model drifting off solid context — and it's measurable, often with another model as a grader (**LLM-as-judge**). _Diagram: The grader checks each sentence against the chunks and flags what they don't support._ #### Faithfulness (groundedness) - **What it is:** Every claim in the answer is backed by the retrieved chunks. - **How it works:** A grader — often another model — checks sentences against the chunks. - **Example:** Two sentences are supported, the third isn't: it gets flagged. - **Catch:** An answer faithful to the chunks still has to answer the question. - **Related:** Measures **grounding**; meaningful after a good **retrieval eval**. ### Golden Set — build the eval from real questions > You test a bridge with the trucks that will actually cross it, not with weights you find convenient. Your eval set — often called a **golden set** — should be **real questions** users ask, including the messy and the out-of-scope ones, each paired with the chunks that should be retrieved and a known-good answer. Run it on every change so you can see whether a new chunking strategy or reranker actually helped. Without this, you're tuning RAG by anecdote — and anecdotes lie. _Diagram: A table of real questions with expected chunks and answers — run on every change._ #### Golden Set (eval set) - **Problem:** Tuning by anecdote: "it seems better now." - **Idea:** Real questions + the chunks that should come back + a known-good answer. - **Example:** A messy question, a paraphrased one, and an out-of-scope one that expects "I don't know." - **Cost:** Time to build and maintain the set. - **Related:** Feeds both **retrieval eval** and **faithfulness**. > **The eval question:** "When an answer is wrong, can I tell whether retrieval missed the chunk or the model fumbled it — or am I guessing?" > RAG fails in two places. Measure retrieval and generation separately, fix retrieval first, and build the eval from real questions. ## 7. Building it well RAG is powerful and often over-reached for. The skill is using it when it earns its place, keeping the data fresh, and not paying for machinery a simpler approach would have skipped. ### Paste First — don't RAG what you could just paste > You don't build a library catalogue for the three books on your desk — you just open them. If the relevant facts are small and fixed — a few documents, a short policy — just **put them in the prompt.** RAG earns its complexity when the knowledge is too big for the context, changes often, or must be filtered per user. Building a retrieval pipeline over three pages is the over-engineering equivalent of an agent loop for a single call. Reach for it when the data forces you. _Diagram: Small and stable — paste it into the prompt. Big, changing or per-user — RAG._ #### Paste First (context stuffing) - **Problem:** A retrieval pipeline gets built where none is needed. - **Idea:** Small, stable facts go straight into the prompt. - **Example:** A two-page policy goes in the prompt, with no index. - **When to use RAG:** The knowledge doesn't fit the context, changes often, or is filtered per user. - **Related:** The bottom rung of the ladder; **RAG** is the next one. ### Index Freshness — stale data is wrong data > A phone book is only useful if it's reprinted — last decade's numbers are confidently, uselessly wrong. RAG's promise is current facts, which means your index has to **stay current.** When source documents change, the chunks and embeddings must be updated — **reindexing** — or the model will ground its answers in confident, outdated text. Plan reindexing as part of the system, not an afterthought — freshness is a feature you have to maintain, not a property you get once. _Diagram: Top: docs change and the index is rebuilt. Bottom: no plan, and the answers go stale._ #### Index Freshness (reindexing) - **Problem:** The docs changed but the index didn't — answers are confidently outdated. - **Idea:** Re-chunk and re-embed changed documents on a plan. - **Example:** The policy is updated — its chunks are rebuilt the same day. - **Cost:** A reindexing pipeline you have to maintain. - **Related:** Keeps **grounding** current; touches **chunks** and **embeddings**. ### Cost & Latency — watch them > Every extra step in the pipeline — embed, search, rerank, stuff context, generate — adds a little time and a little money to every single question. RAG adds stages, and each one costs **latency** and tokens. Retrieving more chunks, reranking, larger context — all improve quality up to a point and raise the bill the whole way. So tune for the **fewest chunks that answer well**, cache where you can, and remember that a great answer that's too slow or too expensive isn't a great answer in production. _Diagram: Every stage adds time and money — and every question pays for it._ #### Cost & Latency - **What it is:** Every RAG stage adds time and tokens to every question. - **How it works:** More chunks, reranking, bigger context — better quality up to a point, always more expensive. - **Example:** Cutting k from 20 to 5 after reranking — same answer, smaller bill. - **Catch:** A great answer that's too slow or too expensive isn't great in production. - **Related:** Pulls against **precision & recall** and **reranking**. ### RAG is one rung; the model stays a component > A good kitchen has a pantry, but the pantry isn't the cook — it's one well-organised input to someone who still has to make the dish. RAG sits on the ladder above a plain prompt and below tools and agents: reach for it when the model needs your facts, and keep it behind a clean interface like any dependency. Retrieval feeds the model; it doesn't replace the judgement around the model. The same discipline as everywhere — use the simplest rung that works, and keep the model swappable. _Diagram: Plain prompt → RAG → tools and agents. Climb only when the task forces you._ ### How the concepts connect > Two shifts in one kitchen: at night the prep — cut and shelve everything; by day the cooking — find, pick and serve. Everything in this course is one chain in two phases. **Offline (indexing)**: documents are cut into **chunks** with **metadata**, turned into **embeddings**, and stored in a **vector database** — and rebuilt when the documents change (**index freshness**). **At query time**: **hybrid search** finds candidates, **reranking** keeps the best, and **precision and recall** decide how many to take. Then **grounding**: the model answers **from the context only**, with **citations** or an honest **refusal**. Off to the side, **paste first**: if the facts are few, you don't need the chain at all. And the **evals** — **retrieval eval**, **faithfulness**, the **golden set** — measure every link. _Diagram: Top: indexing. Middle: the path of a question. Bottom: the evals that measure the whole chain._ #### Before you ship a RAG feature - **Is RAG even needed** — or are the facts small and stable enough to paste into the prompt? - **How are documents chunked** — on meaning, self-contained, with metadata and a little overlap? - **Is retrieval hybrid and reranked**, or a raw vector search you never measured? - **Does the model answer only from retrieved text**, cite sources, and say "I don't know"? - **What's the eval** for retrieval, separate from generation? - **How does the index stay fresh**, and what does a query cost in time and tokens? #### Smell tests that retrieval is the problem - Answers are **confidently wrong**, and you never looked at the chunks that were retrieved. - **Raw vector search** only — no keyword fallback, no reranking. - Chunks split **by character count**, cutting through sentences and tables. - You're **tweaking the prompt and the model** to fix what is actually a retrieval miss. - The index hasn't been **rebuilt** since the source documents changed. #### Signs you built it well - Chunks are **semantic and self-contained**, tagged with metadata for filtering and citations. - Retrieval is **hybrid + reranked**, tuned to the fewest chunks that answer well. - Answers are **grounded and cited**, with an honest refusal when the facts aren't retrieved. - You measure **retrieval and generation separately**, and fixed retrieval first. - The index has a **freshness plan**, and RAG sits behind an interface you could swap. ## Cheat sheet: all 22 concepts ### Basics - **Knowledge Cutoff** (training cutoff): The model only knows training data up to one date — not your documents. - **RAG** (Retrieval-Augmented Generation): Retrieve the relevant chunks, put them in the context, answer from them. - **Grounding** (answer from sources): The answer rests on retrieved text, and you can check it. ### Chunking - **Chunk** (passage): A passage a few hundred words long — the unit of search; size is tuned by measuring. - **Structure-Aware Chunking** (split on structure): Split on headings and paragraphs, with a small overlap. - **Metadata** (chunk labels): Source, date, section, permissions — for filtering and citations. ### Embeddings and search - **Embedding** (vector): A vector — the coordinates of a text's meaning. - **Vector Search** (semantic search): Find the chunks nearest to the question in meaning. - **Vector Database** (ANN index): Stores vectors and finds the nearest in milliseconds. - **Similar ≠ Relevant** (similarity vs relevance): A chunk close in meaning doesn't necessarily hold the answer. ### Retrieval quality - **Hybrid Search** (keyword + vector): Search by words and by meaning together, then merge the results. - **Reranking** (two-stage retrieval): Gather candidates broadly, then carefully pick the best few. - **Precision & Recall** (top-k): Too few chunks — the answer is missed; too many — it drowns. Measure k. ### Grounding - **Context-Only Answering** (answer from context): Answer only from the retrieved chunks, never from the model's memory. - **Citations** (source attribution): Each claim points to its chunk and can be checked. - **Refusal** (“I don't know”): No answer in the chunks — say so honestly instead of inventing. ### Evaluation - **Retrieval Eval** (recall@k): Did the right chunks come back in the top k? Fix this first. - **Faithfulness** (groundedness): Every claim in the answer is backed by the chunks. - **Golden Set** (eval set): Real questions with expected chunks and answers — run on every change. ### Building it well - **Paste First** (context stuffing): A few stable facts — use the prompt, not RAG. - **Index Freshness** (reindexing): Docs changed — reindex, or the answers go stale. - **Cost & Latency** (time and tokens): Every stage costs time and tokens; take the smallest k that answers well. > RAG is a search system with a model on the end. Get the chunks right and the answer follows — the model was rarely the problem. End of express course · 7 chapters · 22 concepts Next comes practice: build the smallest real RAG over a handful of your own documents — chunk on structure, embed, search, and have the model answer with citations and an honest "I don't know." Then, before you tune anything, write ten real questions and check what gets retrieved. But hold one idea above the rest: when the answer is wrong, look at the chunks first. In RAG, the model is rarely the problem — your retrieval is, and that's the half you control.