RAG

slide 1

RAG

Retrieval-Augmented Generation: answering from documents the model has never seen

slide 2

What is RAG?

RAG: look it up first, then answer from what you found.

• Retrieval: search a knowledge source for the passages relevant to the question
• Augmented: add those passages to the prompt, beside the question
• Generation: the model writes its answer from what it was just given
• Named in a 2020 research paper. Its lead author, later:

"We definitely would have put more thought into the name"

slide 3

Two chatbots, two invented policies

Air Canada, 2024

• Its website chatbot told a grieving passenger that a bereavement fare could be claimed after flying
• The airline's own policy said the opposite
• In effect, Air Canada argued its chatbot was "a separate legal entity". The tribunal called that "remarkable" and made it pay

Cursor, 2025

• Users kept being logged out when they switched computers

• The support bot, "Sam", explained a one-device policy: "a core security feature"

• There was no such policy; it was a bug. Users cancelled, and a co-founder apologised

• How do you make a model answer from your documents, not from its imagination?

slide 4

Agenda

Part 1: How RAG works

• Why models need it; web RAG and private RAG
• The pipeline: load, chunk, embed, store, search, generate
• Grounding and citations
• What RAG fixes, and what it costs

Part 2: RAG in 2026

• Million-token context windows: when to skip RAG
• When retrieval goes wrong
• Better retrieval: hybrid, rerank, graph, agentic
• How coding assistants find your code
• Try it now: ground an answer by hand

slide 5

Part 1: How RAG works

From a pile of documents to a grounded answer

slide 6

Why a model needs RAG

• Its knowledge stops at its training cutoff
• It has never seen your documents: your code, your wiki, your contracts
• Asked anyway, it writes the most plausible answer, which may be invented
• Retraining to add facts is slow and costly, and out of date by the next change
• So don't teach the model your documents: find the relevant piece and put it in the prompt

slide 7

Two kinds of RAG

Web RAG

• Retrieves from search engines and public websites, live
• Broad and current; you don't control the sources
• The search engine does the retrieving
• Perplexity, ChatGPT search, Google's AI Overviews

Private RAG ("traditional RAG")

• Retrieves from your own documents: wikis, PDFs, tickets, code

• Narrow and controlled: only what you put in

• Usually a vector database you build and maintain

• NotebookLM, company help bots, Claude and ChatGPT Projects

• Same recipe either way: fetch first, then answer from what was fetched

slide 8

The workflow, phase 1: indexing

Done once, ahead of time, and again whenever the documents change:

1. Load the documents
2. Split into chunks
3. Embed each chunk
4. Store in a vector database

• The result is a searchable index: every chunk's vector, its text and where it came from

slide 9

The workflow, phase 2: query

Done for every question a user asks:

1. Embed the question
2. Find the nearest chunks
3. Add them to the prompt
4. Generate the answer

• Only the last step uses the language model. Everything before it is search

slide 10

Step 1: Load the documents

• Sources: PDFs, Word files, web pages, wikis, code, database rows
• Extract plain text: pypdf, pdfplumber, BeautifulSoup, docx2txt
• Clean it: headers, footers, page numbers, mangled tables
• Record where every piece came from (file, page, section): citations need it later

slide 11

Step 2: Chunk

• Split each document into pieces of a few hundred words
• Why: an embedding sums up a whole piece of text, so one topic per piece keeps it sharp
• Small pieces also mean several fit in one prompt
• Overlap: neighbouring chunks share a few sentences, so a fact on a boundary survives whole
• Strategies: fixed size, by sentence, by paragraph or heading, by topic shift, by function for code

slide 12

Overlap, drawn

1. chunk 1: words 1 to 200
2. chunk 2: words 161 to 360
3. chunk 3: words 321 to 520

• Each chunk starts 40 words before the previous one ended
• A sentence straddling word 200 is cut in chunk 1 and whole in chunk 2
• The price: those 40 words are embedded and stored twice

slide 13

Predict: which chunk size retrieves best?

You split a 40-page manual into chunks. Which size works best?

• One sentence per chunk: as precise as it gets
• One paragraph per chunk
• One page per chunk
• The whole manual as one chunk

slide 14

A paragraph, usually. Here's why

• Too small: "It must be replaced every 12 months." Replaced what? The chunk has lost its subject
• Too large: the one relevant sentence arrives inside a page of other topics
• Measured: "what is a variable" scores 0.80 against the one-sentence definition, 0.42 against the 200-word chunk that holds it
• The tempting answer, one sentence, is easy to find and useless to read
• There is no universal size: measure it on questions whose answers you know

slide 15

Step 3: Embed

• An embedding is a list of numbers that captures what a piece of text means
• Texts with similar meanings get similar lists, whatever words they use
• One chunk, one vector: 384 numbers for all-MiniLM-L6-v2, up to 3,072 for larger models
• Use the same model for the chunks and the questions. Always
• In 2026: OpenAI text-embedding-3, Gemini Embedding, Qwen3-Embedding (open), all-MiniLM-L6-v2 (small, free, runs on a laptop)

slide 16

What an embedding looks like

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")
vector = model.encode("what is a variable")

print(len(vector))    # 384
print(vector[:5])     # 0.057  0.04  -0.052  0.08  -0.1

• No single number means anything on its own. The whole list is the meaning
• The model learned to produce these lists from huge amounts of text

slide 17

Similar meaning, similar direction

Measured with all-MiniLM-L6-v2:

• Cosine similarity compares the directions of two vectors: 1 is the same direction, 0 is unrelated
• "A cat sits on the mat." and "A kitten is lying on the rug.": 0.56
• "A cat sits on the mat." and "The car engine is running fast.": 0.03
• "what is a variable" and "A variable is like a labeled box that stores information.": 0.80
• "what is a variable" and a sentence about bubble sort: −0.09

slide 18

Step 4: Store in a vector database

• Keeps each chunk's vector, its text and its label (file, page) together
• Finds the nearest vectors fast, even among millions, with approximate search (e.g. HNSW)
• Popular: Pinecone (managed), Weaviate, Qdrant, Milvus, Chroma, pgvector (inside PostgreSQL)
• FAISS: a fast similarity-search library rather than a full database
• Metadata filters narrow the search: one product, after a date, only what this user may see

slide 19

Step 5: Search

• The question is embedded with the same model, then compared with every chunk
• The database returns the k closest chunks (here k = 3), each with a score

Query: "what is a variable"
  0.42  introduction_to_programming.txt
  0.26  introduction_to_programming.txt
  0.21  database_fundamentals.txt

• Five short programming documents, cut into 14 chunks. The definition sits in the top chunk

slide 20

Predict: no words in common

how can my code remember a number for later shares no words, apart from "a", with the sentence that answers it: A variable is like a labeled box that stores information. What does the search return?

• Nothing useful: no words match
• Three random chunks
• The same top chunk as "what is a variable"
• An error

slide 21

Same three chunks, in the same order

Query: "how can my code remember a number for later"
  0.31  introduction_to_programming.txt
  0.25  introduction_to_programming.txt
  0.13  database_fundamentals.txt

• Keyword search would find nothing here. Matching meaning, not words, is why embeddings exist
• The common wrong answer, "no shared words, so no match", is how keyword search thinks
• The scores are lower (0.31 against 0.42), but the ranking holds

slide 22

Step 6: Generate from an augmented prompt

What the model actually receives:

Answer the question using ONLY the context below.
If the context does not contain the answer, reply exactly:
I don't know - the provided context does not cover this.
Finish your answer with the document you used.

CONTEXT:
[source: introduction_to_programming.txt]
Introduction to Programming Programming is the process...

QUESTION: What is a variable?

• Instructions, then the retrieved chunks with their labels, then the question

slide 23

Grounding and citations

• Grounded: the answer comes from text you supplied, not from what the model half-remembers
• Each chunk carries a label from the moment it is cut, and the label travels into the prompt
• The model repeats the label it used: it has no other way to know a source

A variable is like a labeled box that stores information
[source: introduction_to_programming.txt].

• A citation with no label behind it is an invention

slide 24

Predict: does RAG stop hallucination?

The answer now comes from real documents, with a citation. Does that eliminate made-up answers?

• Yes: it only uses the retrieved text
• It reduces them a lot, but not to zero
• It makes no difference
• It makes them worse

slide 25

It reduces hallucination. It doesn't remove it

• Stanford, 2024: legal research tools built on RAG, sold as hallucination-free, still made things up on 17–33% of 202 questions
  • Mid-2024 versions, and the vendors disputed the method: read it as a direction, not a precise rate
• The model can misread a chunk, or blend two chunks into a claim neither makes
• A real citation that doesn't support the claim is worse than none: it looks checked
• When retrieval finds nothing useful, the model may answer from memory anyway
• The common wrong answer, "yes, it's grounded now", forgets all three

slide 26

What RAG fixes, and what it costs

Benefits

• Current answers without retraining the model
• Answers grounded in your own documents
• Citations a reader can check
• Far cheaper than fine-tuning just to add facts
• Access control: retrieve only what the asker may see

Drawbacks

• Latency: a search before every answer
• Maintenance: parsing, chunking, embeddings and an index to keep current
• Retrieval is the ceiling: wrong chunks in, wrong answer out
• Chunking can split one fact across two pieces
• Only a few chunks reach the model: what isn't retrieved might as well not exist

slide 27

RAG or fine-tuning?

RAG changes what it knows

• Facts, documents, this morning's numbers
• Updated by re-indexing, in minutes
• Can show its source

Fine-tuning changes how it behaves

• Style, format, a specialised skill

• Updated by training again

• Can't point to where it learned something

• New facts: RAG. New behaviour: fine-tuning. Many products use both

slide 28

Part 2: RAG in 2026

Models now read a million tokens at once. Do we still need any of this?

slide 29

"RAG is dead"?

• 2020: RAG was named when models could read a few thousand tokens at a time
• 2026: frontier models read about a million tokens at once, well over a thousand pages
• So for a small document set, the best design is often: paste the whole thing in
• Claude Projects works this way: everything goes in while it fits; past that, RAG switches on and holds up to 10x more
• Long context didn't kill retrieval. It killed retrieval as the automatic choice

slide 30

Predict: paste it, or build a pipeline?

Five short documents, about 2,000 words in total. Users ask open questions about them. What do you build?

• A chunk, embed and search pipeline with a vector database
• Nothing: paste all five into every prompt
• Fine-tune a model on them
• Summarise each one, then throw the originals away

slide 31

Paste it. Retrieval earns its place at scale

Paste it all (long context)

• It fits comfortably in the window
• Questions need the whole thing, not one fact
• It changes rarely, and few people ask
• No index to build, nothing to go stale

Retrieve (RAG)

• Too big to fit: thousands of documents

• You must cite exactly which source said it

• The data changes every day

• Cost: every question would resend everything

• Different users may see different documents

• The tempting answer, the pipeline, is a lot of machinery for 2,000 words a model can simply read

slide 32

Long context isn't free either

• Context rot: Chroma tested 18 models in 2025; every one got less reliable as its input grew, even on simple tasks
• In one test, a 300-token prompt holding just the relevant part beat the full 113,000-token history, for every model
  • Chroma sells a vector database, so read these as one vendor's measurements; its test code is public
• Lost in the middle: facts buried mid-prompt are used less reliably than facts at either end
• Every token is paid for, and waited for, on every question
• More tokens is not more understanding

slide 33

The 2026 default: retrieve, then read

1. Retrieve generously, cut by a threshold
2. Widen each hit to its section
3. One long-context read of the whole set
4. Answer, citing the labels

• Retrieval finds the pieces; the long read joins them up
• Bounded: cost stays proportional and every citation stays checkable
• This is context engineering: deciding exactly what the model reads

slide 34

Predict: nothing in the corpus matches

The same five programming documents. Someone searches how do I bake sourdough, with k = 3. What comes back?

• An empty result
• An error: no relevant chunks
• Three chunks anyway
• One chunk, flagged as a weak match

slide 35

There is always a nearest neighbour

Query: "how do I bake sourdough"
  0.09  algorithms_overview.txt
  0.09  web_development_intro.txt
  0.04  introduction_to_programming.txt

• Search ranks every chunk and returns the top k, always. "Relevant" is not a concept it has
• The score is the only signal: a 0.20 threshold drops all three, and keeps the best chunks for both variable questions
• The common wrong answer, "an empty result", assumes search knows when to stop

slide 36

The space belongs to the model that made it

Index: all-MiniLM-L6-v2   Question: paraphrase-MiniLM-L3-v2
Query: "what is a variable"
  0.10  web_development_intro.txt
  0.08  database_fundamentals.txt
  0.07  web_development_intro.txt

• No error: both models make 384 numbers, so the sums run. The right chunk isn't even in the top three
• It can half-work by luck (a linked-list question still found its chunks), so the bug hides
• Google's own docs: gemini-embedding-001 and gemini-embedding-2 are incompatible, so upgrading means re-embedding everything

slide 37

Predict: which question defeats top-3?

Same five documents, k = 3. Which question can top-3 retrieval not answer properly?

• What is a variable?
• How can my code remember a number for later?
• Which topics do these five documents cover?
• How do I bake sourdough?

slide 38

Local questions and global questions

Query: "Which topics do these five documents cover?"
  0.32  database_fundamentals.txt
  0.18  web_development_intro.txt
  0.17  algorithms_overview.txt

• 3 chunks from 3 documents: the other 2 are never seen
• From those chunks, 6 of 9 replies said "I don't know" and 3 listed only the three documents they saw. From all five, 9 of 9 named every topic
• Top-k suits local questions, whose answer lives in one place. A global question needs the whole corpus

slide 39

Grounded in garbage

• May 2024: asked how to stop cheese sliding off pizza, Google's AI Overview suggested adding "1/8 cup of non-toxic glue to the sauce"
• The source: an 11-year-old joke comment on Reddit
• Retrieval worked. The source was real. It was a joke
• RAG makes a model as reliable as what it retrieves, and no more

slide 40

Retrieved text is untrusted input

• Whatever lands in the prompt can steer the model, including instructions hidden in a document
• June 2025, "EchoLeak": one email, never even opened, was pulled into Microsoft 365 Copilot's context, and hidden instructions in it made Copilot leak internal data
  • Rated critical (9.3 out of 10), and patched before it was disclosed
• Poisoning: plant crafted documents, and the system retrieves them on cue
• So filter what gets indexed, respect permissions, and never let retrieved text trigger actions unchecked

slide 41

Better retrieval, one layer at a time

• Agentic RAG: the model decides what to search, and searches again · most capable
• Graph RAG: relationships between facts, for global questions
• Rerank: a sharper model reorders the candidates
• Hybrid search: meaning, keywords and filters together
• Naive RAG: embed, take the top k, prompt · simplest

• Each layer costs more, and fixes a failure of the one below it

slide 42

Hybrid search: meaning plus keywords

Embeddings win

• "how do I quit" finds "resignation process"
• The wording varies; the idea doesn't

Keywords win

• ERR_4021, max_attempts, v3.2.1

• Names, codes, file paths: exact strings

• An embedding blurs the exact spelling

• Hybrid retrieval runs both (keyword search such as BM25, plus vectors), merges the two rankings, and adds metadata filters

slide 43

Rerank, and fix the chunks themselves

• Rerank: fetch 50 candidates fast, have a slower, sharper model score each one against the question, keep the best 5
• Contextual retrieval: before embedding, a model writes a line of context onto each chunk
  • "The company's revenue grew by 3% over the previous quarter." Which company? Which quarter? The added line says
• Anthropic, 2024: context plus keyword search cut failed retrievals by 49%; adding a reranker, by 67%
  • Its own benchmark, and relative drops: from 5.7% of searches missing the answer to 1.9%

slide 44

Graph RAG and multimodal RAG

Graph RAG

• A model extracts entities and relationships into a graph, and summarises clusters ahead of time
• Answers global questions like "What are the main themes?" that top-k can't (Microsoft, 2024)
• Expensive to build; worth it for large, connected corpora

Multimodal RAG

• Retrieves images, tables, slides, audio and video, not only text
• 2026: one embedding model can put text, images, video, audio and PDFs in one space (Gemini Embedding 2)
• For when the answer is in a chart, not a sentence

slide 45

Agentic RAG: the model does the searching

1. Read the question
2. Choose a source and a query
3. Search and read
4. Enough? If not, search again
5. Answer with sources

• Router: picks the right source for each question: documents, web or database
• Multi-agent: specialists search different sources and pool what they find
• Deep-research tools are this at scale: dozens of searches for one question
• More calls and cost per question; much better on hard, multi-step questions

slide 46

Claude Code greps; Cursor embeds

Claude Code: agentic search

• No index: the model runs grep and glob, reads files, and repeats
• Its creator, 2026: early versions used RAG and a local vector database, but "agentic search generally works better"
• Simpler: no index to go stale, no copy of your code elsewhere

Cursor: semantic search plus grep

• Splits your repo into chunks by syntax and embeds them

• Its measurement, 2025: adding semantic search answered codebase questions 12.5% more accurately on average (6.5–23.5% by model)

• Its conclusion: grep and embeddings together beat either alone

• Who's right? Each company measured its own product. Same problem, two answers: finding the right context is the whole game

slide 47

Worked case: where is the retry limit?

Prompt: Where is the retry limit for outgoing HTTP requests configured?

1. Search the files for the word "retry"
2. Read the file that matched
3. Answer, citing the file and line

• No index, no embeddings: retrieval is a loop the agent runs at question time

slide 48

When the word isn't there

# net/http_client.py
MAX_ATTEMPTS = 5
BACKOFF_SECONDS = 0.5

• grep for "retry" finds nothing, so a naive agent reports there is no retry limit: confidently wrong
• A good agent treats the empty result as a clue, and tries "attempt" and "backoff"
• On an 8-file toy repo, an embedding search ranked net/http_client.py first (0.53; the runner-up 0.08): the gap an index fills

slide 49

Managed RAG: the pipeline as a service

• Gemini API File Search (Nov 2025): upload files; Google chunks, embeds, retrieves and cites
• OpenAI file search, Claude Projects, NotebookLM: the same pipeline, out of sight
• You still decide what goes in, how it is labelled, and how answers get checked
• When it answers wrongly, knowing the six steps is how you find which one failed

slide 50

Try it now: ground an answer by hand

Paste this into any chatbot:

Answer ONLY from the context below. If the context does
not contain the answer, reply exactly: I don't know.

Context: Loans last 21 days. An item can be renewed twice
unless another member has reserved it. A lost item is
charged at its replacement cost plus a handling fee.

Q1: How many times can I renew a loan?
Q2: What is the handling fee for a lost item?

• Look closely at Q2: the context mentions a handling fee but never says how much
• Then delete the sentence starting "If the context does not contain..." and ask again

slide 51

What the near miss shows

• Retrieval would have found the right chunk: the fee sentence is the closest text there is
• That chunk still doesn't hold the answer: found and answerable are different things
• We ran it 5 times on each of 3 models (October 2026): not one of 30 replies invented a fee
• With the permission line, 15 of 15 said exactly "I don't know"; without it, 13 different wordings: fine for a reader, hard for code to check
• If your model did invent a fee, you saw the failure live. Either way, test near misses, not just off-topic questions

slide 52

Common mistakes

Building it

• A pipeline for a corpus that would fit in the prompt
• A chunk size chosen once and never measured
• Questions embedded with a different model from the chunks
• A threshold written without checking distance versus similarity
• Chunks with no labels, so no citations

Running it

• Blaming the model before looking at what was retrieved
• A stale index: right answers to last month's question
• A global question asked of a top-k system
• Testing the off-topic question but never the near miss
• No set of questions with known answers to test against

slide 53

Summary

• RAG: retrieve the relevant passages, put them in the prompt, generate from them
• Index once (load, chunk, embed, store); for each question, embed it, search, generate
• Embeddings match meaning, not words; use one model throughout; there is always a nearest neighbour
• Grounding and citations reduce made-up answers; they don't remove them
• 2026: if it fits, paste it; if not, retrieve then read, with hybrid search, reranking or an agent doing the searching