RAG — Retrieval-Augmented Generation

Chapter 5 · Phase 4 · practice in Week 7 and Week 8

What you will know after this chapter

Why RAG exists and what problem it solves · the full 5-stage pipeline (chunk → embed → index → retrieve → generate) · chunking strategies and size tradeoffs · how vector search works · hybrid search · what causes RAG to fail · how to evaluate a RAG system

1. Why RAG exists

LLMs have two hard constraints that prevent them from answering questions about your internal data:

Context limits — a 200K token window fits ~450 pages. Your documentation, codebase, or knowledge base is probably larger.
Training cutoff — the model's weights encode knowledge up to a training date. Anything newer or proprietary is unknown to the model.

RAG solves both by retrieving relevant information at query time and injecting it into the prompt. The model's role changes from "remember everything" to "reason over what I'm given."

2. The full RAG pipeline

RAG — complete architecture

Indexing (offline — run once or periodically)

Raw docs
PDFs, pages, tickets

→

Chunker
splits into segments

→

Embedding model
text → vector

→

Vector DB
pgvector, Pinecone, Weaviate

Query (online — every user request)

User question

→

Embedding model
same model as indexing

→

Vector search
top-K by cosine sim

→

Reranker
optional

Retrieved chunks

User question

→

LLM generates answer
with citations

→

Response to user

3. Stage 1 — Chunking

A chunk is the unit of retrieval. You never retrieve an entire document — you retrieve the most relevant segment. Chunk strategy is one of the highest-impact tuning levers in a RAG system.

Chunk size

How many tokens each chunk contains. Typical range: 256–1024 tokens. Too small = each chunk lacks context. Too large = irrelevant content dilutes the relevant signal.

Chunk overlap

How many tokens at the boundary are shared between consecutive chunks. Overlap of 50–100 tokens prevents a sentence that spans a boundary from being split across two chunks.

Chunking strategies

Strategy	How it works	Best for	Weakness
Fixed-size	Split every N tokens with overlap	General purpose, fast setup	Splits mid-sentence; ignores document structure
Sentence / paragraph	Split on natural language boundaries	Prose documents, articles	Uneven chunk sizes; some chunks are too short
Semantic	Embed and split where cosine similarity drops	Long documents with topic shifts	Slow, requires a second embedding pass
Hierarchical	Parent chunk (section) + child chunks (paragraphs)	Structured docs, code bases, legal	More complex retrieval and deduplication
Document-aware	Respect section headers, code blocks, tables	Technical docs, APIs, manuals	Requires custom parsing per document type

PM decision

Start with fixed-size chunks (512 tokens, 50 token overlap). Only move to more complex strategies after measuring retrieval quality. The effort-return curve flattens quickly.

4. Stage 2 — Embedding and indexing

Each chunk is passed through an embedding model to produce a dense vector. This is the representation used for similarity search.

How vector search works

Cosine similarity — the core math

query = "How do I reset my password?" → embed → q_vec
chunk_A = "Account recovery: click Forgot Password..." → a_vec
chunk_B = "Billing: update payment method..." → b_vec

cosine_sim(q_vec, a_vec) = 0.91 ✓ HIGH — return this
cosine_sim(q_vec, b_vec) = 0.23 ✗ LOW — skip

Top-K = [chunk_A, ...] → inject into prompt

Vector database options

Option	Notes	Good when
pgvector	Postgres extension. SQL + vectors in one DB.	You already use Postgres; corpus < 10M vectors
Pinecone	Managed cloud. Fast setup, serverless tier.	Fast iteration, no infra management
Weaviate	Open source, supports multi-modal, hybrid search built-in.	Hybrid search, self-hosted required
Qdrant	Open source, Rust-based, high performance.	High throughput, self-hosted
Chroma	Simple Python-first, good for prototypes.	Local development, quick demos

5. Stage 3 — Retrieval and reranking

Top-K retrieval

The query vector is compared to all indexed vectors. The K most similar chunks (typically K=3–10) are returned. These are injected into the prompt as context.

Hybrid search — keyword + semantic

Pure semantic search misses exact matches (product names, error codes, IDs). Hybrid search combines BM25 keyword matching with semantic similarity and merges results.

Hybrid search merge

Query

→

BM25 keyword results

Semantic vector results

→

RRF merge / reranker
Reciprocal Rank Fusion

→

Top-K final chunks

Reranking

A cross-encoder reranker takes (query, chunk) pairs and scores them with higher accuracy than the bi-encoder used for fast retrieval. It is slower but more precise — run on the top-20 candidates to reorder before taking the top-5.

6. Stage 4 — Generation with context

The retrieved chunks are injected into the prompt with explicit instructions to use only the provided context.

SYSTEM: You are a support assistant. Answer ONLY from the provided context.
If the answer is not in the context, say "I couldn't find that in our docs."
Always cite the source article ID.

CONTEXT:
[Article KB-0042] To export to CSV: navigate to Settings → Data → Export...
[Article KB-0088] Billing changes take effect at the next billing cycle...

USER: How do I export my usage data?

7. RAG failure modes

The most common RAG failures

Wrong chunk retrieved — the most similar chunk is not the most relevant one. Usually a chunking or embedding model problem.
Answer not in corpus — the document exists but was not indexed (wrong file format, crawl missed it, access control excluded it).
Stale index — document updated but not re-indexed. User gets outdated information.
Chunk size mismatch — chunk is too small to contain the full answer (the answer spans multiple chunks). Solution: increase chunk size or use hierarchical retrieval.
Hallucination despite context — model ignores the retrieved context and generates from training weights anyway. Mitigate with stronger system prompt constraints and few-shot examples of citation behavior.
Lost in the middle — when many chunks are retrieved, model quality degrades for content in the middle of the context. Limit K or put most important chunks first/last.

8. How to evaluate a RAG system

Metric	What it measures	How
Retrieval recall	Was the answer-containing chunk in the top-K results?	Compare retrieved IDs to ground-truth IDs for a test set
Answer faithfulness	Is the answer grounded in the retrieved context? (no hallucination)	LLM-as-judge: "Is every claim in the answer supported by the context?"
Answer relevance	Does the answer actually address the question?	LLM-as-judge: "Does this answer the question? Score 1–5."
Context precision	Are all retrieved chunks relevant? (no noise)	LLM-as-judge per chunk: "Is this chunk relevant to the question?"

PM evaluation workflow

1. Collect 50–100 real user questions with known correct answers.
2. Run the RAG pipeline on each.
3. Score retrieval recall (did we get the right chunk?) and answer faithfulness (did the LLM use the context?).
4. Iterate on chunking, embedding model, or prompt until both metrics exceed 85%.

← Visualization

Practice → Week 7 Next: Chatbots →

Test your understanding

Describe the two main pipelines in a RAG system.v

Indexing (offline): Documents -> chunker -> embedding model -> vector database. Query (online): User query -> embedding model -> vector similarity search -> top-K chunks -> LLM prompt -> response. Indexing quality determines retrieval quality; retrieval quality determines answer quality.

What is a chunk, and why does chunk size matter?v

A chunk is a segment of a document (e.g. 256-512 tokens). Too large: irrelevant content wastes context window space. Too small: the chunk lacks context to answer the question. Most production systems use 256-512 token chunks with overlap.

What is the difference between semantic search and keyword search? When does each fail?v

Semantic search finds conceptually similar content even with different phrasing. It fails on specific identifiers (product codes, acronyms). Keyword search is precise on exact terms but misses synonyms. Hybrid retrieval combines both.

What is reranking and why add it to a RAG pipeline?v

Reranking is a second-pass scoring step: after vector retrieval, a cross-encoder re-scores each (query, chunk) pair for relevance. It is slower and more expensive but significantly more accurate than vector similarity alone.

A RAG system retrieves the wrong document. What are the two most common causes?v

(1) Poor chunking: the answer spans a chunk boundary; neither chunk alone is sufficient. Fix: add overlap. (2) Query-document vocabulary mismatch: 'invoice' vs 'bill'. Fix: query expansion or hybrid retrieval.

What is the 'lost-in-the-middle' problem and how do you mitigate it?v

LLMs use information at the beginning or end of a long context better than in the middle. Mitigation: sort retrieved chunks by descending relevance and put the highest-ranked chunk first in the prompt.

How would you evaluate retrieval quality independently of generation quality?v

Build a test set of (question, ideal source document) pairs. For each question, check if the correct chunk appears in the top-K retrieved results. Report Recall@K and MRR. Evaluating retrieval separately isolates whether a bad answer is a retrieval failure or a generation failure.