AI Engineering
Chapter 37
RAG
Picture a support engineer asking: “What is our refund window for enterprise SKUs?” The base model may never have seen your policy PDF. Fine-tuning every week will not scale. Retrieval-Augmented Generation tells a better story: keep the model general, and at question time fetch the paragraphs that matter, then ask it to answer from those paragraphs.
RAG is not a vector-database demo. It is a production pipeline with failure modes, latency budgets, and evaluation.
Two phases, one promise
Indexing happens ahead of time. Documents are split into chunks, each chunk is embedded into a vector, and vectors live in an index with raw text and metadata. Querying happens per question: embed the question, retrieve top-k chunks, optionally rerank, pack them into the prompt, and generate an answer that cites chunk ids.
The promise to users: if the answer is in our corpus, we will find it and show where it came from; if not, we will say we do not know.
Chunking is product design
Too small and you lose context; too large and retrieval gets noisy and expensive. Start around 300–800 tokens with modest overlap. Prefer splitting on headings and paragraphs. Keep metadata: title, URL, updated_at, ACL tags — topical similarity is useless if the user cannot read that document.
| Choice | Trade-off | Interview cue |
|---|---|---|
| Fixed-size chunks | Simple; can bisect sentences | Baseline to beat |
| Structure-aware | Respects headings and code blocks | Better for manuals and repos |
| Parent-child | Retrieve small, expand to parent | Precision plus context |
| Contextual embeddings | Adds document context into vectors | When titles are ambiguous |
Hybrid retrieval beats pure vectors
Keyword search still wins on exact IDs, error codes, and rare proper nouns. Mature systems combine sparse (BM25) and dense retrieval, then fuse scores. A cross-encoder reranker can reorder the top 50 into a sharper top 5 when quality matters more than a few extra milliseconds.
def answer_with_rag(question, store, embed, generate):
q = embed(question)
dense = store.similarity_search(q, k=20)
sparse = store.bm25_search(question, k=20)
fused = reciprocal_rank_fuse(dense, sparse)[:8]
ranked = rerank(question, fused)[:4]
context = "\n\n".join(f"[{c.id}] {c.title}\n{c.text}" for c in ranked)
prompt = f"""Answer using only the sources.
If sources are insufficient, say you do not know.
Cite sources as [id].
Sources:
{context}
Question: {question}
"""
return generate(prompt)
The generation contract
Spell the rules: use only provided sources; cite ids; refuse when unsupported. Delimit sources so prompt injection from a malicious PDF is harder. Log which chunk ids were shown — that audit trail matters when someone asks why the assistant said what it said.
Where RAG fails
- Retrieval miss — the right chunk never entered top-k. Fix chunking, hybrid search, query rewriting.
- Retrieval noise — junk chunks distract the model. Fix reranking and tighter instructions.
- Stale index — docs changed; embeddings did not. Fix incremental indexing and freshness metadata.
- ACL bugs — retrieved a doc the user cannot see. Filter by permissions before prompting.
- Faithfulness errors — model ignored sources. Add citations, lower temperature, answerability checks.
Evaluate retrieval and answers separately
Measure recall@k on labeled question→doc pairs. Separately score answer faithfulness and helpfulness. If you only score final answers, you will not know whether to fix the index or the prompt.
RAG is how private knowledge enters the story without retraining the protagonist. Get retrieval right and generation becomes almost easy; get it wrong and no prompt poetry will save you.
Query rewriting and multi-hop
Users ask vague questions. A cheap first model call can rewrite “that billing thing from last mail” into a search query with account ids and dates pulled from memory. Multi-hop RAG answers questions that need two documents by retrieving, thinking, then retrieving again. Keep hops bounded.
Citations users can click
Return chunk ids that map to URLs or doc anchors in your UI. If the model cites [12] but [12] was not in the prompt, treat that as a failure in validation. Grounding is a product feature, not a vibe.
Interview drill — RAG
Practice the full whiteboard: ingest vs query, ACL in retrieval, abstain, eval.
More drills in the Interview Lab.
Q1. Company knowledge chatbot (full design)
Answer from private docs with citations and permissions.
- Clarify ACL, freshness, latency, languages.
- Draw ingest (chunk→embed→index) separate from query.
- Hybrid retrieve + ACL filter + rerank + grounded generate.
- Abstain on low confidence; log traces for eval.
Full solution with prompt skeleton and failure modes: AI Lab Q1.
Q2. Chunking strategy deep-dive
How do you choose chunk size and boundaries?
- Start 400–800 tokens with overlap; split on headings.
- Keep tables/code blocks intact when possible.
- Evaluate recall@k on a gold set — do not bike-shed forever.
- Store metadata (title, url, updated_at, acl) with every chunk.
Q3. ACL-aware retrieval
User must not see docs they cannot access.
Enforce ACL inside the retriever query (metadata filter / partition). Post-hoc prompt instructions are not a control. Index permission tags with vectors; test with adversarial users in eval.
Q4. Hybrid search
When do keywords beat embeddings?
IDs, error codes, SKUs, rare proper nouns → BM25. Paraphrases → dense. Fuse (RRF) then rerank. Lab Q2.
Q5. Measuring RAG quality
What do you measure before shipping a retriever change?
Retrieval recall@k on gold questions, groundedness of answers, citation accuracy, latency, cost. Separate retrieve vs generate failures. Lab Q3.