Production RAG Systems
Retrieval-augmented generation is easy to demo and hard to productionize. The pipeline below plus a few design decisions — hybrid retrieval, reranking, grounding, and evaluation — are what separate a party trick from a system you can trust.
Eight stages, ingest to answer
The amber stage is the one most demos skip — moderation, grounding checks, and evaluation. It's what makes RAG safe to ship.
Source & ingest
Pull docs from stores, wikis, tickets, PDFs; normalize to text with metadata (source, section, timestamp, ACL).
Chunk
Split on structure (headings, sentences) with overlap; keep chunks small enough to retrieve, large enough to answer.
Embed
Encode chunks with an embedding model; store vectors plus metadata for filtering and citation.
Index
Upsert into a vector DB (HNSW/IVF) with namespaces per tenant; add a keyword index for hybrid search.
Retrieve
Hybrid dense + sparse search with metadata filters; fetch top-k candidates for the query.
Rerank
A cross-encoder reorders candidates by true relevance; drop low-score chunks before they reach the model.
Generate + cite
Prompt the LLM with the ranked context; require citations back to chunk IDs so answers stay grounded.
Guard & evaluate
Moderation, grounding checks, and an eval suite scoring faithfulness, relevance, and answerable-rate.
Where quality is won or lost
Chunking strategy
Structure-aware splitting beats fixed windows. Tune size and overlap per corpus; store section headers with each chunk for context and citations.
Embedding model
Match the model to your domain and language. Re-embed on model changes — mixing embedding spaces silently wrecks recall.
Hybrid retrieval
Dense vectors miss exact terms (SKUs, error codes); combine with BM25/keyword and fuse scores. Filter by metadata before ranking.
Reranking
A cross-encoder reranker on the top-k is the highest-ROI quality lever — it fixes "retrieved but wrong order" far more cheaply than a bigger LLM.
Grounding & citations
Force the model to cite retrieved chunk IDs. No citation → treat as unsupported. This is how you make hallucinations visible.
Evaluation harness
Track retrieval (recall@k, MRR) and generation (faithfulness, answer relevance) on a fixed eval set, run on every change like regression tests.
Symptom → root cause
Confident but wrong answers
The right chunk was never retrieved. Fix retrieval (hybrid + filters + rerank) before touching the prompt.
Answers ignore the context
Context is too long or low-signal. Rerank and trim; put the strongest chunks first.
Stale or leaked data
No re-ingestion or missing ACL metadata. Version the index and filter by tenant/permission at query time.
Great demo, bad production
No eval set. Add offline evals and online feedback before scaling the corpus or the audience.
Two RAG demos you can run now
These live demos put the pipeline to work — one on semantic product search, one on a grounded support chatbot with citations.