2026-10-06 · 12 min read
RAG in Production: A Reference Architecture
A practical blueprint for retrieval-augmented generation that survives real users: ingestion, chunking, hybrid retrieval, re-ranking, permissions, citations and evaluation.
A RAG demo takes an afternoon. A RAG system people trust takes deliberate engineering in the parts demos skip: ingestion, retrieval quality, permissions and evaluation. This is the reference architecture we start from.
1. Ingestion
Connect to sources (SharePoint, Confluence, Google Drive, databases, file shares) with incremental sync, not one-off exports.
Parse to clean text while keeping structure: headings, tables and page numbers matter for both retrieval and citations.
Store metadata with every document: source, owner, last updated, and the access-control list.
Re-index on change, and delete on delete — stale answers are a top cause of lost trust.
2. Chunking
Split by document structure (sections, then paragraphs) rather than a fixed character count, keep chunks to a few hundred tokens, and prepend the document title and section heading to each chunk so it makes sense on its own. Keep a pointer back to the exact location for citations.
3. Retrieval
Hybrid search: combine keyword (BM25) for exact terms, codes and names with vector search for meaning.
Filter by metadata before ranking — date, product, region and, critically, permissions.
Re-rank the top 20–50 candidates with a cross-encoder or LLM re-ranker and pass only the best handful to the model.
Rewrite vague or conversational questions into standalone search queries first.
4. Permissions
Enforce access at retrieval time using the source system's ACLs, synced with the content. Never rely on the model to withhold information it has already been given: if a user cannot open a document, its chunks must never enter their prompt.
5. Generation
Instruct the model to answer only from the provided sources and to say when they are insufficient.
Require citations for each claim and render them as links to the exact source passage.
Use structured output when the answer feeds another system.
Treat retrieved text as untrusted data — it can contain prompt-injection attempts.
6. Evaluation
Build a test set of 100+ real questions with known good sources before tuning anything. Measure retrieval (did the right passage appear in the top results?) separately from generation (was the answer correct, complete and grounded?). Run the suite on every change to chunking, prompts or models, and add every failure users report to it.
7. Operations
Trace every request: query, retrieved chunks, prompt, answer, latency and cost.
Collect thumbs-up/down with a reason, and review negatives weekly.
Watch for drift: new document types and vocabulary degrade retrieval quietly.
Default stack
PostgreSQL with pgvector for most corpora up to millions of chunks, a hosted embedding model, a re-ranker, a frontier model for generation with a smaller model for query rewriting, and an observability tool for tracing and evals. Move to a dedicated vector engine only when measured scale or latency requires it.