Home › Blog › How to Build Production RAG

How to Build Production RAG: Why Your Demo Works and Your Deployment Doesn't

Anyone can build a RAG demo in an afternoon: load a PDF, embed it, retrieve chunks, generate an answer, applause. Then it meets real users and real documents, and it confidently answers questions with the wrong policy version, misses the table that held the actual number, and hallucinates when retrieval comes back empty. This guide is the honest path from that demo to a system a customer can rely on, which is the exact skill Forward Deployed Engineers are paid to have.

The Pipeline, End to End

1. Ingestion. Pull documents from their real homes: wikis, PDFs, tickets, databases. Clean them, preserve structure like headings and tables, and capture metadata such as source, date and access level. Most production failures trace back to sloppy ingestion, not to the model.
2. Chunking. Split documents into retrievable pieces. Respect semantic boundaries, keep chunks self contained, and attach enough context, like the section title, that a chunk makes sense alone. Chunk size is a tuning decision, not a constant: dense policy text and long narrative want different treatment.
3. Embedding and indexing. Convert chunks to vectors with an embedding model and store them in a vector database such as FAISS, ChromaDB or Pinecone, alongside their metadata for filtering.
4. Retrieval. At question time, embed the query, find the nearest chunks, and filter by metadata where it matters, like only current policies or only documents this user may see.
5. Generation. Build a grounded prompt: the question, the retrieved evidence, and firm instructions to answer only from that evidence and to say so when it is insufficient. Cite sources so users can verify.

Here is the skeleton in code, deliberately minimal so the structure stays visible:

# pip install chromadb sentence-transformersimport chromadbfrom sentence_transformers import SentenceTransformerembedder = SentenceTransformer("all-MiniLM-L6-v2")db = chromadb.Client().create_collection("docs")# index: chunks with metadatadb.add(ids=[c["id"] for c in chunks],embeddings=embedder.encode([c["text"] for c in chunks]).tolist(),metadatas=[{"source": c["source"], "date": c["date"]} for c in chunks],documents=[c["text"] for c in chunks],)# retrieve: top matches for a question, filteredhits = db.query(query_embeddings=embedder.encode([question]).tolist(),n_results=5,where={"date": {"$gte": "2026-01-01"}},)context = "\n\n".join(hits["documents"][0])# then: prompt the model with question + context + grounding rules

The Five Upgrades That Separate Production From Demo

Hybrid retrieval

Vector search understands meaning but fumbles exact strings: product codes, error messages, names. Pair it with keyword search and merge the results. Users ask both kinds of questions, and a system that only handles one kind feels broken half the time.

Reranking

First pass retrieval optimises for speed across millions of chunks. Add a second stage where a stronger model rescores the top candidates against the query before generation. It is one of the highest value additions in the whole stack, routinely turning a nearly right context into the right one.

Honest failure behaviour

Decide what happens when retrieval finds nothing good. The production answer is to say so, offer where a human might look, and log the miss. A system that admits ignorance earns trust; one that improvises loses it permanently, usually in front of the customer's boss.

Evaluation as a habit, not an event

Build a test set of real questions with known correct answers, and measure on every change: is the right chunk retrieved, is the answer faithful to the retrieved evidence, is it actually relevant? Frameworks like RAGAS make this systematic. Without it, every tweak is a guess, and regressions ship silently.

Observability and cost control

Trace every request end to end: what was retrieved, what was generated, how long it took, what it cost. Cache frequent queries, batch embeddings, and watch token spend. A pipeline that works but costs ten times the budget is still a failed deployment in the customer's eyes.

The Mistakes We See Most, So You Can Skip Them

• Indexing everything once and never again, so the system confidently serves last year's policy as current truth
• Ignoring permissions, so retrieval happily surfaces documents the asking user was never allowed to see
• Tuning chunk size by vibes instead of against an evaluation set
• Treating hallucination as a model problem when it is usually a retrieval problem wearing a disguise
• Shipping without tracing, then trying to debug a bad answer with no record of what the model was shown

Every one of these is survivable in a demo and fatal in production. That gap is exactly why companies pay a premium for engineers who have crossed it at least once with their own hands.

The Jobs This Knowledge Unlocks

Crossing the demo to production gap once, with your own hands, is the single most valuable line on an AI engineering resume right now. These are the roles that pay for it.

The stack behind these roles

PythonLangChainFAISSChromaDBFastAPIDockerLangSmith