Retrieval Augmented Generation Explained in 2026: How RAG Actually Works, Step by Step

Retrieval augmented generation sits behind nearly every "chat with your data" product shipped in the last two years, and it is the most common thing an AI engineer in India is asked to build in an interview or a first week on the job. The idea takes one sentence to explain. The engineering is where people come unstuck, because a RAG system has five moving parts and the failure of any one of them looks identical from outside: a confident, fluent, wrong answer.

What Retrieval Augmented Generation Actually Is

A language model knows what was in its training data. It does not know your company's leave policy or the runbook for the payment gateway that broke on Tuesday. You have two options: retrain the model on your data, which is expensive and stale the moment a document changes, or find the relevant paragraphs at question time and paste them into the prompt. The second option is retrieval augmented generation.

The mechanism is unglamorous. When a user asks "how many casual leaves do contractors get", the system does not send that question to the model. It sends something closer to this:

You are answering from the documents below. If the answer is not
in them, say you do not know.

[Document 1, hr-policy-2026.pdf, section 4.2]
Contract staff accrue casual leave at 0.5 days per completed month...

[Document 2, hr-policy-2026.pdf, section 4.5]
Casual leave may not be carried into the next financial year...

Question: how many casual leaves do contractors get?

The model's job shrinks from "know everything" to "read these three paragraphs and answer". That is a much easier job, and models are good at it. Which means the quality of your system is decided almost entirely by whether stages one to four put the right paragraphs in that prompt. This is the mental shift that separates people who build working RAG systems from people who build impressive demos: you are not tuning a model, you are building a search engine.

How Does Retrieval Augmented Generation Work? The Five Stages of a RAG Pipeline

Every RAG pipeline, from a 40-line script to a system serving a bank, is these five stages. Two of them run offline, ahead of time. Three run on every single question.

Notice how few of these failures are about the model. Building each of these stages end to end, then breaking them on purpose to see what happens, is the core of the hands-on work in 360DT's live AI Engineer course, which builds RAG systems and tool-using agents in class rather than in slides.

Chunking and Vector Embeddings: Where Most RAG Pipelines Are Won or Lost

Chunking is the least glamorous decision in the pipeline and the one that most often decides whether it works. Split too small and a chunk reads "this is not permitted" with no idea what "this" refers to. Split too large and you retrieve four pages to answer one sentence, diluting the signal. A sane default is 500 to 1,000 tokens with 10 to 20 percent overlap, then adjust against real questions. Overlap matters more than people expect: without it, an answer that straddles a boundary is unretrievable at any chunk size.

import numpy as np

def chunk(text, size=800, overlap=150):
    """Word-window chunker. Crude, but a fine baseline."""
    words, out, i = text.split(), [], 0
    while i < len(words):
        out.append(" ".join(words[i:i + size]))
        i += size - overlap
    return out

What vector embeddings really are

An embedding model reads a chunk and returns a list of numbers, typically 512 to 2,048 of them, positioned so that text with similar meaning lands in a similar place. "Annual leave entitlement" and "how many holidays do I get" end up close together despite sharing no words. That is the entire trick, and it is why vector embeddings answer questions phrased in ways nobody anticipated.

Dimension count is a real engineering decision, not a vanity metric. Per the pgvector project's documentation, the standard vector type indexes up to 2,000 dimensions, while the half-precision halfvec type raises that ceiling to 4,000 at half the storage. Several current models support Matryoshka truncation, which lets you cut a long vector down and keep most of the quality: Voyage AI's voyage-3-large ships at 2,048, 1,024, 512 and 256 dimensions from the same model. Cost is rarely the blocker at prototype scale either, with Voyage's published pricing putting voyage-4-lite at $0.02 per million tokens on top of 200 million free tokens for the voyage-4 generation.

Also read: AI Engineer Roadmap 2026: 7 Steps to Land Your First Role in India.

Why Hybrid Search Beats Pure Vector Search

Here is the failure that surprises every team the first time. A user searches for error code TS-999. Semantic search returns five chunks about error handling in general and not the one page containing the literal string TS-999. Embeddings are built to generalise, and generalising is exactly the wrong behaviour for identifiers, SKUs, dates, version numbers and people's names.

Keyword search, specifically BM25, has the opposite profile: useless at synonyms, perfect at exact strings. Hybrid search runs both and fuses the result lists. The standard fusion method, Reciprocal Rank Fusion, needs no tuning and no score normalisation because it looks only at ranks:

WITH dense AS (
  SELECT id, row_number() OVER (ORDER BY embedding <=> $1) AS rank
  FROM chunks ORDER BY embedding <=> $1 LIMIT 50
),
sparse AS (
  SELECT id, row_number() OVER (
           ORDER BY ts_rank_cd(tsv, plainto_tsquery('english', $2)) DESC) AS rank
  FROM chunks WHERE tsv @@ plainto_tsquery('english', $2) LIMIT 50
)
SELECT id, SUM(1.0 / (60 + rank)) AS rrf_score
FROM (SELECT * FROM dense UNION ALL SELECT * FROM sparse) fused
GROUP BY id
ORDER BY rrf_score DESC
LIMIT 10;

The constant 60 is the conventional RRF damping value; it stops rank-one results from dominating so completely that the second list never contributes. Both indexes live in the same Postgres table:

CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE chunks (
  id        bigserial PRIMARY KEY,
  doc_id    text NOT NULL,
  heading   text,
  content   text NOT NULL,
  embedding vector(1024),
  tsv       tsvector GENERATED ALWAYS AS
              (to_tsvector('english', content)) STORED
);

CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);
CREATE INDEX ON chunks USING gin (tsv);

If you do not pass build parameters, pgvector's documentation states the HNSW index defaults to m = 16 and ef_construction = 64. Raising ef_construction gives a better index at the cost of a slower build, which is a trade you make once at ingest and benefit from on every query.

Choosing a vector store without overthinking it

Two notes on that last row. Managed hybrid search saves you the RRF code above, at the cost of tying retrieval to one cloud, which is a normal trade for teams already building on the Azure AI application stack. And getting documents into any of these indexes is a data engineering job before it is an AI one: incremental loads, deduplication and scheduled refresh, the same patterns you would build in a Microsoft Fabric pipeline.

Reranking: The Cheapest Accuracy Win in a Retrieval Augmented Generation Stack

Retrieval is fast and approximate: it compares a query vector against millions of chunk vectors, so it can only afford to look at each chunk in isolation. A reranker is the opposite, a cross-encoder that reads the query and one candidate chunk together and scores how well that chunk answers this specific question. Far too slow for a whole corpus, perfect for 50 candidates.

So the pattern is retrieve wide, rerank narrow. Pull 50 candidates with hybrid search, rerank, pass the top 5 to the model. You get most of the accuracy of an expensive search for the price of a cheap one.

The economics are friendly. Cohere's pricing lists Rerank v3.5 at $0.001 per search, that is $2.00 per 1,000 searches, where one search means a query plus up to 100 documents, and Cohere's documentation puts the free trial key at 1,000 calls a month and 20 requests a minute. For a prototype serving your own team, that trial allowance is often enough to run for weeks.

Build a Minimal Retrieval Augmented Generation Pipeline in About 40 Lines

Nothing here needs a framework. This is the whole pipeline in pure Python with numpy, so you can see there is no magic in it. Install one dependency:

pip install numpy

Then the pipeline. The only thing abstracted away is embed(), because the exact call differs by provider; swap in whichever embedding API you chose above, and make sure it returns one vector per input string.

import numpy as np

def embed(texts: list[str]) -> np.ndarray:
    """Call your embedding provider. Returns (n, dim) float32."""
    raise NotImplementedError

def normalise(m):
    return m / np.linalg.norm(m, axis=1, keepdims=True)

# offline: build the index once
docs   = {"hr-policy-2026.pdf": open("hr.txt").read()}
chunks = [(name, c) for name, text in docs.items() for c in chunk(text)]
matrix = normalise(embed([c for _, c in chunks]))

# online: answer one question
def answer(question, k=5):
    q = normalise(embed([question]))[0]
    scores = matrix @ q                      # cosine, rows are unit length
    top = np.argsort(-scores)[:k]
    context = "\n\n".join(
        f"[{chunks[i][0]}]\n{chunks[i][1]}" for i in top
    )
    prompt = (
        "Answer only from the documents below. Cite the file name "
        "for each claim. If the answer is not present, say so.\n\n"
        f"{context}\n\nQuestion: {question}"
    )
    return prompt   # send this to your model of choice

Two details in that snippet do real work. Normalising every row means the dot product is cosine similarity, so the whole retrieval step is one matrix multiply. And the instruction to say so when the answer is absent is not politeness; it is the single most effective anti-hallucination control in a RAG system, and it costs one sentence.

Run it against a folder of your own documents and you have a working system. Then break it: ask something answered by a table, ask something using an acronym the docs never spell out, ask something the docs genuinely do not cover. The gap between this and production is entirely in how you handle those three cases. The next step after that is wiring the same retrieval layer in as a callable tool so an agent decides when to search rather than searching every time, which is the tool-use work practised live in 360DT's CCDV-F developer prep course.

Also read: MCP Explained: The 18% Most People Underestimate, on the protocol used to expose retrieval tools to a model.

Case Study: How Anthropic Cut Retrieval Failures by 67%

This is the most useful published RAG engineering story available, because it isolates each change and reports the number. Anthropic's engineering team documented a technique they call Contextual Retrieval, aimed at one specific problem: a chunk pulled out of a document loses the context that made it meaningful. A chunk reading "revenue grew 3% over the previous quarter" is useless without knowing which company and which quarter. Their fix was to generate a short, chunk-specific context line from the parent document and prepend it before embedding. Then they layered on the techniques above. Anthropic's published results:

Read that table again with the earlier point in mind. Not one row involves a better language model. The gains came from splitting text more thoughtfully, searching two ways instead of one, and spending a few milliseconds reordering results. Knowing which of these trade-offs to reach for first is architecture-level judgement, and on a RAG project it is worth more than any amount of prompt tinkering.

Six Ways RAG Breaks in Production, and How to Fix Each One

The last row is the one that separates a project from a product. Treating retrieval quality as a regression-tested property, with monitoring and rollback, is straightforward operations work, and it is the same discipline taught around model deployment in the MLOps Engineer course.

How to Evaluate a RAG System Before You Trust It

Split evaluation in two, because the fixes are completely different. Retrieval quality asks whether the right chunk was in the results at all. Answer quality asks whether the model used it properly. Measure retrieval first: if recall is 60%, no amount of prompt engineering will save you, because four questions in ten never had a chance. Build a golden set of 50 real questions with the chunk IDs that should answer each. Fifty is enough to be useful and small enough that you will actually do it.

hits = 0
for question, expected_ids in golden_set:
    retrieved = [c.id for c in retrieve(question, k=5)]
    hits += any(cid in retrieved for cid in expected_ids)

print(f"recall@5 = {hits / len(golden_set):.1%}")

Run it on every change to chunk size, embedding model, k, or fusion weights. One number, one command, and suddenly every tuning argument on the team becomes an experiment instead of an opinion. Only once recall@5 is comfortably above 90% does it make sense to start grading the generated answers for faithfulness and citation accuracy.

Also read: Agentic AI Jobs in India 2026, on where RAG skills sit in the current hiring market.

That is the whole thing: chunk carefully, embed sensibly, retrieve two ways, rerank, and instruct the model to abstain. Build the 40-line version this weekend against documents you actually care about, then measure recall before you touch anything else. That single habit will put you ahead of most people who list RAG on their CV.

If you would rather build it with an instructor watching your screen, the AI Engineer Course runs live on Saturday and Sunday evenings, 8 to 11 PM IST, and goes from a first retrieval pipeline to agents that call tools and act. Not ready for a paid programme? Start with a free webinar or a free demo class, or scan the full certifications overview.

Frequently asked questions

About this guide. 360 Digital Transformation is an independent training provider. We are not affiliated with the certification bodies, vendors or open source projects mentioned, and our courses are exam preparation rather than official training. Tools and versions change quickly; commands and figures cited were checked on 8 September 2026.