DevPath
Tracks

The RAG Pipeline End to End

Ingest, retrieve, augment the prompt, generate, and verify citations: a complete retrieval-augmented generation pipeline you can run offline with stand-ins for the embedding model and the llm.

intermediate45 min#rag#pipeline#citations#grounding#abstention

Prerequisites: RAG: Chunking and Retrieval, Prompting and Structured Output

Why it matters

A language model only knows what was in its training data and what you put in its prompt. It does not know your company's refund policy, last week's incident report, or a private codebase. Fine-tuning for every change is slow and expensive. Retrieval-augmented generation (RAG) solves this by looking up relevant passages at question time and giving them to the model along with the question. The answer can then be grounded in current, private text, cited, and updated by editing a document rather than retraining a model.

Intuition

Think of an open-book exam. A student who has read widely (the model) still does better when allowed to look things up, as long as they find the right page, read it carefully, and note where each fact came from. RAG is the open-book setup: a librarian (retrieval) fetches a few pages, the student (generation) writes the answer using only those pages, and a marker (verification) checks that every cited page was really one of the ones provided.

How it works

The pipeline has two phases.

Offline: ingest
Load documents
Chunk + metadata
Embed
Store in index
Online: answer
Embed question
Retrieve top k
Score gate
Build prompt
LLM call
Verify citations
Two separate paths share one index. Ingest runs when documents change; the online path runs per question and can stop early at the score gate.

Offline: ingest.

  1. Load documents from files, a wiki, a database.
  2. Chunk them (previous lesson) and attach metadata: source, title, position, permissions.
  3. Embed each chunk with an embedding model.
  4. Store chunk text, metadata, and vector in an index or vector store.

Re-run ingestion when documents change, using stable chunk ids so old versions are replaced, not duplicated.

Online: answer.

  1. Retrieve: embed the question with the same model, fetch the top k chunks (hybrid search and reranking if needed), and apply metadata and permission filters.
  2. Gate: if the best score is below a threshold, the knowledge base probably lacks the answer. Abstain instead of letting the model improvise.
  3. Augment: build a prompt containing instructions, the numbered sources, and the question. State clearly that the model must answer only from the sources, cite by number, and say it does not know otherwise. Place sources and instructions where they are separated from the user's text.
  4. Generate: call the model.
  5. Verify and return: parse the citations, check that each refers to a source you actually supplied, and return the answer together with the sources so the user can inspect them.

Here is one question travelling through the online path, using the lesson's own example.

UserRetrieverPrompt builderLLMCitation checkHow do I reset my password?security#0, score 0.63rules + [1] source + questionanswer ... [1]answer + 1 sourceverified answer + sources
If the retriever returned nothing above the threshold, the flow would end right after step 2 with I don't know, and the LLM would never be called.
 question
    |
    v
 [embed] -> [retrieve top k] -> [score gate] --low--> "I don't know"
                                    | ok
                                    v
              [build prompt: rules + [1] [2] sources + question]
                                    |
                                    v
                               [ LLM call ]
                                    |
                                    v
                  [check citations are in sources] -> answer + sources

Where it goes wrong. Failures come from every stage: the answer was never ingested; chunking split the fact; retrieval ranked it too low; the prompt had too many distracting sources; the model ignored the sources or cited incorrectly. Debug stage by stage by logging the retrieved chunks and scores for every request. The next lesson shows how to measure each stage.

Context budget. More retrieved chunks give higher recall but cost tokens and can dilute attention. Typical systems retrieve a wider set, rerank, and pass only a handful.

This is the exact prompt the lesson's code builds for the password question, drawn as a stack.

SYSTEM rulesanswer only from the numbered sources, cite like [1], else say you don't know
SOURCES[1] Reset your password from the login page using your email.
QUESTIONHow do I reset my password?
The prompt is rules, then numbered sources, then the question. The numbers are what make citations checkable: the model may only cite [1], and the code verifies that.
OperationTimeNotes
Ingest n chunksO(n) embedding callsBatch them; re-embed only changed chunks
Retrieve (brute force)O(n * d)ANN index for large n
Prompt sizeO(k * chunk length)Directly drives cost and latency
GenerateO(output tokens) passesUsually the slowest step
Citation checkO(answer length)Cheap, always worth running

Code: a complete offline pipeline

Two things are replaced by small deterministic stand-ins so the whole program runs with no network and no API key. embed is a hashed bag-of-words (a real embedding model call would go here), and llm is a fake model that copies the best-matching numbered source and cites it (a real chat-completion call would go here). Everything between them, including the prompt, the score gate, and the citation check, is exactly what a production pipeline does.

import math
import re
 
STOP = {"the", "a", "an", "to", "of", "in", "from", "can", "is", "are", "do", "does", "how",
        "what", "i", "my", "your", "by", "for", "and", "on", "within", "using", "be", "it", "long"}
 
DOCS = {
    "refunds": "Customers can request a refund within 30 days of purchase. "
               "Refunds are issued to the original payment method within 5 business days.",
    "shipping": "Standard shipping takes 5 to 7 business days. "
                "Express shipping takes 2 business days and costs extra.",
    "security": "Reset your password from the login page using your email. "
                "Two factor authentication can be enabled in account settings.",
}
 
 
def words(text):
    return [w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP]
 
 
def embed(text, dim=64):  # stand-in for a real embedding model call
    v = [0.0] * dim
    for w in words(text):
        h = 2166136261
        for ch in w:
            h = ((h ^ ord(ch)) * 16777619) % 4294967296
        v[h % dim] += 1.0
    n = math.sqrt(sum(x * x for x in v)) or 1.0
    return [x / n for x in v]
 
 
def llm(prompt):  # stand-in for a real chat model call
    question = prompt.split("QUESTION: ")[1]
    q = set(words(question))
    best, best_overlap = None, 0
    for line in prompt.split("\n"):
        m = re.match(r"\[(\d+)\] (.*)", line)
        if m and len(q & set(words(m.group(2)))) > best_overlap:
            best, best_overlap = m, len(q & set(words(m.group(2))))
    if best is None:
        return "I don't know."
    return best.group(2) + " [" + best.group(1) + "]"
 
 
def citations_ok(answer, n_sources):
    cited = {int(n) for n in re.findall(r"\[(\d+)\]", answer)}
    return cited <= set(range(1, n_sources + 1))
 
 
class RAG:
    def __init__(self, min_score=0.3):
        self.chunks, self.min_score = [], min_score
 
    def ingest(self, docs):
        for doc_id, text in docs.items():
            for n, sentence in enumerate(s.strip() for s in text.split(". ") if s.strip()):
                self.chunks.append({"id": f"{doc_id}#{n}", "text": sentence.rstrip(".") + ".",
                                    "vec": embed(sentence)})
 
    def retrieve(self, question, k=2):
        q = embed(question)
        scored = sorted(((-sum(a * b for a, b in zip(q, c["vec"])), c["id"], c) for c in self.chunks),
                        key=lambda t: (t[0], t[1]))
        return [(-neg, c) for neg, _, c in scored[:k] if -neg >= self.min_score]
 
    def build_prompt(self, question, hits):
        lines = ["SYSTEM: Answer using only the numbered sources. Cite like [1]. "
                 "If they do not contain the answer, say you don't know.", "SOURCES:"]
        lines += [f"[{i}] {c['text']}" for i, (_, c) in enumerate(hits, start=1)]
        return "\n".join(lines) + "\nQUESTION: " + question
 
    def ask(self, question):
        hits = self.retrieve(question)
        if not hits:
            return {"answer": "I don't know.", "sources": [], "verified": True}
        answer = llm(self.build_prompt(question, hits))
        return {"answer": answer, "sources": [f"{c['id']}:{s:.2f}" for s, c in hits],
                "verified": citations_ok(answer, len(hits))}
 
 
rag = RAG()
rag.ingest(DOCS)
print("indexed", len(rag.chunks), "chunks")
print(rag.build_prompt("How do I reset my password?", rag.retrieve("How do I reset my password?")))
for q in ("How long does express shipping take?", "How do I reset my password?", "What is the CEO salary?"):
    r = rag.ask(q)
    print("Q:", q)
    print("A:", r["answer"], "| sources:", " ".join(r["sources"]), "| verified:", r["verified"])
print("cites [1] of 2 sources:", citations_ok("Refunds take 5 days [1]", 2))
print("cites [7] of 2 sources:", citations_ok("Refunds take 5 days [7]", 2))

Both versions print:

indexed 6 chunks
SYSTEM: Answer using only the numbered sources. Cite like [1]. If they do not contain the answer, say you don't know.
SOURCES:
[1] Reset your password from the login page using your email.
QUESTION: How do I reset my password?
Q: How long does express shipping take?
A: Express shipping takes 2 business days and costs extra. [1] | sources: shipping#1:0.41 | verified: True
Q: How do I reset my password?
A: Reset your password from the login page using your email. [1] | sources: security#0:0.63 | verified: True
Q: What is the CEO salary?
A: I don't know. | sources:  | verified: True
cites [1] of 2 sources: True
cites [7] of 2 sources: False

The first question retrieves the express-shipping sentence and the answer cites source 1. Only one chunk clears the score threshold in each answered case, so each prompt has a single source. The CEO question retrieves nothing above the threshold, so the pipeline abstains without calling the model at all, which saves cost and avoids a made-up answer. The last two lines exercise the citation check directly: citing source 7 when only two were supplied fails verification.

Common pitfalls

  • No abstain path: if retrieval always returns something, the model will always answer. Add a score gate and an explicit "I don't know" instruction.
  • Unverified citations: models can cite sources that do not exist or do not support the claim. Check ids in code, and for high-stakes use also check that quoted text appears in the cited chunk.
  • Prompt injection through documents: retrieved text is untrusted. A document saying "ignore your instructions" can hijack the model. Delimit sources, keep instructions firm, and limit what the model can do with its output.
  • Skipping permissions: retrieval must apply the user's access rights before text reaches the prompt, not after the answer is written.
  • Stale index: edited or deleted documents linger in the index. Track document versions and delete chunks for removed sources.
  • Too many chunks in the prompt: extra context raises cost and can hurt accuracy. Retrieve wide, rerank, pass few.
  • Not logging retrievals: without the retrieved chunks and scores for each question you cannot tell a retrieval failure from a generation failure.

Practice

Return structured answers easy

Hint

Reuse the validate-and-retry idea from the prompting lesson.

Show solution

Ask the model for JSON with answer and citations (a list of source numbers). Validate that the keys exist and each citation is within 1..k. On failure, retry once with the error message. This is easier to verify than parsing brackets from free text.

Add metadata filtering medium

Hint

Filter before scoring, not after taking the top k.

Show solution

Give each chunk metadata such as audience: public | staff. In retrieve, skip chunks whose audience the current user may not see, then score and take the top k. Filtering after taking the top k can leave you with nothing even though allowed matches exist.

Handle follow-up questions medium

Hint

Retrieval sees only the text you embed. What does 'what about express?' mean on its own?

Show solution

Before retrieval, rewrite the follow-up into a standalone question using the conversation (an llm call, such as turning "what about express?" into "How long does express shipping take?"). Retrieve with the rewritten question and answer with the history included. Embedding the raw follow-up retrieves poorly because it lacks context.

Decide when to re-index hard

Hint

Consider documents changing, deletions, and model upgrades.

Show solution

Keep a content hash per document and per chunk. On each ingest run, re-embed only chunks whose hash changed, upsert them by stable id, and delete ids no longer present. A new embedding model or chunking scheme invalidates everything, so build a fresh index alongside the old one, evaluate it, then switch traffic over. Record model name and chunking parameters with the index.

Interview questions

  1. Describe a RAG system. An ingest phase (chunk, embed, store) and a query phase (retrieve, augment the prompt, generate, verify), so the model answers from supplied evidence.
  2. RAG or fine-tuning? RAG injects fresh or private facts and supports citations and easy updates; fine-tuning changes style or skills and is a poor way to store facts that change.
  3. How do you reduce hallucination in RAG? Good retrieval, instructions to answer only from sources, an abstain path, citation verification, and evaluating faithfulness.
  4. How do you handle documents the user must not see? Enforce permissions in retrieval through metadata filters, before anything enters the prompt.
  5. Why log retrieved chunks? To tell whether a bad answer came from retrieval or from generation.

Interviewers look for: clear separation of ingest and query paths, an abstain strategy, citation checking, and awareness that retrieved text is untrusted input.

Quick check

A user asks something your documents do not cover, yet the model gives a confident answer. Which design change addresses this directly?

Finished this lesson?

Further resources

Related lessons