The RAG Pipeline End to End
Ingest, retrieve, augment the prompt, generate, and verify citations: a complete retrieval-augmented generation pipeline you can run offline with stand-ins for the embedding model and the llm.
intermediate45 min#rag#pipeline#citations#grounding#abstention
Prerequisites: RAG: Chunking and Retrieval, Prompting and Structured Output
Why it matters
A language model only knows what was in its training data and what you put in its prompt. It does not know your company's refund policy, last week's incident report, or a private codebase. Fine-tuning for every change is slow and expensive. Retrieval-augmented generation (RAG) solves this by looking up relevant passages at question time and giving them to the model along with the question. The answer can then be grounded in current, private text, cited, and updated by editing a document rather than retraining a model.
Intuition
Think of an open-book exam. A student who has read widely (the model) still does better when allowed to look things up, as long as they find the right page, read it carefully, and note where each fact came from. RAG is the open-book setup: a librarian (retrieval) fetches a few pages, the student (generation) writes the answer using only those pages, and a marker (verification) checks that every cited page was really one of the ones provided.
How it works
The pipeline has two phases.
Offline: ingest.
- Load documents from files, a wiki, a database.
- Chunk them (previous lesson) and attach metadata: source, title, position, permissions.
- Embed each chunk with an embedding model.
- Store chunk text, metadata, and vector in an index or vector store.
Re-run ingestion when documents change, using stable chunk ids so old versions are replaced, not duplicated.
Online: answer.
- Retrieve: embed the question with the same model, fetch the top
kchunks (hybrid search and reranking if needed), and apply metadata and permission filters. - Gate: if the best score is below a threshold, the knowledge base probably lacks the answer. Abstain instead of letting the model improvise.
- Augment: build a prompt containing instructions, the numbered sources, and the question. State clearly that the model must answer only from the sources, cite by number, and say it does not know otherwise. Place sources and instructions where they are separated from the user's text.
- Generate: call the model.
- Verify and return: parse the citations, check that each refers to a source you actually supplied, and return the answer together with the sources so the user can inspect them.
Here is one question travelling through the online path, using the lesson's own example.
question
|
v
[embed] -> [retrieve top k] -> [score gate] --low--> "I don't know"
| ok
v
[build prompt: rules + [1] [2] sources + question]
|
v
[ LLM call ]
|
v
[check citations are in sources] -> answer + sources
Where it goes wrong. Failures come from every stage: the answer was never ingested; chunking split the fact; retrieval ranked it too low; the prompt had too many distracting sources; the model ignored the sources or cited incorrectly. Debug stage by stage by logging the retrieved chunks and scores for every request. The next lesson shows how to measure each stage.
Context budget. More retrieved chunks give higher recall but cost tokens and can dilute attention. Typical systems retrieve a wider set, rerank, and pass only a handful.
This is the exact prompt the lesson's code builds for the password question, drawn as a stack.
| Operation | Time | Notes |
|---|---|---|
| Ingest n chunks | O(n) embedding calls | Batch them; re-embed only changed chunks |
| Retrieve (brute force) | O(n * d) | ANN index for large n |
| Prompt size | O(k * chunk length) | Directly drives cost and latency |
| Generate | O(output tokens) passes | Usually the slowest step |
| Citation check | O(answer length) | Cheap, always worth running |
Code: a complete offline pipeline
Two things are replaced by small deterministic stand-ins so the whole program runs with no network and no API key. embed is a hashed bag-of-words (a real embedding model call would go here), and llm is a fake model that copies the best-matching numbered source and cites it (a real chat-completion call would go here). Everything between them, including the prompt, the score gate, and the citation check, is exactly what a production pipeline does.
import math
import re
STOP = {"the", "a", "an", "to", "of", "in", "from", "can", "is", "are", "do", "does", "how",
"what", "i", "my", "your", "by", "for", "and", "on", "within", "using", "be", "it", "long"}
DOCS = {
"refunds": "Customers can request a refund within 30 days of purchase. "
"Refunds are issued to the original payment method within 5 business days.",
"shipping": "Standard shipping takes 5 to 7 business days. "
"Express shipping takes 2 business days and costs extra.",
"security": "Reset your password from the login page using your email. "
"Two factor authentication can be enabled in account settings.",
}
def words(text):
return [w for w in re.findall(r"[a-z0-9]+", text.lower()) if w not in STOP]
def embed(text, dim=64): # stand-in for a real embedding model call
v = [0.0] * dim
for w in words(text):
h = 2166136261
for ch in w:
h = ((h ^ ord(ch)) * 16777619) % 4294967296
v[h % dim] += 1.0
n = math.sqrt(sum(x * x for x in v)) or 1.0
return [x / n for x in v]
def llm(prompt): # stand-in for a real chat model call
question = prompt.split("QUESTION: ")[1]
q = set(words(question))
best, best_overlap = None, 0
for line in prompt.split("\n"):
m = re.match(r"\[(\d+)\] (.*)", line)
if m and len(q & set(words(m.group(2)))) > best_overlap:
best, best_overlap = m, len(q & set(words(m.group(2))))
if best is None:
return "I don't know."
return best.group(2) + " [" + best.group(1) + "]"
def citations_ok(answer, n_sources):
cited = {int(n) for n in re.findall(r"\[(\d+)\]", answer)}
return cited <= set(range(1, n_sources + 1))
class RAG:
def __init__(self, min_score=0.3):
self.chunks, self.min_score = [], min_score
def ingest(self, docs):
for doc_id, text in docs.items():
for n, sentence in enumerate(s.strip() for s in text.split(". ") if s.strip()):
self.chunks.append({"id": f"{doc_id}#{n}", "text": sentence.rstrip(".") + ".",
"vec": embed(sentence)})
def retrieve(self, question, k=2):
q = embed(question)
scored = sorted(((-sum(a * b for a, b in zip(q, c["vec"])), c["id"], c) for c in self.chunks),
key=lambda t: (t[0], t[1]))
return [(-neg, c) for neg, _, c in scored[:k] if -neg >= self.min_score]
def build_prompt(self, question, hits):
lines = ["SYSTEM: Answer using only the numbered sources. Cite like [1]. "
"If they do not contain the answer, say you don't know.", "SOURCES:"]
lines += [f"[{i}] {c['text']}" for i, (_, c) in enumerate(hits, start=1)]
return "\n".join(lines) + "\nQUESTION: " + question
def ask(self, question):
hits = self.retrieve(question)
if not hits:
return {"answer": "I don't know.", "sources": [], "verified": True}
answer = llm(self.build_prompt(question, hits))
return {"answer": answer, "sources": [f"{c['id']}:{s:.2f}" for s, c in hits],
"verified": citations_ok(answer, len(hits))}
rag = RAG()
rag.ingest(DOCS)
print("indexed", len(rag.chunks), "chunks")
print(rag.build_prompt("How do I reset my password?", rag.retrieve("How do I reset my password?")))
for q in ("How long does express shipping take?", "How do I reset my password?", "What is the CEO salary?"):
r = rag.ask(q)
print("Q:", q)
print("A:", r["answer"], "| sources:", " ".join(r["sources"]), "| verified:", r["verified"])
print("cites [1] of 2 sources:", citations_ok("Refunds take 5 days [1]", 2))
print("cites [7] of 2 sources:", citations_ok("Refunds take 5 days [7]", 2))const STOP = new Set(["the", "a", "an", "to", "of", "in", "from", "can", "is", "are", "do", "does", "how",
"what", "i", "my", "your", "by", "for", "and", "on", "within", "using", "be", "it", "long"]);
const DOCS = {
refunds:
"Customers can request a refund within 30 days of purchase. " +
"Refunds are issued to the original payment method within 5 business days.",
shipping:
"Standard shipping takes 5 to 7 business days. " +
"Express shipping takes 2 business days and costs extra.",
security:
"Reset your password from the login page using your email. " +
"Two factor authentication can be enabled in account settings.",
};
const words = (text) => (text.toLowerCase().match(/[a-z0-9]+/g) || []).filter((w) => !STOP.has(w));
function embed(text, dim = 64) { // stand-in for a real embedding model call
const v = new Array(dim).fill(0);
for (const w of words(text)) {
let h = 2166136261;
for (const ch of w) h = Math.imul(h ^ ch.charCodeAt(0), 16777619) >>> 0;
v[h % dim] += 1;
}
const n = Math.sqrt(v.reduce((s, x) => s + x * x, 0)) || 1;
return v.map((x) => x / n);
}
function llm(prompt) { // stand-in for a real chat model call
const question = prompt.split("QUESTION: ")[1];
const q = new Set(words(question));
let best = null, bestOverlap = 0;
for (const line of prompt.split("\n")) {
const m = line.match(/^\[(\d+)\] (.*)/);
if (m) {
const overlap = [...new Set(words(m[2]))].filter((w) => q.has(w)).length;
if (overlap > bestOverlap) { best = m; bestOverlap = overlap; }
}
}
if (best === null) return "I don't know.";
return best[2] + " [" + best[1] + "]";
}
function citationsOk(answer, nSources) {
const cited = [...answer.matchAll(/\[(\d+)\]/g)].map((m) => Number(m[1]));
return cited.every((n) => n >= 1 && n <= nSources);
}
class RAG {
constructor(minScore = 0.3) {
this.chunks = [];
this.minScore = minScore;
}
ingest(docs) {
for (const [docId, text] of Object.entries(docs)) {
text.split(". ").map((s) => s.trim()).filter(Boolean).forEach((sentence, n) => {
this.chunks.push({ id: `${docId}#${n}`, text: sentence.replace(/\.$/, "") + ".", vec: embed(sentence) });
});
}
}
retrieve(question, k = 2) {
const q = embed(question);
return this.chunks
.map((c) => [q.reduce((s, a, i) => s + a * c.vec[i], 0), c])
.sort((a, b) => b[0] - a[0] || (a[1].id < b[1].id ? -1 : 1))
.slice(0, k)
.filter(([score]) => score >= this.minScore);
}
buildPrompt(question, hits) {
const lines = [
"SYSTEM: Answer using only the numbered sources. Cite like [1]. " +
"If they do not contain the answer, say you don't know.",
"SOURCES:",
...hits.map(([, c], i) => `[${i + 1}] ${c.text}`),
];
return lines.join("\n") + "\nQUESTION: " + question;
}
ask(question) {
const hits = this.retrieve(question);
if (hits.length === 0) return { answer: "I don't know.", sources: [], verified: true };
const answer = llm(this.buildPrompt(question, hits));
return {
answer,
sources: hits.map(([s, c]) => `${c.id}:${s.toFixed(2)}`),
verified: citationsOk(answer, hits.length),
};
}
}
const rag = new RAG();
rag.ingest(DOCS);
console.log("indexed", rag.chunks.length, "chunks");
console.log(rag.buildPrompt("How do I reset my password?", rag.retrieve("How do I reset my password?")));
const py = (b) => (b ? "True" : "False");
for (const q of ["How long does express shipping take?", "How do I reset my password?", "What is the CEO salary?"]) {
const r = rag.ask(q);
console.log("Q:", q);
console.log("A:", r.answer, "| sources:", r.sources.join(" "), "| verified:", py(r.verified));
}
console.log("cites [1] of 2 sources:", py(citationsOk("Refunds take 5 days [1]", 2)));
console.log("cites [7] of 2 sources:", py(citationsOk("Refunds take 5 days [7]", 2)));Both versions print:
indexed 6 chunks
SYSTEM: Answer using only the numbered sources. Cite like [1]. If they do not contain the answer, say you don't know.
SOURCES:
[1] Reset your password from the login page using your email.
QUESTION: How do I reset my password?
Q: How long does express shipping take?
A: Express shipping takes 2 business days and costs extra. [1] | sources: shipping#1:0.41 | verified: True
Q: How do I reset my password?
A: Reset your password from the login page using your email. [1] | sources: security#0:0.63 | verified: True
Q: What is the CEO salary?
A: I don't know. | sources: | verified: True
cites [1] of 2 sources: True
cites [7] of 2 sources: False
The first question retrieves the express-shipping sentence and the answer cites source 1. Only one chunk clears the score threshold in each answered case, so each prompt has a single source. The CEO question retrieves nothing above the threshold, so the pipeline abstains without calling the model at all, which saves cost and avoids a made-up answer. The last two lines exercise the citation check directly: citing source 7 when only two were supplied fails verification.
Common pitfalls
- No abstain path: if retrieval always returns something, the model will always answer. Add a score gate and an explicit "I don't know" instruction.
- Unverified citations: models can cite sources that do not exist or do not support the claim. Check ids in code, and for high-stakes use also check that quoted text appears in the cited chunk.
- Prompt injection through documents: retrieved text is untrusted. A document saying "ignore your instructions" can hijack the model. Delimit sources, keep instructions firm, and limit what the model can do with its output.
- Skipping permissions: retrieval must apply the user's access rights before text reaches the prompt, not after the answer is written.
- Stale index: edited or deleted documents linger in the index. Track document versions and delete chunks for removed sources.
- Too many chunks in the prompt: extra context raises cost and can hurt accuracy. Retrieve wide, rerank, pass few.
- Not logging retrievals: without the retrieved chunks and scores for each question you cannot tell a retrieval failure from a generation failure.
Practice
Return structured answers easy
Hint
Reuse the validate-and-retry idea from the prompting lesson.
Show solution
Ask the model for JSON with answer and citations (a list of source numbers). Validate that the keys exist and each citation is within 1..k. On failure, retry once with the error message. This is easier to verify than parsing brackets from free text.
Add metadata filtering medium
Hint
Filter before scoring, not after taking the top k.
Show solution
Give each chunk metadata such as audience: public | staff. In retrieve, skip chunks whose audience the current user may not see, then score and take the top k. Filtering after taking the top k can leave you with nothing even though allowed matches exist.
Handle follow-up questions medium
Hint
Retrieval sees only the text you embed. What does 'what about express?' mean on its own?
Show solution
Before retrieval, rewrite the follow-up into a standalone question using the conversation (an llm call, such as turning "what about express?" into "How long does express shipping take?"). Retrieve with the rewritten question and answer with the history included. Embedding the raw follow-up retrieves poorly because it lacks context.
Decide when to re-index hard
Hint
Consider documents changing, deletions, and model upgrades.
Show solution
Keep a content hash per document and per chunk. On each ingest run, re-embed only chunks whose hash changed, upsert them by stable id, and delete ids no longer present. A new embedding model or chunking scheme invalidates everything, so build a fresh index alongside the old one, evaluate it, then switch traffic over. Record model name and chunking parameters with the index.
Interview questions
- Describe a RAG system. An ingest phase (chunk, embed, store) and a query phase (retrieve, augment the prompt, generate, verify), so the model answers from supplied evidence.
- RAG or fine-tuning? RAG injects fresh or private facts and supports citations and easy updates; fine-tuning changes style or skills and is a poor way to store facts that change.
- How do you reduce hallucination in RAG? Good retrieval, instructions to answer only from sources, an abstain path, citation verification, and evaluating faithfulness.
- How do you handle documents the user must not see? Enforce permissions in retrieval through metadata filters, before anything enters the prompt.
- Why log retrieved chunks? To tell whether a bad answer came from retrieval or from generation.
Interviewers look for: clear separation of ingest and query paths, an abstain strategy, citation checking, and awareness that retrieved text is untrusted input.
Quick check
A user asks something your documents do not cover, yet the model gives a confident answer. Which design change addresses this directly?
Finished this lesson?