RAG: Chunking and Retrieval
How to split documents into chunks, why overlap and metadata matter, and how BM25, dense search, reciprocal rank fusion, and reranking combine into strong retrieval.
intermediate50 min#rag#chunking#bm25#hybrid-search#rrf#reranking
Prerequisites: Vectors and Embeddings, Vector Databases
Why it matters
Retrieval-augmented generation (RAG) gives a language model the facts it needs by looking them up and pasting them into the prompt. The model can only be as good as what you hand it. If retrieval returns the wrong passages, the best model will answer confidently from the wrong text; if it returns the right passage cut in half, the answer will be incomplete. In practice most RAG quality problems are retrieval problems, and most retrieval problems start with how documents were chunked.
Intuition
You are studying for an exam with a 500-page textbook and can bring only three pages to the desk. Which three? First you decide how to cut the book into pages (chunking): too coarse and a page is mostly irrelevant; too fine and a sentence loses the context that gave it meaning. Then you flip through to find pages that match the question. You might search the index by exact words (keyword search) and also skim by topic (semantic search), then compare the two shortlists. Finally you read the top candidates properly and keep the best three (reranking).
How it works
Chunking. A chunk is the unit you embed, index, and retrieve. Common strategies:
- Fixed size (by tokens, words, or characters): simple and predictable, but may cut mid-sentence.
- Sentence or paragraph based: respects natural boundaries; chunk sizes vary.
- Structure aware: split on headings, sections, or code functions, so a chunk is a coherent unit. Usually the best choice when documents have structure (Markdown, HTML, source code).
- Semantic: split where the topic shifts, detected with embeddings. More work; gains vary.
Size trade-off. Small chunks (a few sentences) match queries precisely and waste fewer prompt tokens, but may lack context. Large chunks keep context but dilute the embedding (one vector averaging several topics) and cost more tokens. A few hundred tokens is a common starting point; tune it against an evaluation set rather than guessing.
Overlap. Repeat a little text (for example 10 to 20 percent) between neighbouring chunks so a fact that straddles a boundary still appears whole in at least one chunk. More overlap means more chunks, more storage, and more duplicate hits.
Metadata. Store with each chunk: source document, section title, position, date, language, access group. Metadata enables filtering (only this product, only this year, only what this user may see), citations that point back to the source, and prefixing a chunk with its title so it makes sense on its own.
document: [ w1 w2 w3 w4 w5 w6 w7 w8 w9 w10 w11 w12 ... ]
chunk 0 : [ w1 ........................ w6 ]
chunk 1 : [ w5 ........................ w10 ] overlap = w5 w6
chunk 2 : [ w9 ................ w14 ]
The chunker slides a window over the words. Step through it on a tiny example.
Illustrative sizes (6 words, overlap 2) so the picture stays small; the lesson code uses size 14 and overlap 4 the same way. A fact in words 4 and 5 appears whole in two chunks.
Keyword retrieval with BM25. BM25 scores a chunk by the query terms it contains. Each term contributes more if it is rare across the collection (inverse document frequency, IDF) and if it appears in the chunk often, with diminishing returns (k1) and a length normalisation (b) so long chunks do not win just by being long:
idf(t) = ln(1 + (N - df + 0.5) / (df + 0.5))
score = sum over query terms of idf(t) * tf * (k1 + 1) / (tf + k1 * (1 - b + b * len / avglen))
BM25 is excellent for exact tokens such as names, error codes, and product IDs, and it needs no model. It misses synonyms.
Dense retrieval. Embed chunks and the query, and take the nearest vectors. It finds paraphrases and related concepts, but can miss exact identifiers and rare terms.
Hybrid search and reciprocal rank fusion (RRF). Combine both. Their scores are on different scales, so rather than adding raw scores, fuse ranks: each result gets 1 / (c + rank) from each list (rank starting at 1, with c commonly 60), and these are summed. Documents ranked well by either method rise; those ranked well by both rise most. RRF needs no tuning of score scales.
Reranking. First-stage retrieval is cheap and approximate: fetch, say, the top 20 to 50. A reranker (typically a cross-encoder model that reads query and chunk together) re-scores that short list more accurately, and you keep the best 3 to 5. It costs a model call per candidate, which is why it runs only on the shortlist.
The stages fit together as a recall-then-precision pipeline.
| Operation | Time | Notes |
|---|---|---|
| Chunking a document | O(length) | One pass; overlap multiplies chunk count slightly |
| BM25 query (inverted index) | O(postings of query terms) | Only chunks containing a query term are touched |
| Dense query (brute force) | O(n * d) | ANN index makes it ~O(log n) |
| RRF fusion | O(k) | Merge two short ranked lists |
| Rerank top m | m model calls | Most accurate and most expensive stage |
Code: chunking, BM25, hybrid fusion
Everything below runs offline. The dense side uses a stand-in embedding (hashed bag of words), so it behaves much like keyword search; with a real embedding model you would swap embed for an API call and gain true semantic matching. Likewise rerank is a crude stand-in (fraction of query terms covered) for a cross-encoder call.
import math
import re
DOCS = {
"refunds": "Refund policy. Customers can request a refund within 30 days of purchase. "
"Refunds are issued to the original payment method. Digital goods are refundable only if unused.",
"shipping": "Shipping times. Standard shipping takes 5 to 7 business days. Express shipping takes 2 days. "
"Orders to remote regions may need extra time. Tracking numbers are emailed after dispatch.",
"security": "Account security. Reset your password from the login page using your email. "
"Enable two factor authentication to protect your account. Never share your password with support staff.",
}
def words(text):
return re.findall(r"[a-z0-9]+", text.lower())
def chunk(doc_id, text, size=14, overlap=4):
ws, step, out = text.split(), size - overlap, []
for n, start in enumerate(range(0, len(ws), step)):
out.append({"id": f"{doc_id}#{n}", "text": " ".join(ws[start:start + size]),
"meta": {"doc": doc_id, "start": start}})
if start + size >= len(ws):
break
return out
class BM25:
def __init__(self, chunks, k1=1.5, b=0.75):
self.chunks, self.k1, self.b = chunks, k1, b
self.toks = [words(c["text"]) for c in chunks]
self.avg = sum(len(t) for t in self.toks) / len(self.toks)
self.df = {}
for t in self.toks:
for w in set(t):
self.df[w] = self.df.get(w, 0) + 1
def search(self, query, k=3):
n, scored = len(self.chunks), []
for c, toks in zip(self.chunks, self.toks):
s = 0.0
for w in set(words(query)):
tf = toks.count(w)
if tf:
idf = math.log(1 + (n - self.df[w] + 0.5) / (self.df[w] + 0.5))
s += idf * tf * (self.k1 + 1) / (tf + self.k1 * (1 - self.b + self.b * len(toks) / self.avg))
scored.append((-s, c["id"]))
scored.sort()
return [(cid, -neg) for neg, cid in scored[:k] if neg < 0]
def embed(text, dim=64): # stand-in for a real embedding model
v = [0.0] * dim
for w in words(text):
h = 2166136261
for ch in w:
h = ((h ^ ord(ch)) * 16777619) % 4294967296
v[h % dim] += 1.0
n = math.sqrt(sum(x * x for x in v)) or 1.0
return [x / n for x in v]
def dense_search(chunks, query, k=3):
q = embed(query)
scored = sorted((-sum(a * b for a, b in zip(q, embed(c["text"]))), c["id"]) for c in chunks)
return [(cid, -neg) for neg, cid in scored[:k]]
def rrf(lists, c=60):
fused = {}
for lst in lists:
for rank, (cid, _) in enumerate(lst, start=1):
fused[cid] = fused.get(cid, 0.0) + 1.0 / (c + rank)
return sorted(fused.items(), key=lambda kv: (-kv[1], kv[0]))
def rerank(chunks, query, candidates): # stand-in for a cross-encoder
by_id = {c["id"]: c for c in chunks}
q = set(words(query))
scored = [(-len(q & set(words(by_id[cid]["text"]))) / len(q), i, cid)
for i, (cid, _) in enumerate(candidates)]
return [cid for _, _, cid in sorted(scored)]
chunks = [c for doc_id, text in DOCS.items() for c in chunk(doc_id, text)]
print("chunks:", len(chunks))
for c in chunks[:2]:
print(c["id"], "|", c["text"])
query = "refund for a digital purchase within 30 days"
bm = BM25(chunks).search(query)
de = dense_search(chunks, query)
fused = rrf([bm, de])
print("bm25 :", " ".join(f"{cid}:{s:.3f}" for cid, s in bm))
print("dense:", " ".join(f"{cid}:{s:.3f}" for cid, s in de))
print("rrf :", " ".join(f"{cid}:{s:.4f}" for cid, s in fused[:3]))
print("rerank:", rerank(chunks, query, fused[:3]))const DOCS = {
refunds:
"Refund policy. Customers can request a refund within 30 days of purchase. " +
"Refunds are issued to the original payment method. Digital goods are refundable only if unused.",
shipping:
"Shipping times. Standard shipping takes 5 to 7 business days. Express shipping takes 2 days. " +
"Orders to remote regions may need extra time. Tracking numbers are emailed after dispatch.",
security:
"Account security. Reset your password from the login page using your email. " +
"Enable two factor authentication to protect your account. Never share your password with support staff.",
};
const words = (text) => text.toLowerCase().match(/[a-z0-9]+/g) || [];
function chunk(docId, text, size = 14, overlap = 4) {
const ws = text.split(/\s+/);
const step = size - overlap;
const out = [];
for (let start = 0, n = 0; start < ws.length; start += step, n++) {
out.push({
id: `${docId}#${n}`,
text: ws.slice(start, start + size).join(" "),
meta: { doc: docId, start },
});
if (start + size >= ws.length) break;
}
return out;
}
const byScore = (a, b) => a[0] - b[0] || (a[1] < b[1] ? -1 : a[1] > b[1] ? 1 : 0);
class BM25 {
constructor(chunks, k1 = 1.5, b = 0.75) {
this.chunks = chunks;
this.k1 = k1;
this.b = b;
this.toks = chunks.map((c) => words(c.text));
this.avg = this.toks.reduce((s, t) => s + t.length, 0) / this.toks.length;
this.df = new Map();
for (const t of this.toks) {
for (const w of new Set(t)) this.df.set(w, (this.df.get(w) || 0) + 1);
}
}
search(query, k = 3) {
const n = this.chunks.length;
const scored = this.chunks.map((c, i) => {
const toks = this.toks[i];
let s = 0.0;
for (const w of new Set(words(query))) {
const tf = toks.filter((x) => x === w).length;
if (tf) {
const df = this.df.get(w);
const idf = Math.log(1 + (n - df + 0.5) / (df + 0.5));
s += (idf * tf * (this.k1 + 1)) / (tf + this.k1 * (1 - this.b + (this.b * toks.length) / this.avg));
}
}
return [-s, c.id];
});
scored.sort(byScore);
return scored.slice(0, k).filter(([neg]) => neg < 0).map(([neg, id]) => [id, -neg]);
}
}
function embed(text, dim = 64) { // stand-in for a real embedding model
const v = new Array(dim).fill(0);
for (const w of words(text)) {
let h = 2166136261;
for (const ch of w) h = Math.imul(h ^ ch.charCodeAt(0), 16777619) >>> 0;
v[h % dim] += 1;
}
const n = Math.sqrt(v.reduce((s, x) => s + x * x, 0)) || 1;
return v.map((x) => x / n);
}
function denseSearch(chunks, query, k = 3) {
const q = embed(query);
const scored = chunks.map((c) => {
const e = embed(c.text);
return [-q.reduce((s, a, i) => s + a * e[i], 0), c.id];
});
scored.sort(byScore);
return scored.slice(0, k).map(([neg, id]) => [id, -neg]);
}
function rrf(lists, c = 60) {
const fused = new Map();
for (const lst of lists) {
lst.forEach(([id], i) => fused.set(id, (fused.get(id) || 0) + 1 / (c + i + 1)));
}
return [...fused].sort((a, b) => b[1] - a[1] || (a[0] < b[0] ? -1 : 1));
}
function rerank(chunks, query, candidates) { // stand-in for a cross-encoder
const byId = new Map(chunks.map((c) => [c.id, c]));
const q = new Set(words(query));
return candidates
.map(([id], i) => {
const shared = [...new Set(words(byId.get(id).text))].filter((w) => q.has(w)).length;
return [-shared / q.size, i, id];
})
.sort((a, b) => a[0] - b[0] || a[1] - b[1])
.map((x) => x[2]);
}
const chunks = Object.entries(DOCS).flatMap(([id, text]) => chunk(id, text));
console.log("chunks:", chunks.length);
for (const c of chunks.slice(0, 2)) console.log(c.id, "|", c.text);
const query = "refund for a digital purchase within 30 days";
const bm = new BM25(chunks).search(query);
const de = denseSearch(chunks, query);
const fused = rrf([bm, de]);
const fmt = (list, digits) => list.map(([id, s]) => `${id}:${s.toFixed(digits)}`).join(" ");
console.log("bm25 :", fmt(bm, 3));
console.log("dense:", fmt(de, 3));
console.log("rrf :", fmt(fused.slice(0, 3), 4));
console.log("rerank:", JSON.stringify(rerank(chunks, query, fused.slice(0, 3))).replace(/"/g, "'").replace(/,/g, ", "));Both versions print:
chunks: 9
refunds#0 | Refund policy. Customers can request a refund within 30 days of purchase. Refunds are
refunds#1 | of purchase. Refunds are issued to the original payment method. Digital goods are refundable
bm25 : refunds#0:10.090 refunds#1:2.567 refunds#2:1.701
dense: refunds#0:0.619 security#0:0.316 refunds#1:0.250
rrf : refunds#0:0.0328 refunds#1:0.0320 security#0:0.0161
rerank: ['refunds#0', 'refunds#1', 'security#0']
The chunk lines show the overlap in action: the end of one chunk reappears at the start of the next. The BM25 and dense lists are produced by different scorers on different scales, which is exactly why fusion uses ranks rather than raw scores. The reranker only reorders the short fused list.
Here is the RRF arithmetic behind the rrf line above, with c = 60. A dash means the chunk was not in that list's top 3, so it earns nothing from it.
Common pitfalls
- Chunks that lose their context: a chunk saying "it expires after 30 days" is useless without knowing what "it" is. Prefix chunks with the document and section title.
- Splitting through tables and code: fixed-size splitters can cut a table row or a function in half. Use structure-aware splitting for those.
- Too much overlap: it multiplies storage and fills the top results with near-duplicates. Deduplicate or diversify results.
- Embedding-only retrieval: exact strings such as order numbers or error codes often need keyword search. Hybrid fixes this.
- Adding raw BM25 and cosine scores: the scales differ, so the larger-scale score dominates. Use rank fusion or normalise carefully.
- Retrieving too few or too many: a tiny
krisks missing the answer; a hugekfloods the prompt with noise. Retrieve wide, rerank, then pass a few. - Never re-chunking after changes: when you change chunk size or embedding model, rebuild and re-evaluate. Measure, do not assume.
Practice
Chunk by sentences with a size cap easy
Hint
Accumulate sentences until adding the next would exceed the cap, then start a new chunk.
Show solution
Split text on sentence boundaries, then loop: append the sentence to the current chunk if the word count stays within the cap, otherwise emit the chunk and start a new one with that sentence. To add overlap, begin the new chunk with the last sentence of the previous one. O(length) time.
Compute BM25 by hand for one term medium
Hint
Count N and df first, then plug into idf, then the tf part.
Show solution
Suppose N = 100 chunks, the term appears in df = 10, k1 = 1.5, b = 0.75, a chunk has tf = 2 and average length. The IDF is ln(1 + 90.5 / 10.5), about 2.26. The length factor is 1 (length equals average), so the tf part is 2 * 2.5 / (2 + 1.5) = 1.43. The score is about 3.2. A rarer term (smaller df) raises IDF; a longer chunk lowers the tf part.
Show why RRF beats summing scores medium
Hint
Give the two scorers different ranges, such as 0 to 1 and 0 to 30.
Show solution
If BM25 scores range 0 to 30 and cosine scores 0 to 1, a plain sum is almost entirely BM25, so the dense result barely matters. RRF uses only ranks, so each retriever contributes on equal footing: rank 1 gives 1/61, rank 2 gives 1/62, and so on. A chunk ranked near the top by both lists beats a chunk ranked first by one and missing from the other.
Tune chunk size scientifically hard
Hint
You need questions with known answer chunks, and a metric from the evaluation lesson.
Show solution
Build a set of real questions each labelled with the passage that answers it. For each candidate configuration (size 100, 200, 400 tokens; overlap 0, 10, 20 percent), re-chunk, re-index, run retrieval, and compute recall@k and MRR. Pick the configuration with the best metrics at acceptable token cost and index size. Re-check when the documents or embedding model change.
Interview questions
- How do you choose a chunk size? Start with a few hundred tokens, respect document structure, then tune against a labelled question set; trade precision against context and token cost.
- Why use overlap? So facts straddling a boundary survive intact in some chunk, at the cost of duplication.
- BM25 versus embeddings? BM25 excels at exact and rare terms with no model; embeddings handle paraphrase and meaning; hybrid search combines their strengths.
- What is reciprocal rank fusion? Sum
1 / (c + rank)across ranked lists; it merges retrievers without comparing incompatible scores. - What does a reranker add? A more accurate second-stage scorer, usually a cross-encoder, applied only to a short candidate list.
Interviewers look for: treating chunking as a tunable design choice, using metadata for filtering and citations, and explaining hybrid plus rerank as a recall-then-precision pipeline.
Quick check
Your RAG system fails on queries containing exact product codes, though paraphrased questions work well. What is the most appropriate fix?
Finished this lesson?