How LLMs Work: Tokens, Context, and Sampling
Tokens, the context window, next-token prediction, and how temperature and top-p turn scores into text. Includes why models hallucinate and a sampler built from scratch.
intermediate45 min#llm#tokens#context-window#sampling#temperature#hallucination
Prerequisites: Vectors and Embeddings, Unicode, Encodings, and Endianness
Why it matters
Whether you call a hosted model or run your own, you will write code around a large language model (LLM): build prompts, budget their size, parse the output, and decide how much randomness to allow. The model behaves much less mysteriously once you know the few mechanical facts underneath. Costs are measured in tokens, input has a hard size limit, output is sampled from a probability distribution, and confident wrong answers are a built-in property rather than a rare bug. This lesson covers those facts and lets you build the sampler yourself.
Intuition
An LLM is a very sophisticated autocomplete. Given the text so far, it predicts what comes next. It does not predict a whole sentence at once. It predicts one small piece, appends it, and repeats. If you type "The capital of France is", the model scores every possible next piece: " Paris" gets a very high score, " Lyon" a low one, " banana" a tiny one. A separate step, sampling, picks one. Run it in a loop and you have a chat reply.
Two consequences follow. First, the model is optimised to produce plausible continuations, not to look things up, so a fluent but false statement can score well. Second, because a choice is made at each step, the same prompt can give different answers, and you control how adventurous that choice is.
How it works
The toy tokenizer in the code below splits "The unbelievable cat sat on the mat." into 10 tokens. A leading space belongs to the token, shown here as a dot.
Tokens. Models do not read characters or words. A tokenizer splits text into tokens, pieces from a fixed vocabulary learned from data: common words are one token, rarer words split into several sub-word pieces, and unusual strings fall back to bytes or single characters. Consequences: cost and limits are counted in tokens, not words; non-English text, code, and numbers often use more tokens per word; and models can be bad at character-level tasks such as counting letters because they never see letters directly. (See the Unicode lesson for why bytes and characters differ.) A rough rule for English prose is that a token is around three-quarters of a word, but always count with the provider's own tokenizer when it matters.
Context window. The model can attend to a maximum number of tokens at once, covering your input and its output together. Anything beyond is simply not seen. Longer contexts cost more compute and money, and models can use information in the middle of a very long prompt less reliably than at the edges. Applications must decide what to keep: recent messages, a summary of older ones, or retrieved snippets (the subject of the RAG lessons).
Next-token prediction. For each step the network outputs one raw score (a logit) per vocabulary entry. A softmax turns scores into probabilities that sum to 1:
p_i = exp(z_i / T) / sum_j exp(z_j / T)
where z are the logits and T is the temperature.
The loop below shows generation for the prompt from the code. The first token comes from the lesson's toy scores; the later steps are illustrative.
Click Next. The model never writes a whole sentence at once: it appends one sampled token, then feeds the longer text back in.
Sampling controls.
- Greedy: always take the highest-probability token. Deterministic, but can be repetitive.
- Temperature
T: divides the logits before softmax.T < 1sharpens the distribution (more predictable),T > 1flattens it (more surprising), andTapproaching 0 becomes greedy. - Top-p (nucleus): sort tokens by probability, keep the smallest set whose probabilities add up to at least
p, renormalise, and sample from that set. This cuts off the long tail of unlikely tokens while still allowing variety. - Top-k: keep only the
kmost likely tokens.
Low temperature suits extraction, classification, and code. Higher temperature suits brainstorming. Even at temperature 0, some hosted systems are not perfectly reproducible, so never build logic that needs bit-identical output.
Hallucination. The model produces text that looks like its training data. When the answer is not in its weights, or the context does not contain it, it still produces something plausible. Mitigations are architectural rather than magic: supply the facts in the prompt (RAG), ask for citations you can check, allow "I don't know", constrain the output format, verify with code, and keep a human in the loop where errors are costly.
| Operation | Time | Notes |
|---|---|---|
| Generate one token | One forward pass | Cost grows with context length |
| Generate m tokens | m sequential passes | Output is produced one token at a time |
| Softmax over vocab V | O(V) | Per generated token |
| Attention over context n | O(n^2) per layer in the basic form | Why long contexts are expensive |
Code: tokenizer, softmax, temperature, top-p
A real model supplies the logits. Here a hand-written table of logits for the prompt "The capital of France is" stands in for the model, so you can see exactly what sampling does. The tokenizer is also a toy (greedy longest match against a tiny vocabulary), just enough to show sub-word splitting and token budgets. To use a real model you would replace LOGITS with the scores or the sampled text from the provider; the sampling maths is what runs inside or beside the model.
import math
VOCAB = {"The", " un", "believ", "able", " cat", " sat", " on", " the", " mat", "."}
def tokenize(text):
tokens, i = [], 0
while i < len(text):
for size in range(min(8, len(text) - i), 0, -1):
piece = text[i:i + size]
if piece in VOCAB or size == 1: # fall back to one character
tokens.append(piece)
i += size
break
return tokens
def fit(messages, budget):
"""Keep the newest messages that fit in the token budget."""
kept, used = [], 0
for m in reversed(messages):
n = len(tokenize(m))
if used + n > budget:
break
kept.append(m)
used += n
return list(reversed(kept)), used
class Rng:
def __init__(self, seed):
self.s = seed
def next(self):
self.s = (self.s * 1664525 + 1013904223) % 4294967296
return self.s / 4294967296
LOGITS = {" Paris": 5.0, " Lyon": 2.5, " London": 2.0, " a": 1.0, " the": 0.5, " banana": -1.0}
def softmax(logits, temperature):
scaled = {t: z / temperature for t, z in logits.items()}
m = max(scaled.values())
exps = {t: math.exp(z - m) for t, z in scaled.items()}
total = sum(exps.values())
probs = {t: e / total for t, e in exps.items()}
return sorted(probs.items(), key=lambda kv: (-kv[1], kv[0]))
def top_p(ranked, p):
kept, cum = [], 0.0
for tok, pr in ranked:
kept.append((tok, pr))
cum += pr
if cum >= p:
break
total = sum(pr for _, pr in kept)
return [(tok, pr / total) for tok, pr in kept]
def sample(ranked, rng):
r, cum = rng.next(), 0.0
for tok, pr in ranked:
cum += pr
if r < cum:
return tok
return ranked[-1][0]
toks = tokenize("The unbelievable cat sat on the mat.")
print(len(toks), "|".join(toks))
print("zebra ->", len(tokenize("zebra")), "tokens")
history = ["The cat sat.", "The cat sat on the mat.", "The mat."]
kept, used = fit(history, 8)
print("kept", len(kept), "of", len(history), "messages using", used, "tokens")
for T in (0.5, 1.0, 2.0):
ranked = softmax(LOGITS, T)
print(f"T={T}", " ".join(f"{t.strip()}:{p:.3f}" for t, p in ranked[:4]))
nucleus = top_p(softmax(LOGITS, 1.0), 0.9)
print("top-p 0.9 keeps:", " ".join(f"{t.strip()}:{p:.3f}" for t, p in nucleus))
for T in (0.5, 2.0):
rng, counts = Rng(42), {}
ranked = softmax(LOGITS, T)
for _ in range(1000):
t = sample(ranked, rng).strip()
counts[t] = counts.get(t, 0) + 1
print(f"1000 draws at T={T}:", " ".join(f"{t.strip()}:{counts.get(t.strip(), 0)}" for t, _ in ranked[:3]))const VOCAB = new Set(["The", " un", "believ", "able", " cat", " sat", " on", " the", " mat", "."]);
function tokenize(text) {
const tokens = [];
let i = 0;
while (i < text.length) {
for (let size = Math.min(8, text.length - i); size >= 1; size--) {
const piece = text.slice(i, i + size);
if (VOCAB.has(piece) || size === 1) { // fall back to one character
tokens.push(piece);
i += size;
break;
}
}
}
return tokens;
}
// Keep the newest messages that fit in the token budget.
function fit(messages, budget) {
const kept = [];
let used = 0;
for (const m of [...messages].reverse()) {
const n = tokenize(m).length;
if (used + n > budget) break;
kept.push(m);
used += n;
}
return { kept: kept.reverse(), used };
}
class Rng {
constructor(seed) {
this.s = seed;
}
next() {
this.s = (Math.imul(this.s, 1664525) + 1013904223) >>> 0;
return this.s / 4294967296;
}
}
const LOGITS = { " Paris": 5.0, " Lyon": 2.5, " London": 2.0, " a": 1.0, " the": 0.5, " banana": -1.0 };
function softmax(logits, temperature) {
const scaled = Object.entries(logits).map(([t, z]) => [t, z / temperature]);
const m = Math.max(...scaled.map(([, z]) => z));
const exps = scaled.map(([t, z]) => [t, Math.exp(z - m)]);
const total = exps.reduce((s, [, e]) => s + e, 0);
return exps
.map(([t, e]) => [t, e / total])
.sort((a, b) => b[1] - a[1] || (a[0] < b[0] ? -1 : 1));
}
function topP(ranked, p) {
const kept = [];
let cum = 0.0;
for (const [tok, pr] of ranked) {
kept.push([tok, pr]);
cum += pr;
if (cum >= p) break;
}
const total = kept.reduce((s, [, pr]) => s + pr, 0);
return kept.map(([tok, pr]) => [tok, pr / total]);
}
function sample(ranked, rng) {
const r = rng.next();
let cum = 0.0;
for (const [tok, pr] of ranked) {
cum += pr;
if (r < cum) return tok;
}
return ranked[ranked.length - 1][0];
}
const toks = tokenize("The unbelievable cat sat on the mat.");
console.log(toks.length, toks.join("|"));
console.log("zebra ->", tokenize("zebra").length, "tokens");
const history = ["The cat sat.", "The cat sat on the mat.", "The mat."];
const { kept, used } = fit(history, 8);
console.log("kept", kept.length, "of", history.length, "messages using", used, "tokens");
const fmt = (ranked, n) => ranked.slice(0, n).map(([t, p]) => `${t.trim()}:${p.toFixed(3)}`).join(" ");
for (const T of [0.5, 1.0, 2.0]) {
console.log(`T=${T.toFixed(1)}`, fmt(softmax(LOGITS, T), 4));
}
const nucleus = topP(softmax(LOGITS, 1.0), 0.9);
console.log("top-p 0.9 keeps:", fmt(nucleus, nucleus.length));
for (const T of [0.5, 2.0]) {
const rng = new Rng(42);
const counts = {};
const ranked = softmax(LOGITS, T);
for (let i = 0; i < 1000; i++) {
const t = sample(ranked, rng).trim();
counts[t] = (counts[t] || 0) + 1;
}
const line = ranked.slice(0, 3).map(([t]) => `${t.trim()}:${counts[t.trim()] || 0}`).join(" ");
console.log(`1000 draws at T=${T.toFixed(1)}:`, line);
}Both versions print:
10 The| un|believ|able| cat| sat| on| the| mat|.
zebra -> 5 tokens
kept 1 of 3 messages using 3 tokens
T=0.5 Paris:0.990 Lyon:0.007 London:0.002 a:0.000
T=1.0 Paris:0.859 Lyon:0.071 London:0.043 a:0.016
T=2.0 Paris:0.556 Lyon:0.159 London:0.124 a:0.075
top-p 0.9 keeps: Paris:0.924 Lyon:0.076
1000 draws at T=0.5: Paris:984 Lyon:13 London:3
1000 draws at T=2.0: Paris:545 Lyon:158 London:122
Look at what temperature does: at T=0.5 the top token dominates, at T=2.0 the tail gets real probability, and top-p discards the unlikely tokens before sampling. The draw counts at low temperature are almost all " Paris", while at high temperature other tokens show up regularly.
Common pitfalls
- Counting words instead of tokens: budgets, prices, and limits are per token. Measure with the real tokenizer.
- Forgetting that output counts toward the window: if the prompt fills the context, there is no room left to answer.
- Using high temperature for structured tasks: extraction and JSON generation want low temperature, otherwise you add random errors.
- Assuming determinism: the same prompt can give different outputs, even at temperature 0 on some systems. Tests should check properties, not exact strings.
- Stuffing the whole context: more text is not always better. Irrelevant material costs money and can distract the model.
- Trusting recall of facts: names, numbers, citations, and API signatures are common hallucination targets.
Practice
Implement top-k sampling easy
Hint
Rank by probability, cut to the first k, renormalise, then sample as before.
Show solution
Take the ranked list from softmax, slice [:k], divide each probability by the sum of the kept ones, and pass the result to sample. With k = 1 this is greedy decoding. Unlike top-p, the cutoff does not adapt to how peaked the distribution is.
Why does dividing by temperature before softmax flatten the distribution? medium
Hint
Compare exp(z) ratios for T = 1 and T = 2 when two logits differ by 2.
Show solution
Two logits differing by 2 have probability ratio exp(2), about 7.4, at T = 1. At T = 2 the scaled difference is 1 and the ratio is exp(1), about 2.7, so the gap shrinks. As T grows, all scaled logits approach each other and the distribution tends toward uniform; as T shrinks toward 0, the largest logit takes nearly all the mass (greedy).
Build a sliding-window chat history that always keeps the system prompt medium
Hint
Reserve budget for the system message and for the reply before filling with history.
Show solution
Compute available = context_limit - reserved_for_reply - tokens(system). Walk the history from newest to oldest adding messages while they fit, then reverse. Prepend the system message. If something important is dropped, consider replacing the dropped part with a short summary instead of discarding it.
Reduce hallucinated citations hard
Hint
Combine retrieval, an explicit abstain option, and a programmatic check.
Show solution
Retrieve source passages and put them in the prompt with ids. Instruct the model to answer only from them, cite ids, and say it does not know otherwise. After generation, verify in code that every cited id exists and that quoted text appears in that passage; reject or retry otherwise. This turns an unverifiable claim into a checkable one. The RAG pipeline lesson builds this loop.
Interview questions
- What is a token and why does it matter? A sub-word unit from a learned vocabulary. Pricing, context limits, and latency are all counted in tokens, and tokenization explains odd behaviour such as weak letter counting.
- How does temperature work? It divides logits before softmax: low values sharpen the distribution toward the top token, high values flatten it.
- Top-p versus top-k? Top-k keeps a fixed number of candidates; top-p keeps a variable number covering a probability mass, adapting to model confidence.
- Why do LLMs hallucinate? They are trained to continue text plausibly, not to verify truth; when knowledge is missing they still produce fluent output.
- Why is generation slower than reading the prompt? Prompt tokens are processed in parallel, but output tokens are produced one after another, each needing a forward pass.
Interviewers look for: describing generation as a loop of sampling one token at a time, connecting token counts to cost and limits, and offering concrete hallucination mitigations instead of "just prompt better".
Quick check
You want a classifier prompt to give the same label on repeated runs as often as possible. Which setting helps most?
Finished this lesson?
Further resources
- Attention Is All You Need (paper) (article)
- Large language model (Wikipedia) (article)
- Anthropic documentation (docs)