Vectors and Embeddings
How text becomes a list of numbers, why distance in that space means similarity, and how to compare vectors.
intermediate40 min#vectors#embeddings#cosine-similarity#rag
Prerequisites: Arrays and Dynamic Arrays
Why it matters
Search by keyword fails when people use different words for the same idea ("car" vs "automobile"). Embeddings fix this by turning text, images, or any item into a vector: a fixed-length list of numbers where similar meanings land close together. They power semantic search, recommendations, duplicate detection, clustering, and the retrieval step of RAG.
Intuition
Imagine a map where every sentence is a pin. Sentences about cooking cluster in one region, football in another, and "pasta recipe" sits near "how to boil spaghetti" but far from "offside rule". An embedding model is the cartographer: it reads a sentence and returns the pin's coordinates. A real map has 2 coordinates; embeddings typically have hundreds or thousands (for example 384, 768, or 1536), so the "map" has that many dimensions. We cannot draw it, but distance still works.
The coordinates have no human-readable meaning individually. Meaning lives in the geometry: nearby points are similar.
How it works
Every use of embeddings follows the same pipeline: text goes through a model once, and the resulting vector is stored or compared.
- Embedding model: a neural network trained so that related inputs produce nearby vectors. You send text in and get back a vector like
[0.12, -0.03, ...]. - Same model for everything: vectors from different models live in different spaces and cannot be compared. Embed your documents and your queries with the same model.
- Compare with a similarity measure:
- Dot product: sum of element-wise products. Larger means more similar (when magnitudes matter).
- Cosine similarity: the dot product divided by both lengths. It measures the angle between vectors, ignoring length, and ranges from -1 to 1. This is the usual default for text.
- Euclidean distance: straight-line distance. Smaller means more similar.
- Normalize once, then use the dot product: if every vector is scaled to length 1, cosine similarity equals the dot product, which is cheaper to compute at scale.
The dot product, term by term, using the vectors from the code below. Each column multiplies, then the products are added.
| Operation | Time | Notes |
|---|---|---|
| Dot product of two d-dim vectors | O(d) | One pass |
| Brute-force search over n vectors | O(n * d) | Exact; fine up to ~100k vectors |
| Approximate search (HNSW, IVF) | ~O(log n) | Next lessons: faster, slightly inexact |
Code: similarity from scratch
We use tiny hand-made 3-dimensional vectors so you can check the math by hand. Real embeddings come from a model, but the comparison code is identical.
import math
def dot(a, b):
return sum(x * y for x, y in zip(a, b))
def norm(a):
return math.sqrt(dot(a, a))
def cosine(a, b):
return dot(a, b) / (norm(a) * norm(b))
def top_k(query, items, k=2):
"""Brute-force nearest neighbours: score everything, keep the best k."""
scored = [(cosine(query, vec), name) for name, vec in items.items()]
return sorted(scored, reverse=True)[:k]
docs = {
"pasta recipe": [0.9, 0.1, 0.0],
"boil spaghetti": [0.8, 0.2, 0.1],
"offside rule": [0.0, 0.1, 0.9],
}
query = [0.85, 0.15, 0.05] # pretend this is "how to cook noodles"
for score, name in top_k(query, docs):
print(f"{score:.3f} {name}")const dot = (a, b) => a.reduce((sum, x, i) => sum + x * b[i], 0);
const norm = (a) => Math.sqrt(dot(a, a));
const cosine = (a, b) => dot(a, b) / (norm(a) * norm(b));
// Brute-force nearest neighbours: score everything, keep the best k.
function topK(query, items, k = 2) {
return Object.entries(items)
.map(([name, vec]) => [cosine(query, vec), name])
.sort((x, y) => y[0] - x[0])
.slice(0, k);
}
const docs = {
"pasta recipe": [0.9, 0.1, 0.0],
"boil spaghetti": [0.8, 0.2, 0.1],
"offside rule": [0.0, 0.1, 0.9],
};
const query = [0.85, 0.15, 0.05]; // pretend this is "how to cook noodles"
for (const [score, name] of topK(query, docs)) {
console.log(`${score.toFixed(3)} ${name}`);
}The two cooking documents score near 1 and "offside rule" is left out of the top 2. This top_k function is already a working (if tiny) vector search; the next lessons make it fast with indexes.
Common pitfalls
- Comparing across models or dimensions: the lengths may even match by accident. The results are meaningless.
- Forgetting to normalize: with dot product on un-normalized vectors, long vectors win regardless of meaning.
- Embedding huge documents as one vector: one vector cannot capture a 50-page document. Split into chunks (covered in the RAG lesson).
- Assuming "similar" means "relevant": an embedding finds topical closeness. A question and its negation can sit close together, so retrieval often needs reranking or filters.
Practice
Implement Euclidean distance and nearest neighbour easy
Hint
Square the differences, sum, square root. For nearest neighbour, smaller is better.
Show solution
dist(a, b) = sqrt(sum((x - y)^2)). Loop over all items, track the smallest distance seen and its name. O(n * d) time. Note you may skip the square root when only comparing distances.
Show that normalized dot product equals cosine medium
Hint
Divide each vector by its own norm first, then look at what the denominator becomes.
Show solution
If |a| = |b| = 1, then cosine(a, b) = dot(a, b) / (1 * 1) = dot(a, b). So normalize every stored vector once at insert time and use the cheaper dot product at query time.
Detect near-duplicate documents hard
Hint
Brute force is O(n^2) comparisons. What index structure could cut that down?
Show solution
Embed each document, normalize, and for each one query an approximate nearest-neighbour index for neighbours above a similarity threshold (for example 0.95). Group by connected components (union-find). Brute force is O(n^2 * d), which does not scale past tens of thousands of documents.
Interview questions
- What is an embedding? A dense, fixed-length vector learned so that semantically similar inputs are close together.
- Cosine similarity or Euclidean distance? For text, cosine is the common default because it ignores vector length; on normalized vectors they give the same ranking.
- Why not just use keyword search? Keywords miss synonyms and paraphrases; embeddings capture meaning but can miss exact terms like IDs and names, so production systems often combine both (hybrid search).
- Why does dimension matter? More dimensions can capture more nuance but cost more memory and compute, so many teams pick a smaller model or truncate dimensions when quality allows.
Interviewers look for: knowing that the model must match between index and query, and being able to explain the brute-force cost and why approximate indexes exist.
Quick check
You normalize all vectors to length 1. Which is true?
Finished this lesson?
Further resources
- Word embedding (Wikipedia) (article)
- Cosine similarity (Wikipedia) (article)
- Getting started with embeddings (Hugging Face) (article)