DevPath
Tracks

Vectors and Embeddings

How text becomes a list of numbers, why distance in that space means similarity, and how to compare vectors.

intermediate40 min#vectors#embeddings#cosine-similarity#rag

Prerequisites: Arrays and Dynamic Arrays

Why it matters

Search by keyword fails when people use different words for the same idea ("car" vs "automobile"). Embeddings fix this by turning text, images, or any item into a vector: a fixed-length list of numbers where similar meanings land close together. They power semantic search, recommendations, duplicate detection, clustering, and the retrieval step of RAG.

Intuition

Imagine a map where every sentence is a pin. Sentences about cooking cluster in one region, football in another, and "pasta recipe" sits near "how to boil spaghetti" but far from "offside rule". An embedding model is the cartographer: it reads a sentence and returns the pin's coordinates. A real map has 2 coordinates; embeddings typically have hundreds or thousands (for example 384, 768, or 1536), so the "map" has that many dimensions. We cannot draw it, but distance still works.

The coordinates have no human-readable meaning individually. Meaning lives in the geometry: nearby points are similar.

dimension 1dimension 2pasta recipeboil spaghettioffside rulered cardquery: cook noodles
CookingFootballQuery
A 2D slice of an embedding space. Meaning becomes distance: the query lands next to the cooking texts.

How it works

Every use of embeddings follows the same pipeline: text goes through a model once, and the resulting vector is stored or compared.

Texta sentence or chunk
Embedding modelsame model every time
Vectore.g. 384 numbers
Compare or storesimilarity, index
The model sits in the middle of the pipeline. Documents and queries both pass through the same model, so their vectors land in the same space.
  1. Embedding model: a neural network trained so that related inputs produce nearby vectors. You send text in and get back a vector like [0.12, -0.03, ...].
  2. Same model for everything: vectors from different models live in different spaces and cannot be compared. Embed your documents and your queries with the same model.
  3. Compare with a similarity measure:
    • Dot product: sum of element-wise products. Larger means more similar (when magnitudes matter).
    • Cosine similarity: the dot product divided by both lengths. It measures the angle between vectors, ignoring length, and ranges from -1 to 1. This is the usual default for text.
    • Euclidean distance: straight-line distance. Smaller means more similar.
  4. Normalize once, then use the dot product: if every vector is scaled to length 1, cosine similarity equals the dot product, which is cheaper to compute at scale.

The dot product, term by term, using the vectors from the code below. Each column multiplies, then the products are added.

dim 1dim 2dim 3sum
query
0.85
0.15
0.05
pasta recipe
0.9
0.1
0
query x pasta
0.765
0.015
0
0.78
offside rule
0
0.1
0.9
query x offside
0
0.015
0.045
0.06
Multiply matching positions, then add. The query shares its large first value with the pasta recipe, so the sum is 0.78. It barely overlaps with the offside rule, so the sum is only 0.06.
abc
Cosine similarity compares angles. a and b point almost the same way (similar, cosine near 1); c points away (cosine near or below 0).
OperationTimeNotes
Dot product of two d-dim vectorsO(d)One pass
Brute-force search over n vectorsO(n * d)Exact; fine up to ~100k vectors
Approximate search (HNSW, IVF)~O(log n)Next lessons: faster, slightly inexact

Code: similarity from scratch

We use tiny hand-made 3-dimensional vectors so you can check the math by hand. Real embeddings come from a model, but the comparison code is identical.

import math
 
def dot(a, b):
    return sum(x * y for x, y in zip(a, b))
 
def norm(a):
    return math.sqrt(dot(a, a))
 
def cosine(a, b):
    return dot(a, b) / (norm(a) * norm(b))
 
def top_k(query, items, k=2):
    """Brute-force nearest neighbours: score everything, keep the best k."""
    scored = [(cosine(query, vec), name) for name, vec in items.items()]
    return sorted(scored, reverse=True)[:k]
 
docs = {
    "pasta recipe":     [0.9, 0.1, 0.0],
    "boil spaghetti":   [0.8, 0.2, 0.1],
    "offside rule":     [0.0, 0.1, 0.9],
}
query = [0.85, 0.15, 0.05]  # pretend this is "how to cook noodles"
 
for score, name in top_k(query, docs):
    print(f"{score:.3f}  {name}")

The two cooking documents score near 1 and "offside rule" is left out of the top 2. This top_k function is already a working (if tiny) vector search; the next lessons make it fast with indexes.

Common pitfalls

  • Comparing across models or dimensions: the lengths may even match by accident. The results are meaningless.
  • Forgetting to normalize: with dot product on un-normalized vectors, long vectors win regardless of meaning.
  • Embedding huge documents as one vector: one vector cannot capture a 50-page document. Split into chunks (covered in the RAG lesson).
  • Assuming "similar" means "relevant": an embedding finds topical closeness. A question and its negation can sit close together, so retrieval often needs reranking or filters.

Practice

Implement Euclidean distance and nearest neighbour easy

Hint

Square the differences, sum, square root. For nearest neighbour, smaller is better.

Show solution

dist(a, b) = sqrt(sum((x - y)^2)). Loop over all items, track the smallest distance seen and its name. O(n * d) time. Note you may skip the square root when only comparing distances.

Show that normalized dot product equals cosine medium

Hint

Divide each vector by its own norm first, then look at what the denominator becomes.

Show solution

If |a| = |b| = 1, then cosine(a, b) = dot(a, b) / (1 * 1) = dot(a, b). So normalize every stored vector once at insert time and use the cheaper dot product at query time.

Detect near-duplicate documents hard

Hint

Brute force is O(n^2) comparisons. What index structure could cut that down?

Show solution

Embed each document, normalize, and for each one query an approximate nearest-neighbour index for neighbours above a similarity threshold (for example 0.95). Group by connected components (union-find). Brute force is O(n^2 * d), which does not scale past tens of thousands of documents.

Interview questions

  1. What is an embedding? A dense, fixed-length vector learned so that semantically similar inputs are close together.
  2. Cosine similarity or Euclidean distance? For text, cosine is the common default because it ignores vector length; on normalized vectors they give the same ranking.
  3. Why not just use keyword search? Keywords miss synonyms and paraphrases; embeddings capture meaning but can miss exact terms like IDs and names, so production systems often combine both (hybrid search).
  4. Why does dimension matter? More dimensions can capture more nuance but cost more memory and compute, so many teams pick a smaller model or truncate dimensions when quality allows.

Interviewers look for: knowing that the model must match between index and query, and being able to explain the brute-force cost and why approximate indexes exist.

Quick check

You normalize all vectors to length 1. Which is true?

Finished this lesson?

Further resources

Related lessons