Embeddings and similarity

A language model only knows what was in its training data and what's in the request. Retrieval-augmented generation (RAG) fills the gap: before the model answers, your code finds the passages of your own documents that are most relevant to the question and puts them in the prompt. The model then answers from them. This course is about doing the finding well.

The usual way to find passages by meaning is embeddings. An embedding model turns a piece of text into a vector: a long list of numbers, often hundreds or a few thousand of them. It's trained so that texts with similar meanings get vectors pointing in similar directions. "How do I return a bird bath?" and "refund policy for garden items" share almost no words but end up close together.

Closeness is usually measured with cosine similarity: the cosine of the angle between two vectors. It's 1 when they point the same way, 0 when they're unrelated (at right angles), and negative when they point apart. It ignores length, so a long passage and a short one can still match. Here it is on tiny made-up three-number vectors:

import math

def cosine(a, b):
    dot = sum(x * y for x, y in zip(a, b))
    return dot / (math.sqrt(sum(x * x for x in a)) * math.sqrt(sum(y * y for y in b)))

question = [0.9, 0.1, 0.3]
refunds = [0.8, 0.2, 0.4]
shipping = [0.1, 0.9, 0.2]
print(round(cosine(question, refunds), 3))
print(round(cosine(question, shipping), 3))

Many embedding models return vectors of length 1 (normalised). Then cosine similarity is just the dot product, which is quicker to compute, and many vector stores use it for that reason.

Two rules matter. Embed the questions and the documents with the same model: vectors from different models aren't comparable. And if you change embedding models, re-embed everything. A score on its own means little; compare scores from the same model on the same data, and set any cut-off from real examples.