Vector stores and search

A vector store holds each chunk's vector with its text and metadata, and answers one question fast: which stored vectors are nearest to this one? That's nearest-neighbour search, and the answer is the top k chunks (k is often somewhere from 3 to 20).

For a few thousand chunks you can compare the question with every vector: exact search, simple and fine. For millions, stores use approximate nearest-neighbour (ANN) indexes, such as HNSW graphs, that check only a small part of the data and return nearly the best matches far faster. You trade a little recall for a lot of speed, and most indexes have a setting for where to sit on that trade-off.

There are dedicated vector databases, search engines with vector support, libraries such as FAISS that run inside your program, and extensions such as pgvector that add vector columns to PostgreSQL. The ideas are the same in all of them; if your data already lives in a database you run, adding vectors there is often the simplest start.

Metadata filters narrow the search: only documents this user may see, only the current policy, only this customer's files. Filtering for access belongs in the retrieval step itself, so a chunk the user shouldn't see never reaches the prompt.

chunks = [
    {"id": 1, "text": "Late fees: 2% a month", "score": 0.82, "year": 2023},
    {"id": 2, "text": "Late fees: 1.5% a month", "score": 0.80, "year": 2025},
    {"id": 3, "text": "Office opening hours", "score": 0.31, "year": 2025},
]
current = [c for c in chunks if c["year"] == 2025]
best = max(current, key=lambda c: c["score"])
print(best["id"], best["text"])

An index is only as good as it is current. When a document changes, re-chunk and re-embed it and delete its old chunks; when one is deleted, delete its chunks. Run this as a pipeline with a record of what was indexed when, not as a script someone remembers to run.