Chunking documents

You don't embed a 60-page manual as one vector: one vector can't stand for 60 pages of meaning, and you can't put the whole thing in every prompt. You split documents into chunks, embed each, and retrieve the few that match.

Chunk size is a trade-off. Small chunks match precisely but can lose the context that makes them make sense ("It must be signed by both parties": what must?). Big chunks keep context but blur several topics into one vector and use more of the prompt. A few hundred tokens is a common starting point; test it on your own documents and questions rather than trusting any one number.

Overlap repeats the end of one chunk at the start of the next, so a sentence that straddles a boundary is whole in at least one chunk. Here, chunks of six words that overlap by two:

words = "late fees are two percent per month after the thirty day grace period ends".split()
size, overlap = 6, 2
chunks = [words[i:i + size] for i in range(0, len(words) - overlap, size - overlap)]
for c in chunks:
    print(" ".join(c))

Split along the document's own structure where you can: headings, sections, paragraphs, list items, rows of a table. A chunk that starts mid-sentence or cuts a table in half is hard to find and hard to read. Many teams prepend the document title and section heading to every chunk, so "Section 4.2 Late fees" travels with the text.

Keep metadata with each chunk: the source document, its section, page, date and who's allowed to see it. You'll filter on it (only this year's policies), cite it (page 12), and use it to remove a chunk when its document changes.

Tables, scanned PDFs and slides need care. Text pulled from a PDF can come out with columns interleaved or headers repeated on every page. Look at what your extraction actually produces before you tune anything else: bad text in means bad chunks out.