Chunking, search and reranking

How you split documents decides what can be found. Chunks that are too big bury the answer in noise and use up the context window; chunks that are too small lose the sentence that explains them. Bedrock Knowledge Bases offers several strategies: fixed-size chunks (a set number of tokens with some overlap between neighbours), hierarchical chunking (small child chunks are searched, and their larger parent chunk is returned for context), semantic chunking (splits where the meaning changes), and no chunking (each file is one chunk, for documents you've already split). For anything else, a Lambda function can do custom chunking during ingestion.

Pick the strategy from the content. Short FAQ entries suit no chunking or small fixed chunks. Long manuals with sections suit hierarchical chunking. Text where topics change mid-page suits semantic chunking. Overlap between fixed chunks keeps a sentence that straddles a boundary findable from both sides.

Vector search finds meaning; keyword search finds exact words. Part numbers, error codes, names and acronyms are where pure vector search fails, because "E-4031" means little as a vector. Hybrid search runs both and combines the scores. OpenSearch Service does this natively, and Knowledge Bases can use hybrid search with stores that support it.

A reranker improves the order. Retrieval returns, say, the top 50 chunks quickly; a reranker model then reads the question with each chunk and scores how well it answers it, and only the best few go into the prompt. Bedrock offers reranker models, and Knowledge Bases can apply one to its results. It costs a little latency and buys a lot of relevance.

Sometimes the question is the problem. Query expansion asks a model to rewrite a short or vague question into a fuller one, or into several versions. Query decomposition splits a compound question ("compare our 2024 and 2025 refund policies") into sub-questions that are each retrieved separately; Knowledge Bases has a query decomposition option, and Step Functions or Lambda can do it in your own pipeline.

Knowledge Bases has two APIs. Retrieve returns the matching chunks with their sources and scores, and your code builds the prompt and calls the model, which gives full control. RetrieveAndGenerate retrieves and answers in one call, with citations that point back to the source documents. Agents and other apps can also reach a vector search through function calling, or through a Model Context Protocol (MCP) server that exposes search as a tool.