Hybrid search and reranking

Vector search is good at meaning and bad at exact strings. Ask for invoice INV-20931, error code E1047 or a person's surname, and the vectors for a different invoice number look almost the same. Keyword search (usually ranked with BM25, a formula that rewards rare words that appear in a passage) finds exact terms, codes and names reliably, but misses paraphrases.

Hybrid search runs both and merges the results. A simple, popular way to merge is reciprocal rank fusion (RRF): each list contributes 1 / (k + rank) for every chunk it returns, with k often 60, and chunks are sorted by the total. It only uses ranks, so you don't have to make BM25 scores and cosine scores comparable.

keyword = ["c7", "c2", "c9"]   # best first
vector = ["c2", "c5", "c7"]

scores = {}
for results in (keyword, vector):
    for rank, chunk in enumerate(results, start=1):
        scores[chunk] = scores.get(chunk, 0) + 1 / (60 + rank)

for chunk in sorted(scores, key=scores.get, reverse=True):
    print(chunk, round(scores[chunk], 4))

First-stage search is built for speed: the question and each chunk are turned into vectors separately, then compared. A reranker reads the question and a chunk together and scores how well that chunk answers it. That's much more accurate and much slower, so the usual pattern is to fetch a few dozen candidates with hybrid search and rerank them down to the best handful. Rerankers are often cross-encoder models; a language model can also do it.

Rewriting the question can help too: expand abbreviations, add the product name from the conversation, or turn a follow-up ("and for Canada?") into a complete question before searching.