Hybrid search and reranking
Combine keyword and vector search with reciprocal rank fusion, add a cross-encoder reranker over the top 50, and budget the latency of each stage.
On this page
Vector search finds paraphrase and misses exact tokens. Ask it for error E4012 and it returns chunks about errors in general, because an embedding model blurs a code it has never seen into "some identifier". Keyword search does the opposite: it finds E4012 instantly and misses "how do I get my money back" against a page titled "Refund policy". Neither is the retriever; the retriever is both, with their results fused, and then a slower, more accurate model reordering the handful that survive.
Keyword search is an inverted index with a scoring rule
A keyword index maps each term to the list of chunks containing it — a hash map from term to posting list. A query of four terms fetches four lists and merges them; for a 100k-chunk corpus the lists are short and the merge takes a few milliseconds. BM25 is the scoring rule on top: a chunk scores higher the more often it contains a query term, the rarer that term is across the corpus, and the shorter the chunk is — three factors that between them rank "E4012 timeout on upload" above a chunk that mentions "upload" twenty times. Prefix matching for an autocomplete box is a trie, a different structure for a different query shape, and a search-autocomplete design covers it.
Keyword search is what the corpus's own vocabulary needs: product codes, names, version strings, the exact phrase from a contract. Vector search is what the user's vocabulary needs. A corpus of support tickets wants both.
Fusing two lists by rank, not by score
A BM25 score of 14.2 and a cosine similarity of 0.83 are on unrelated scales, so adding them is meaningless and any weighting is a guess. Reciprocal rank fusion sidesteps the scales by using only each list's ranking: a chunk's fused score is the sum over the lists of 1 ÷ (60 + its rank in that list), with 60 a constant that keeps a first place from dominating. A chunk missing from a list contributes nothing from it.
Worked: chunk A ranks 1st by keyword and 30th by vector; chunk B ranks 5th and 4th. A scores 1/61 + 1/90 ≈ 0.0164 + 0.0111 = 0.0275. B scores 1/65 + 1/64 ≈ 0.0154 + 0.0156 = 0.0310. B wins: good on both lists beats excellent on one. That is the behaviour wanted — a chunk both retrievers agree on is more likely right than one only the keyword index liked — and it needs no tuning. Take the top 100 from each list, fuse, keep the top 50.
A cross-encoder over the top 50
The embedding used for search is a bi-encoder: the query and each chunk are embedded separately, and the comparison is a dot product that can be precomputed for the whole corpus. That is what makes it fast and what makes it approximate — the model never sees the question and the chunk together. A cross-encoder does: it reads the query and one chunk as a single input and outputs one relevance score, with attention running across both. It is markedly better at ranking and cannot be precomputed, since the input includes the query. One forward pass per pair.
So it runs on a short list. Fifty pairs of a 50-token query and a 400-token chunk is 50 × 450 ≈ 22,500 tokens through a small model — around 150 ms batched on an accelerator, closer to a second on CPU. Over 100,000 chunks it would be 45M tokens and minutes; over fifty it is a stage. The reranker keeps the top five, and those are the chunks that enter the prompt.
The reranker can only reorder what it is given. If the chunk with the answer is not in the fused 50, the cross-encoder cannot find it, and its precision at 5 will look fine while recall at 50 is the problem. Measure both: recall at 50 after fusion, which is what the first two stages owe the reranker, and precision at 5 after reranking, which is what the reranker owes the prompt.
Latency per stage
Assume a 100k-chunk corpus, brute-force vectors, and the keyword and vector searches run in parallel. Order-of-magnitude figures:
| Stage | Work | Latency |
|---|---|---|
| Embed the query | one forward pass of the embedding model, one round trip | ~20 ms |
| Keyword search | BM25 over the inverted index, top 100 | ~5 ms, in parallel |
| Vector search | 100k × 1,024 multiply-adds, top 100 | ~30 ms |
| Fuse | reciprocal rank over two lists of 100, keep 50 | under 1 ms |
| Rerank | cross-encoder over 50 pairs, keep 5 | ~150 ms |
| Before the model | ~200 ms |
Then the model: prefill of a 2,500-token prompt and the decode of the answer, which is seconds and dwarfs everything above it. Retrieval is rarely the long pole of a request, but it is a fixed 200 ms added to every request, and it is measured at the tail: the p95 of the reranker on a shared accelerator can be three times its median. Budget the p95 of each stage, as the latency lesson does for the model.
When the budget is tight, the lever is the reranked list, not the retrieved one. Reranking 20 instead of 50 is 60 ms instead of 150 ms and costs almost nothing in precision at 5 if fusion is good; retrieving 20 instead of 100 from each index saves a few milliseconds and silently lowers recall at 50. Cache the query embedding for repeated questions — a support corpus sees the same 200 questions all day — and the 20 ms disappears for them.
The order of improvements
The mistake: a team adds a reranker to a vector-only pipeline because "the results are badly ordered", and recall does not move, because the chunks with the answers were never retrieved. The fix order is set by the ceiling each stage imposes. First chunking and the heading path, which decide what can be found. Then hybrid retrieval, which decides whether it is found — on a corpus with codes and names, adding keyword search to vectors is often worth ten points of recall at 50. Then the reranker, which decides whether it lands in the top five. Each step is checked on the same labelled set before the next is started.
In an interview
What is being tested is whether you know retrieval is a pipeline of stages with a cost and a ceiling each, not one call to a vector store. Draw the stages and put a number on each: embed, two searches in parallel, fuse by rank, rerank the top 50, five into the prompt, about 200 ms in total. Say why fusion is by rank ("the scores are on different scales") and why the reranker runs on 50 and not 100,000 ("one forward pass per pair; it cannot be precomputed").
The mistake that loses points is bolting a reranker onto a pipeline whose recall at 50 nobody measured — the reranker reorders the wrong fifty, the answer never enters the prompt, and the candidate reports better ordering of chunks that do not contain it.
Check yourself
1. The corpus is 400,000 support tickets full of order numbers, SKUs and customer names. Vector only, keyword only, or hybrid?
Hybrid, and the keyword side carries the weight: an order number has no paraphrase, so an embedding of it is noise, and a question quoting one must hit the exact token. Vectors still earn their place on "my parcel never arrived" against tickets that say "delivery failed". Run both, fuse by rank, and expect the keyword list to dominate the fused top 50 for any query containing an identifier.
2. Retrieval has a 100 ms budget at p95. Which stage do you cut?
The reranker, which is three-quarters of the 200 ms. Rerank the top 15 instead of 50 — about 45 ms, which brings the table to about 95 ms — and verify precision at 5 on the labelled set. Keep the two retrievers at 100 each: they cost 30 ms in parallel and are what protects recall. The table's figures are typical, not p95, so 95 ms is no headroom: cache embeddings for repeated questions to take the next 20 ms off them, and measure each stage's p95 before calling the budget met.
3. Precision at 5 is 0.9 and users still say the bot misses things. What number do you look at?
Recall at 50 after fusion. Precision at 5 only says the reranker chose well among what it was given; if the answering chunk is in the fused 50 for only 70% of questions, the pipeline's ceiling is 70% and the reranker is not where the problem is. Then chunking and the keyword side, in that order.