I stopped trusting vector search on its own
·3 min read
We had a support assistant that could answer "how do I cancel my plan" beautifully and could not find a customer's order when you pasted the order number in.
Which, if you think about it for two seconds, is exactly backwards from what a support tool needs.
The cause was not the model. We had gone vector-only because every tutorial in 2023 said embeddings were the modern way and keyword search was the old way. That framing is wrong and it cost us about six weeks.
What embeddings are bad at
Dense retrieval compresses a passage into one vector. That is a feature. It is how you get "how do I stop paying you" to match a page titled "Cancelling your subscription" with no shared words.
It is also the weakness. One vector for a whole passage means detail gets averaged away. In practice ours reliably missed:
- Order numbers, SKUs, error codes, anything that looks like a random string
- Rare proper nouns, including our own feature names
- Negation. "Plans that do not include support" would happily match the page about plans that do
- Anything where the exact word mattered more than the vibe of the sentence
BM25 has the mirror-image profile. Nails the identifier, misses the paraphrase.
So you run both. This is not a clever insight, it is just what works, and it took me embarrassingly long to accept because hybrid felt like admitting the old thing was still good.
The numbers on our own set
We built a set of 180 real support questions with the correct source document labelled by hand. Took two people a day and a half. Best money we spent that quarter.
| Setup | Recall@10 | Notes |
|---|---|---|
| Dense only | 61% | Fell over on identifiers |
| BM25 only | 68% | Fell over on paraphrases |
| Hybrid, fused | 82% | Big jump, cheap to add |
| Hybrid plus reranker | 89% | Slower, worth it for us |
Those numbers are ours. Yours will be different, and that is the actual point. Public benchmark scores told me almost nothing useful about our corpus, our acronyms, or our permission structure.
The reranker is the highest leverage thing you are not doing
A cross-encoder reads the question and the candidate together instead of comparing two pre-computed vectors. It is far too slow to run over a whole corpus. It is perfectly fine over the top 50 candidates.
Retrieve broadly, then rank narrowly. Ours added around 180ms and moved recall seven points. For a support tool that trade is obvious. For an autocomplete it would not be.
Where this is going
Google put Gemini Embedding 2 into public preview in March, mapping text, images, video, audio and documents into one space. ColPali-style approaches skip OCR entirely and embed the page as an image, which matters enormously if your documents are the kind where layout carries meaning. Invoices, spec sheets, medical forms.
I have not shipped multimodal retrieval in production yet, so I will not pretend to have strong opinions on it. What I will say is that the direction is consistent: more retrieval channels, not fewer, with something smart doing the merging.
If you take one thing
Build a small labelled set from your own real questions before you pick anything. A hundred rows is enough to expose the failure you did not know you had. Ours showed up in the first twenty.