Vector search is a fine starting point. It is not a complete retrieval strategy. In production RAG systems, semantic similarity routinely returns content that sounds right and is wrong. Hybrid retrieval combining dense embeddings with classic keyword search and a principled re-ranking step - that is what separates demos from systems your customers trust with their data.
The Standard Tutorial Stops Too Early
Most introductions to retrieval-augmented generation end at the same five steps: load documents, chunk them, generate embeddings, store in a vector database, retrieve the top-k semantically similar chunks. That pipeline ships. It does not always answer the question correctly.
Production retrieval quality depends on four decisions the tutorials skip: how you parse documents, how you chunk them, how you combine retrieval signals and how you re-rank what you return.
Why Pure Vector Search Misses
Semantic Similarity Is Not the Same as Intent
Embeddings capture relationships beautifully - "coffee" lives near "espresso". They do not always capture exact constraints. Ask "What was our Q2 2024 EMEA tax split?" and a vector retriever will happily surface the 2023 split, the 2025 outlook and the global aggregate. Recall goes up. Precision collapses.
The Specificity Problem
Vector models are extraordinary at meaning. They are unreliable at exact identifiers. Dates, SKUs, policy numbers, product code names, regulatory clauses - these are the tokens your users actually mention and they are precisely where pure semantic search is least disciplined.
The Hidden Costs of Fuzzy Retrieval
- Latency rises as more low-relevance chunks travel through the pipeline
- Context windows fill with text that crowds out the answer
- Token spend goes up alongside hallucination risk
The Hybrid Approach in One Paragraph
Run vector retrieval for breadth. Run BM25 for exact-match precision. Fusing the two rankings with a method designed for the job - Reciprocal Rank Fusion is the most common - and re-rank the merged set with a cross-encoder when the answer surface is small and the cost of error is high.
A Working Pattern
Step 1: Build Both Indexes - Maintain a dense index (FAISS, pgvector or your provider of choice) alongside a BM25 index over the same corpus. The BM25 index is cheap to maintain and pays for itself the first time a user types an exact identifier.
Step 2: Retrieve in Parallel - Issue both retrievals in parallel. Return their ranked lists separately. Keep them small — top-k around five to ten on each side is enough for most queries.
Step 3: Fuse with Reciprocal Rank Fusion - BM25 and vector similarity scores live on different scales and cannot be compared directly. Reciprocal Rank Fusion uses rank position rather than raw score, which makes the fusion robust:
RRF(d) = Σ over retrievers r of 1 / (k + rank_r(d)), with k as a small smoothing constant (60 is common).
Step 4: Re-Rank When the Answer Has to Be Right - For high-stakes questions, run a cross-encoder over the fused top-k. The latency cost is real. So is the precision lift. Reserve this step for queries where the wrong answer is expensive.
What the Behaviour Looks Like in Production
- BM25 surfaces the support FAQ that contains the exact product name a user mentioned
- Vector search surfaces a security policy that is semantically related but never uses the user's exact phrasing
- The fused result keeps the strongest semantic answer at the top while preserving the literal-match safety net underneath
Final Takeaway
Vector search is a strong default. It is rarely a complete strategy. Whenever your user's intent depends on exact terms — dates, identifiers, policy names, SKUs, clause numbers — hybrid retrieval delivers materially better precision at modest additional complexity. Faster answers, lower token spend, and outputs your reviewers can defend.