# Reranking Vector Search: How Rerankers Improve RAG

> Learn how reranking vector search rescoring improves RAG relevance, where rerankers fit, and how to balance retrieval accuracy, latency, and cost.

- Author: Swarnava Dutta (https://swarnava.dev)
- Published: 2026-07-27
- Tags: Reranking Vector Search, Reranking Semantic Search, Reranking AI Search
- Reading time: 11 min (2419 words)
- Canonical: https://swarnava.dev/blogs/reranking-vector-search-rag

---

![Illustration of reranking vector search: A wide conveyor belt spans the frame carrying a loose cluster of fish; midway, a](/images/blogs/reranking-vector-search-rag-hero.jpg)

A few years back I spent most of a Saturday convinced our RAG pipeline had a broken embedding model. The retrieved chunks looked fine individually, cosine similarity scores all sat in a tight, healthy band, but the answers the LLM produced kept ignoring the one paragraph that actually contained the fix for the bug we were chasing. It took an embarrassing amount of debugging to realize the embedding model wasn't broken at all. It just wasn't supposed to be the last word on relevance.

That's the gap reranking vector search fills. Vector retrieval is built for speed and recall: it scans millions of embeddings and hands back a broad set of plausible matches in milliseconds. But "plausible" and "best" are different things, and similarity search alone routinely buries the strongest piece of evidence at position eight instead of position one.

A reranker adds a slower, sharper second pass on top of that candidate set, and this piece walks through exactly how that pass works, when it earns its latency cost, and how to wire it into a production RAG system without it becoming the new bottleneck.

## What Reranking Vector Search Does in Two-Stage Retrieval

Reranking is the second pass in a two-stage retrieval pipeline: it takes a limited set of candidates a vector search already pulled back and rescores each one against the query using a more expensive relevance model. It doesn't touch the rest of the corpus. It just reorders what's already in front of it.

The flow looks like this: a user query gets embedded, that embedding hits the vector index and returns, say, the top 50 or 100 candidates, the reranker scores every one of those query-document pairs individually, and only the highest-scoring handful get passed into the LLM's context window as the final answer's grounding. Fast, broad recall first. Slow, precise scoring second.

### Stage One: Retrieve a High-Recall Candidate Set

Stage one runs an approximate nearest-neighbor search over your embedding index and returns top-k candidates, usually ranked by cosine similarity or dot product. Its job isn't to nail the perfect order - it's to make sure the right chunk is *somewhere* in that set. That means good chunking, sane metadata filters, and sometimes hybrid retrieval mixing keyword and vector signals matter more here than squeezing extra precision out of the embedding model itself, a point I've written about in more depth around [chunking strategies for RAG](/blogs/best-chunking-strategies-rag).

### Stage Two: Rescore, Reorder, and Select Context

The reranker takes each candidate paired with the original query, runs a joint pass over both, and outputs a relevance score. Candidates get sorted by that score, then trimmed to a top-n or filtered by a threshold before assembly into the final context. Nothing new gets fetched - this stage only reshuffles what stage one already found.

## How a Cross-Encoder Reranker Scores Query-Document Pairs

A cross-encoder reranker doesn't compare two pre-computed vectors. It concatenates the query and a single candidate document into one input sequence, tokenizes them together with a separator token, and feeds that whole pair through a transformer in one forward pass. Every token in the query can attend to every token in the document, layer after layer, before the model outputs a single relevance score.

That joint attention is the whole point. Because the model sees query and document together, it can catch that "not recommended for pregnant patients" flips the meaning a plain similarity score would happily ignore. Negation, exact entity matches, unit constraints, date ranges - these are exactly the things independent embeddings smear into a vague nearby-vector, and the Sentence-Transformers project's own cross-encoder documentation describes this joint-attention setup as the reason cross-encoders tend to rank relevance more accurately than bi-encoder cosine similarity, at the cost of speed ([Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillations](https://arxiv.org/abs/2109.13059v4), 2021).

In practice you'll pick from three tiers: off-the-shelf pretrained rerankers (Cohere Rerank, BGE-reranker), models fine-tuned on your own query-document pairs when domain jargon or negative examples matter, or hosted API rerank endpoints you call at inference time with zero infra to manage.

### Bi-Encoder Retrieval vs. Cross-Encoder Reranking

Bi-encoders embed query and document separately, so you can index millions of documents once and compare vectors cheaply forever. Cross-encoders score a pair jointly at query time - no reusable index, no shortcuts, which is exactly why they stay limited to a shortlist. That's the whole tradeoff in one sentence: bi-encoders scale to a corpus, cross-encoders scale to a shortlist, and a solid pipeline uses each for what it's good at.

## Recall vs. Context Windows: Why Top-k Retrieval Falls Short

The naive fix for weak retrieval is to just crank up top-k - pull 20 chunks instead of 5, and hope the answer's in there somewhere. It usually works, in the sense that recall goes up. But every extra chunk you shove into the context window is also extra tokens the model has to read, weigh, and mostly discard.

That's not free. Liu et al.Language models are generally much better at using information placed at the start or end of a long context than information buried in the middle. Pad your context with five near-duplicate chunks and two tangentially-related ones, and you've diluted the signal the generator actually needs.

Cosine similarity alone can't fix this. It struggles with near-duplicate passages that all score nearly identically, keyword overlap that looks relevant but isn't, and multi-constraint queries - "return policy for opened electronics bought online" - where no single chunk satisfies every clause equally well. Reranking splits the difference: retrieve broad and sloppy for recall, then let the sharper second-stage scoring shrink that set to what actually deserves context window space, a tradeoff Pinecone's own writeup on [recall versus context windows](https://www.pinecone.io/learn/series/rag/rerankers/#Recall-vs.-Context-Windows) covers well.

## Reranking vs. Vector Search: Accuracy, Latency, and Cost

Reranking isn't a replacement for vector search - it's a layer on top of it. You still need an index, still need embeddings, still need stage one to do its recall job. The question is never "reranker or vector search," it's whether your pipeline stops at vector-only, adds hybrid keyword-plus-vector retrieval, or goes the full retrieve-then-rerank route.

Cost scales with candidate-set size and document length, since every extra candidate is another forward pass through the reranker. A bigger reranker model or unbatched requests multiply that further, and hosted rerank APIs typically price per-document or per-query, so shrinking your candidate set before reranking pays off twice: less latency, smaller bill.

My own default, and one both Cohere's and BGE's rerank docs point toward as a reasonable starting point, is to retrieve somewhere around 50 to 100 candidates and rerank down to a top-3 to top-10 for context. Don't treat that as gospel; your document length, model choice, and latency budget will push it in either direction, so measure on your own corpus before locking in numbers.

### When the Accuracy Gain Justifies a Reranker

Reranking earns its keep on complex or multi-constraint queries, precision-sensitive answers, heterogeneous corpora, and anywhere your context window is tight enough that ordering really matters. Domain jargon and long-tail phrasing are classic cases. So are corpora full of near-duplicate chunks, where vector similarity alone can't tell one paragraph from another. If your corpus is small and homogeneous, top results already look strong, or latency budgets are generous but tight relevance isn't critical, vector-only retrieval is often good enough.

## Implementing Reranking in RAG from Index to Final Context

Wiring a reranker into an existing pipeline is mostly plumbing, not modeling, but the order of operations matters more than it looks. The sequence looks like this:

- Prepare and chunk documents, then embed and index them as usual.
- Apply metadata filters (date, source, permissions) *before* retrieval, not after - otherwise you're paying the reranker to score documents the user was never allowed to see.
- Retrieve your candidate pool from the vector index.
- Batch those query-document pairs to the reranker.
- Deduplicate near-identical results.
- Assemble the final context in ranked order.

Keep every ID around: document ID, source metadata, the original retrieval score, and the reranker's score. When a user reports a wrong answer, that trail is the only way you'll figure out whether retrieval missed the chunk or the reranker buried it.

Latency discipline matters because reranking sits directly in the request path, right before generation starts. Batch calls where the API allows it, run reranking asynchronously against other pipeline steps when you can, cache scores for repeated queries, and set a timeout with a fallback to the original vector-similarity order rather than failing the request outright.

### Candidate Size, Thresholds, and Context Assembly

Tune candidate pool size for recall, not comfort - I generally start around 75 candidates per query and adjust from there once I've measured recall against a real eval set, rather than assuming a bigger pool always helps. Compare fixed top-n selection against a minimum-score threshold or an adaptive cutoff that stops once scores drop off a cliff. Before final assembly, deduplicate near-duplicate passages, consider expanding to parent documents for fuller context, and order what's left so the generator sees the strongest evidence first.

## How to Evaluate a Transformer Reranker - and When to Skip It

You can't decide whether a transformer reranker is worth its latency tax on vibes alone. Build a real query set first - a few hundred queries with relevance judgments, weighted toward the hard cases. The first time I built one of these sets I deliberately stacked in queries I knew would break naive similarity: negated constraints ("not compatible with"), multi-clause asks, near-duplicate FAQ entries that differed by one word, and the mangled, half-typed phrasing our support team actually pasted in, not the clean questions from the demo script.

Measure retrieval quality directly, not just "does the answer look right":

- **Recall@k** - is the right chunk in the candidate set at all?
- **MRR** - how high does the first relevant result land?
- **NDCG** - how good is the full ranking, not just the top hit?
- **Precision at cutoff** - of what actually makes the final context, how much is relevant?

Then go one level up and check downstream RAG outcomes: answer correctness, faithfulness against cited sources, whether citations point at real evidence, latency at p95 and p99, and cost per query. A reranker that boosts NDCG but tanks p95 latency past your SLA hasn't helped you.

Run the ablation properly - no reranker, versus a couple of candidate models, versus different candidate-set and context sizes. If the lift over vector-only ranking is marginal and the tail latency isn't, skip it or simplify: cheaper model, smaller candidate pool, or hybrid retrieval instead.

## Production Failure Modes: Truncation, Drift, and Tail Latency

Reranking failures rarely announce themselves. Every reranker has a maximum input length, and long documents get silently truncated to fit. I learned this the hard way on a policy-document index: users kept asking about a clause we knew was in the source PDF, the reranker kept scoring the right chunk low, and nothing in the logs looked wrong - no errors, no timeouts, just a quietly mediocre score. It turned out the chunk we'd indexed ran long enough that the actual clause sat past the reranker's token limit and got cut off before scoring ever saw it. Tightening chunk size at index time and adding an overlap window fixed it completely, and it's the first thing I check now when relevance looks inexplicably flat.

Reranker scores aren't calibrated the way you'd want. An illustrative score of, say, 0.82 from one query doesn't mean the same thing as a 0.82 from another, and scores don't transfer cleanly across models, domains, or languages. Treat scores as ordering signals within one query's candidate set, never as an absolute threshold across queries.

Watch for domain shift too: a reranker tuned on generic web text will happily favor confident, fluent phrasing over the authoritative-but-dry source that's actually correct. Multilingual gaps and evaluation sets that go stale as your corpus grows will quietly erode relevance without any alarm firing.

Instrument the whole pipeline: p95 and p99 latency, timeout and empty-result rates, score-distribution drift, and how much reranking actually changes rank order versus vector similarity alone. Ship model or index changes behind versioned evals and canary releases, with a fallback path always ready - Milvus's overview of [when to apply vector reranking](https://milvus.io/ai-quick-reference/what-is-vector-reranking-and-when-should-you-apply-it) is a decent sanity check when you're deciding whether a given corpus needs this machinery at all.

## FAQ

### Can someone explain in detail how a reranker works?

A reranker takes a query and one candidate document at a time and feeds them into a transformer as a single joined input, usually query and document text separated by a special token. The model runs full self-attention across both sequences together, so every query token attends to every document token and vice versa, layer after layer. That joint pass produces one output - a relevance score - rather than two separate embeddings you'd later compare with cosine similarity.

This differs fundamentally from vector search's bi-encoder step, which embeds query and document independently and never lets them attend to each other. The reranker repeats that joint scoring for every candidate in your shortlist, sorts by score, and hands back an ordered list. It's slower precisely because it's doing real cross-attention instead of a cheap vector comparison, which is why it only ever runs on a small candidate set, not your whole index.

### Does reranking replace hybrid search?

No - reranking and hybrid search solve different problems and usually stack together. Hybrid search combines keyword and vector signals at retrieval time to widen recall, while reranking rescores whatever candidates come out of that retrieval stage, hybrid or not.

### How much latency does reranking add?

It depends mainly on candidate-set size, document length, and model size, since the reranker runs a full forward pass per candidate rather than a cheap vector lookup. A small candidate pool with a lightweight model adds a modest, often barely noticeable delay; a large pool with a big model or an unbatched hosted API call can add enough that it shows up clearly in your p95 latency. Measure it on your own stack rather than assuming a fixed number - the variables move too much between setups.


## References

- [Trans-Encoder: Unsupervised sentence-pair modelling through self- and mutual-distillations](https://arxiv.org/abs/2109.13059v4) - Fangyu Liu, Yunlong Jiao, Jordan Massiah et al. (2021)