Reranking

A bi-encoder turns the query and every document into independent vectors and uses a dot product. That's why ANN works — you can index the document vectors once. A cross-encoder concatenates the query and a candidate document and feeds them through a full transformer, letting every token of the query attend to every token of the document. Much higher precision; impossible to pre-index.

Retrieve cheap, rerank expensive. Stage 1 (ANN) returns 50–200 candidates in <20ms. Stage 2 (cross-encoder) re-scores those candidates in 50–200ms and returns the true top-k. Net result: ANN recall + cross-encoder precision.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-base")

def search_then_rerank(q, k_final=5, k_retrieve=100):
    candidates = vector_db.search(embed(q), k=k_retrieve)
    pairs = [(q, c.text) for c in candidates]
    scores = reranker.predict(pairs)               # one forward pass per pair
    ranked = sorted(zip(candidates, scores), key=lambda x: -x[1])
    return [c for c, _ in ranked[:k_final]]

ColBERT splits the difference. It encodes each token of the query and document independently (so document tokens can be indexed), but at query time it computes the max-similarity of each query token against all document tokens and sums them — "late interaction". You get most of a cross-encoder's accuracy with vector-DB-style indexing, at the cost of much larger indexes (one vector per token, not per document).

# Score formula:  Σ_{q ∈ Q}  max_{d ∈ D}  q · d
def colbert_score(Q, D):           # Q: (n_q, dim), D: (n_d, dim)
    sim = Q @ D.T                  # (n_q, n_d)
    return sim.max(axis=1).sum()

Top-k from ANN often contains near-duplicates (three chunks of the same paragraph). Maximal Marginal Relevance re-ranks for diversity: greedily pick the document that maximizes λ · sim(q, d) − (1 − λ) · max sim(d, already_picked). Trades a little relevance for much better coverage in the prompt window.

def mmr(query_vec, candidates, k=5, lam=0.7):
    selected, remaining = [], list(candidates)
    while remaining and len(selected) < k:
        def score(c):
            rel = cos(query_vec, c.vec)
            div = max((cos(c.vec, s.vec) for s in selected), default=0)
            return lam * rel - (1 - lam) * div
        best = max(remaining, key=score)
        selected.append(best); remaining.remove(best)
    return selected

On BEIR-style benchmarks, adding a cross-encoder reranker on top of a strong bi-encoder typically lifts nDCG@10 by 5–15 points absolute. On private RAG corpora the gap is usually larger because off-the-shelf embeddings are weaker on domain text than the reranker, which was trained on a much broader supervision signal.