DocsDatabases

Sharding & replication

One node tops out around 50–200M vectors before either RAM or query latency breaks. Past that, the database has to split the index across machines — and that's where ANN gets architecturally interesting.

Why vector sharding is different

A SQL shard owns a key range and a query usually touches one shard. A vector kNN query has no key — the nearest neighbors could live on any shard. So every query is a scatter-gather: ask every shard for its local top-k, merge the results, return the global top-k. Tail latency = the slowest shard.

Query fan-out

1 → N shards per kNN

Top-k inflation

Ask each shard for k or 2k

Bottleneck

Slowest shard (p99 tail)

Write path

Hash on vector ID, not content

Sharding strategies

Random (hash-based) sharding is the default in Pinecone, Qdrant, and Milvus. Each vector is assigned a shard by hashing its ID. Load is balanced and any shard's local distribution looks like the global one, so each shard's top-k is statistically meaningful.

Semantic sharding clusters vectors first (k-means on a sample) and routes each vector to the shard of its nearest centroid. Queries only fan out to the few shards whose centroids are close to the query — fewer shards touched, lower cost per query — at the price of unbalanced load and degraded recall on out-of-cluster queries. Used by Vespa and some Milvus tunings.

Tenant sharding partitions by customer / namespace. Cheap to isolate, but a tenant with 10× the data becomes a hot shard. Common in multi-tenant SaaS.

The scatter-gather query path

python

# Pseudocode — what the router does for one kNN query
async def knn(query_vec, k=10):
    per_shard_k = k * 2   # over-fetch to survive imbalance
    tasks = [
        shard.search(query_vec, k=per_shard_k)
        for shard in shards
    ]
    partials = await asyncio.gather(*tasks)   # parallel fan-out
    merged = heapq.nsmallest(
        k,
        (hit for shard_hits in partials for hit in shard_hits),
        key=lambda h: h.distance,
    )
    return merged

The 2k rule

Each shard returns its local top-k, but the global top-k may be unevenly distributed. Over-fetching 2k per shard recovers almost all of the recall lost to imbalance for less than 2× the bandwidth.

Replication and consistency

Each shard is replicated for HA and read throughput. Replicas of the same shard hold the same vectors but each builds its own ANN graph, because HNSW construction is non-deterministic. Two replicas will return slightly different top-k for the same query — usually fine, but it breaks bitwise reproducibility tests.

Writes follow the standard primary-replica pattern: append to a write-ahead log, replicate to followers, then apply to the in-memory index. ANN index updates are expensive (HNSW insert is O(log N) graph edits), so most systems batch and rebuild segments rather than mutate the live index in place — the same LSM idea SQL engines use.

Operational consequences

p99 latency

Dominated by slowest shard

Resharding cost

Re-embed? Usually no. Re-index? Yes.

Compaction

Segment merge rebuilds ANN graph

Cross-shard filter

Pre-filter on every shard

Don't over-shard

Each shard adds fan-out cost. 8 shards × 5ms p99 is much faster than 64 shards × 8ms p99. Right-size shards to ~10–50M vectors and add replicas, not more shards, when QPS climbs.

Quick Quiz

Test yourself · 3 questions

Q1.

Why does every vector kNN query touch every shard by default?

Q2.

What does the '2k rule' fix?

Q3.

Why do replicas of the same shard sometimes return different top-k?