Before writing a single line of load-generation code for this benchmark, we had to agree on what<\/em> we were measuring. In a distributed vector database, standard metrics like “latency” and “CPU” behave differently. Here is how we defined our scorecard and the hidden traps we found in each metric.<\/p>\n\n\n\n In a partitioned index, “average” latency (Avg) is a vanity metric. It hides the failures. We focused exclusively on P99 (99th percentile) and P99.9 latency.<\/p>\n\n\n\n “Fast” is useless if the results are not relevant. But at a billion -vector scale, we can’t manually check if the results are “relevant.” We rely on mathematical proximity.<\/p>\n\n\n\n Methodology:<\/strong> We used CPU usage to calibrate our “concurrent workers.” We found the sweet spot was usually slightly higher than the physical core count to mask I\/O waits Ppushing it further caused “thrashing,” where throughput actually dropped<\/em> as we added more load.<\/p>\n\n\n\n When you read whitepapers from vector database vendors, they often gloss over where the data came from. They say “We tested on 1 billion vectors,”<\/em> as if they just downloaded a zip file.<\/p>\n\n\n\n In reality, for a performance engineer benchmarking a hyperscale vector index, finding the dataset is often harder than running the test itself. We didn\u2019t just need \u201cdata.\u201d We needed a billion vectors to find the scale limits y<\/em> vectors that looked like production data to know whether those limits meant anything for a real workload. It turned out those are two different datasets.<\/p>\n\n\n\n We ended up running three datasets. Each one exists because the previous one couldn’t answer a question we were investigating.<\/p>\n\n\n\n SIFT-1B is the source of every billion-scale number in this post. We picked it for one reason: it is the only 1B dataset that ships with published ground truth, so recall at a billion could be checked against the official answers rather than our own. The catch is that it is uint8 at 128 dimensions. Our production workload, and most modern AI workloads, use float32 at 1000+ dimensions. Testing a float-optimized index with 128-dimensional integer data is like testing a Ferrari with diesel fuel. It proves the engine turns over at scale, but the memory-bandwidth patterns are completely different.<\/p>\n\n\n\n LAION-100M fixes the type problem. These are real CLIP image embeddings, float32, with 768 dimensions, and public ground truth. Loading them was the main reason we wrote a Couchbase driver for VectorDBBench. At 100M x 768 dimensions, you are moving ~300 GB of floats, which is enough to see how a float index behaves when the per-vector cost is six times SIFT. It is still a tenth of the scale, and 768 dimensions is still well short of what a 7B-class LLM emits.<\/p>\n\n\n\n BookCorpus-10M is the one we had to build. Nothing public exists at 1536+ dimensions at any interesting scale. The rest of this chapter is about why that is and what it took to build our own.<\/p>\n\n\n\n After finding SIFT and LAION, we looked for one more dataset: a float32 dataset at LLM dimensionality (1536 to 4096) at anything close to a billion scale. There isn’t one.<\/p>\n\n\n\n So the obvious next thought was to generate our own using a real model. But that ran into a different wall.<\/p>\n\n\n\n We gave up on 1B real embeddings and asked a different question: What is the biggest production-like<\/em> dataset we can afford to build once and never build again?<\/p>\n\n\n\n The answer was 10 million vectors at 4096 dimensions. We pulled the BookCorpus dataset from Hugging Face and ran it through Llama 2 (served via Ollama) on a cluster of five GPU machines, each one working on its own slice of the corpus. Instead of writing vectors to files and loading them later, each machine wrote straight into the Couchbase Key-Value service as it went. End to end, the run took 16 hours at roughly 173 vectors\/sec aggregate.<\/p>\n\n\n\n At 4096 dimensions, this data set has higher dimensionality than any public benchmark dataset. It is also uncomfortably close to what real LLM pipelines emit, which was exactly the point.<\/p>\n\n\n\n The other half of the trick is cbbackupmgr. As soon as the load finished, we took a backup of the bucket. The 10M x 4096-dataset is about 164 GB of raw floats, and restoring it takes just 2\u20133 minutes. We have restored it dozens of times since. We paid for 16 hours of GPU time once and never had to pay for it again.<\/p>\n\n\n\n So the ladder is deliberate: 1B SIFT for scale, 100M LAION for real floats, and 10M BookCorpus for LLM dimensionality. Where the results disagree, the 10M set is usually the one that tells you something about your actual workload.<\/p>\n\n\n\n SIFT-1B and LAION-100M come with their answers in the box. The moment we built our own dataset, we lost that, and this turned out to be the most painful part of the whole exercise. To measure performance, we need latency and throughput. To measure correctness, we need Recall@K. And to measure Recall@K, we need ground truth. That\u2019s where things fall apart.<\/p>\n\n\n\n We stopped trying to compute one giant ground truth and partitioned the problem instead.<\/p>\n\n\n\n We split the 10M set into 10 partitions of 1M vectors each, about 16 GB per partition, sized so that brute-force on a single partition fits comfortably on one machine. Every vector carries its partition ID, and we computed exact ground truth for our query set within each partition. Then we query con<\/em> the partition key, so the database only searches that partition, and compare the results against its ground truth.<\/p>\n\n\n\n Two things emerged that weren\u2019t expected:<\/p>\n\n\n\n The trade-off is that this gives you recall within<\/em> a partition, not global recall across the full set. In hindsight, the gap is smaller than it looks: keep the per-partition top-K lists for each query, then merge them into a global top-K with a K-way sort of P x K candidates. The “memory wall” was never really about memory; it was about doing one full pass over the data. The method doesn\u2019t change at 100M or 1B vectors, only the number of partitions does.<\/p>\n\n\n\n SIFT-1B and LAION-100M ship query sets drawn from the same distribution as the base data. Our own dataset shipped nothing, and we quickly realized that you can’t just generate a random query set.<\/p>\n\n\n\n This one turned out to be the easiest to solve, because we had real text sitting underneath the vectors.<\/p>\n\n\n\n First, we held out 1,000 of the generated vectors and used them as queries. Because they came from the same model over the same corpus, they land inside real clusters and have real near neighbors. (Make sure the held-out vectors are no<\/em> in the index. Otherwise, the nearest neighbor of every query is itself at distance zero, making your recall numbers fictional.)<\/p>\n\n\n\n Second, and this is the recommended approach to anyone with access to the source text: we generated 1,000 natural-language questions about passages in the corpus and embedded those within the same model. These look like actual user queries. They are related to the data but were never inserted into the index, making this a harder and more realistic case.<\/p>\n\n\n\n The held-out set tells you the index’s best case. The question set tells you what users will to see. We ended up reporting recall for both, and the gap between them was informative on its own.<\/p>\n\n\n\n When you scale a vector index to a billion items, you inevitably move from a single node to a distributed cluster, and your index gets split into partitions. From that point on, every query has a hidden price tag attached to it that has nothing to do with how fast any single node is.<\/p>\n\n\n\n In a standard distributed vector search, every query is a “scatter-gather” operation. Your query is broadcast to every partition in the cluster. Every partition must wake up, use its CPU to walk its local HNSW\/Vamana graph, and report back. The coordinator then merges the results. That works, and with a well-built engine, it scales: add nodes, the partitions spread out, and QPS goes up. We saw exactly that on the composite index.<\/p>\n\n\n\n But notice what doesn’t<\/em> change when you add nodes. A query that touches 100 partitions still touches 100 partitions. You have more machines doing the same total work per query, which means your throughput per core, and therefore your cost per query, is flat. Scatter-gather lets you buy more throughput. It never lets you get it for free.<\/p>\n\n\n\n The thing that made this concrete for us was watching per-partition CPU during a global search on the 1B SIFT run. A query for one vector lit up every partition in the cluster. The overwhelming majority of them did a full graph walk only to contribute nothing to the final top-K. The answer lived in a handful of partitions; the rest were burning CPU to confirm they didn’t have it.<\/p>\n\n\n\n That is the scatter-gather tax: not that it fails to scale, but that most of the work it scales is wasted.<\/p>\n\n\n\n This is where partition elimination changes the game.<\/p>\n\n\n\n From a benchmarking perspective, partition elimination is the one lever that changes the cost<\/em> of a query rather than just the number of machines paying it.<\/p>\n\n\n\n We are still looking for a standardized solution to these problems. To make 1B+ benchmarking accessible to non-FAANG engineers, the industry needs:<\/p>\n\n\n\n The Metrics That Matter (And How to Measure Them) When you\u2019re testing at the scale of a few million vectors, you can get away with basic scripts. But when you cross the 1 billion vector threshold, “performance” stops being just about speed and starts being about survival \u2013the survival of the memory subsystem, network bandwidth, […]<\/p>\n","protected":false},"author":85776,"featured_media":6268,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"_acf":"","footnotes":"","_ppma_block_editor_authors":""},"categories":[715],"tags":[],"ppma_author":[1057],"class_list":["post-6267","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-vector-search"],"yoast_head":"\n1. Latency: The “Tail” Wags the Dog (P99)<\/h2>\n\n\n\n
\n
2. Recall@K vs. Accuracy@K<\/h2>\n\n\n\n
\n
3. Throughput (QPS)<\/h2>\n\n\n\n
\n
4. Index Build Time<\/h2>\n\n\n\n
\n
5. Memory: RSS vs. Mapped<\/h2>\n\n\n\n
\n
\n
6. CPU Usage: The Concurrency Trap<\/h2>\n\n\n\n
\n
\n
![\"\"](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/5\/2026\/11\/Screenshot-2026-11-04-at-4.10.47-PM.png\")
<\/figure>\n\n\n\nThe Billion-Vector Search for a Dataset (and the Missing Truth)<\/h2>\n\n\n\n
Three datasets, Three jobs<\/h2>\n\n\n\n
Dataset<\/td> Vectors<\/td> Dims\/ type<\/td> Ground truth<\/td> What it was for<\/td><\/tr> SIFT-1B<\/td> 1,000,000,000<\/td> 128-d uint8<\/td> Published by SIFT (10k queries)<\/td> Does the system survive a billion vectors? Build time, memory, P99 under full scatter-gather, partition elimination.<\/td><\/tr> LAION-100M<\/td> 100,000,000<\/td> 768-d float32 (CLIP)<\/td> Ships with the VectorDBBench dataset bundle<\/td> Does it hold up on real<\/em> float embeddings at a dimensionality where memory bandwidth starts to bite?<\/td><\/tr> BookCorpus-10M (ours)<\/td> 10,000,000<\/td> 4096-d float32 (Llama 2)<\/td> Built ourselves<\/td> What does an LLM-scale embedding actually do to the index? Recall real text with real queries.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n 1. The Open Source Desert for High-Dimensional dataset (and Why We Couldn’t Just Generate Our Own)<\/h2>\n\n\n\n
\n
\n
What we actually did<\/h3>\n\n\n\n
2. The Ground Truth Paradox<\/h2>\n\n\n\n
\n
What we actually did<\/h3>\n\n\n\n
\n
3. The Query Set Mismatch<\/h2>\n\n\n\n
\n
What we actually did<\/h3>\n\n\n\n
The “Scatter-Gather” Tax: The Power of Partition Elimination<\/h2>\n\n\n\n
Field Notes: Where the CPU Actually Goes<\/h2>\n\n\n\n
\n
![\"\"](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/sites\/5\/2026\/11\/Screenshot-2026-11-04-at-4.18.12-PM.png\")
<\/figure>\n\n\n\nWhy This Is a Performance Tester’s “Cheat Code”:<\/h2>\n\n\n\n
\n
Future Prospects: What We Need to Solve This<\/h2>\n\n\n\n
\n