Most teams building semantic search learn the same lesson late. What ultimately controls recall, query latency, and hardware cost isn’t the embedding model or which database they choose. It’s the vector index configuration, which is typically set early and rarely revisited.
By the time a poor choice shows itself, the damage is already done. Recall has been drifting for weeks. A cluster runs out of memory under peak load. An upgrade that looked routine turns into a rewrite. None of these problems present themselves at creation time, which is a large part of why they keep happening.
So it is worth slowing down on the step everyone rushes past: what actually happens when you create a vector index, why that single step carries so much weight, and how the wrong choice can quietly become a production incident.
Creating an index is a commitment, not a setting
Adding a vector index is not a simple structural change, it has to be trained. The build process samples your data, learns a set of cluster centroids, decides how each vector will be compressed, and lays out a graph structure either in memory or on disk. Once the index is built, those decisions are effectively frozen into it. Changing them later almost always means dropping the index and rebuilding it. On a large dataset, that rebuild can take hours and consume a significant amount of CPU and I/O.
That is the part most teams underestimate. The CREATE statement looks like one tidy block of configuration. What it really does is commit the application to a memory model, a compression scheme, a query syntax, and a service boundary – and all four of those are painful to walk back.
A typical creation statement carries a dozen consequential parameters. The vector dimension has to match the embedding model exactly. The similarity metric (cosine, dot product, or Euclidean) has to match how that model was trained. The IVF cluster count controls how finely the space gets partitioned. The quantization setting decides how aggressively each vector gets compressed, where SQ8 trims memory by roughly 75% while keeping recall high, and SQ4 trims it further at a real accuracy cost. The training sample size determines how representative the cluster centroids will be. Every one of these is a lever that affects the recall, latency, and cost of every query the index will ever serve.
The trade-off you cannot configure your way out of
Whatever you create, it lands somewhere inside a three-way tension. You can favor two of these corners at a time, but never all three at once.
Push for higher recall and you search more clusters, evaluate more candidates, and add a reranking pass, so latency rises and cost climbs with it. Push for lower latency and you search fewer clusters and compress harder, so recall slips. Try to cut the infrastructure bill and you push the index onto disk or quantize more, which trades against the other two. There is no setting that wins on all three. The index type and the parameters chosen at creation decide which corner of this trade-off space is even reachable, and no amount of later tuning can move you beyond it.
Three indexes, three different bets
Couchbase offers several vector index strategies, and they are not easily interchangeable. Each one makes a different structural bet about where the data lives and how a query flows through it, and that bet is locked in the moment you create the index.
A Hyperscale Vector Index is built for the billion-vector range. It splits routing from search. A small graph of cluster centroids stays in RAM and routes each query to the few clusters that matter, while the actual vectors sit on SSD in a disk-friendly Vamana graph. The reason this matters is simple arithmetic. One billion vectors at 768 dimensions is around 3 TB of raw float data, and no sensible server holds that in memory. By keeping only the routing layer resident, this index serves enormous corpora on a fraction of the RAM and holds its latency steady as the data grows.
A Composite Vector Index makes the opposite bet: filter first. It’s a global secondary index that carries a vector column alongside the scalar fields, using IVF clustering. If the query also has constraints on scalar fields, say a tenant and a category, those filters are applied before the vector search runs. When the filter is narrow enough to keep only a small slice of the corpus, the expensive nearest-neighbor work runs over that slice instead of the whole dataset. This is the right tool for multi-tenant applications and strongly partitioned catalogs. The catch is memory pressure. To hit low latency you want the working set resident in the Index Service’s memory quota, and if the indexer is pushed past that quota it stops serving scans until memory is freed. At 100M vectors with 1536 dimensions, the quantized vector payload alone (SQ8, one byte per dimension) is roughly 144 GB before centroids, graph structure, and scalar columns.
An FTS Search Vector Index lives in the Full-Text Search service and is the only one that fuses keyword relevance and vector similarity inside a single query. If the product needs fuzzy matching, stemming, and autocomplete alongside semantic search, this is the natural fit, and it is the simplest to operate. Its practical ceiling sits near 100M vectors, beyond which its memory-mapped storage starts to strain.
Where the wrong choice turns into an incident
The danger is not that one index is good and another is bad. It is that the failure modes are quiet, and the cost of a poor design decision often appears much later. A few patterns show up again and again.
There is the simplicity trap. A team picks FTS because it is the easiest to stand up. It performs beautifully at 20M vectors. At 130M, it is quietly fighting memory pressure, and moving off it means crossing a service boundary and rewriting every query from the FTS search syntax to the index service syntax. The easy choice at creation becomes the expensive migration later.
There is the scale-fixation trap. A team picks the Hyperscale Index for its billion-scale ambitions, but 95% of real traffic is tenant-filtered. They left the filter-first advantage on the table and are paying SSD latency for searches that could have run over a fraction of a percent of the data in RAM.
And there is codebook drift, which is the one almost nobody sees coming. Every IVF-based index trains its cluster centroids on a data sample taken at build time. As new data flows in and drifts away from that original sample, the centroids stop being representative and recall slowly decays. No error fires. No alert trips. Results just get gradually worse. The fix is a periodic rebuild, and it is trivial once you know the hazard exists – something most teams do not realize until it has already cost them.
All of this is why decisions at creation cannot be a simple checkbox at the end of a project. The right index is a function of current scale, filter selectivity, keyword needs, existing infrastructure, and projected growth all at once, and no single signal settles it. The logic runs through a short sequence of questions, and the order in which they are asked matters.
The first question addresses whether the workload needs real keyword search alongside semantic search. If it does and the corpus sits under 100M vectors, the FTS index covers both in one place. If the corpus runs past that ceiling, the answer shifts to a hybrid setup that pairs a Hyperscale index for the vector side with a separate FTS index for the keyword side.
If keyword search is not in the picture, scale comes next. Under 100M vectors most options remain open. Beyond 100M, the FTS index drops out for memory purposes and the contest narrows to the Composite and Hyperscale indexes.
Filter selectivity usually breaks that tie. When queries carry a narrow scalar filter that keeps only a thin slice of the corpus eligible, the Composite index and its filter-first execution win. When filters are broad or absent, the Hyperscale index is the better option, since the Composite index would pay for a scalar layer it never gets to use.
Growth underpins the whole setup. A dataset at 20M today with 150M on the roadmap should be created with that migration path already in mind, not just the shape of the data right now. Each of these questions represents a place where a reasonable engineer can take a reasonable-looking wrong turn, and the answers interact with one another. The full picture includes even more factors, such as available RAM, the cost of a second-stage LLM reranking pass, and the migration friction between one index type and another.
The right answer starts with the right questions
There is a pattern to how good recommendations are made. A solutions engineer sitting with a customer does not open by talking about indexes. They open by asking questions, and the questions are deliberately ordered. They need to understand the use case before they address configuration, because an index recommendation made without that context is just a guess dressed up as expertise.
The use case has layers that need to be surfaced. What is the application actually doing with the retrieved results? A RAG pipeline feeding an LLM needs different recall characteristics than a real-time product recommendation widget. How is the data partitioned in practice, not just in theory? A team might describe their workload as “multi-tenant,” but if one tenant holds 80% of the vectors, the filter selectivity math looks completely different from the uniform case. Is the current scale the real number, or is it a comfortable approximation of something that is actually growing? The answer matters because the right index for today’s 30M vectors and the right index for next year’s 200M vectors are sometimes different objects, and the migration between them has a cost that should factor into the design decision now.
Then there are the infrastructure questions, which are often the most overlooked. Does the cluster already run the Index Service for other workloads? Does it have a Full-Text Search service provisioned? The technically superior index choice is sometimes the wrong practical choice if it requires standing up a service that does not exist yet, adding nodes, and retraining the operations team. A slightly less optimal index that runs inside an already deployed service can outperform the theoretical winner by months.
None of this is complicated once someone knows to ask the right quesitons. The difficulty is that these questions rarely get asked together, systematically, before the CREATE statement runs. Engineers tend to anchor on the most salient signal – usually scale or the latest technology they read about – and work outward from there. That anchoring is where mismatches originate.
An advisor built around asking the right questions
The Vector Index Advisor was built around the observation that the quality of the recommendation is entirely downstream of the quality of the information gathered before making it. The agent does not open with an answer. It opens with a conversation, working through the same diagnostic sequence a solutions engineer would use: asking about scale, filter patterns, keyword requirements, existing services, and expected growth, with concrete options to anchor each answer.
The reason for concrete options matters. A question like “How selective are your filters?” will get a vague answer from most people. A question that offers “less than 5% of the corpus, between 5 and 20%, more than 20%, or no filters” gets a specific one. The specificity is what makes the downstream reasoning reliable. Each answer narrows the decision space in a measurable way, and the agent tracks what it has confirmed, what is still open, and what implications are already derivable from what it knows.
Underneath the conversation, the agent runs deterministic checks against real data rather than relying on the model to estimate. Recommendations are compared against a library of real-world workloads to identify cases where a similar situation was resolved differently. Benchmark data from actual Couchbase test runs informs the tuning guidance so that the numbers given are grounded in measured performance rather than rules of thumb.
The full reasoning trace is visible step by step in the interface, which is more than just a debugging aid. It means the recommendation comes with its reasoning attached, allowing the person reading it can follow the logic, challenge an assumption, or update an answer. A recommendation that can be interrogated builds more confidence than one that simply arrives.
The point
Vector search has a deceptive curve. Getting something working is quick. Getting the right thing working at scale is a domain of its own, and the index created at the start sets the ceiling on recall, latency, cost, and migration pain long before anyone tunes a parameter. Most expensive surprises, traced back through incident reports stem at the same place: a decision at creation made without the full picture of the workload it was designed to support. Treat index creation as the architectural decision it actually is, ask the questions that surface the real use case before writing the configuration, and most of those surprises never happen.
Leave a comment
You must be logged in to post a comment.