{"id":17138,"date":"2025-05-22T09:54:28","date_gmt":"2025-05-22T16:54:28","guid":{"rendered":"https:\/\/www.couchbase.com\/blog\/?p=17138"},"modified":"2025-06-13T22:45:04","modified_gmt":"2025-06-14T05:45:04","slug":"semantic-similarity-with-focused-selectivity","status":"publish","type":"post","link":"https:\/\/www.couchbase.com\/blog\/pt\/semantic-similarity-with-focused-selectivity\/","title":{"rendered":"Similaridade sem\u00e2ntica com seletividade focada"},"content":{"rendered":"

Why does semantic search need selectivity?<\/h2>\n

Up until now, we\u2019ve viewed a vector embedding as a complete, stand-alone entity – focused entirely on the meaning it encodes. While this enables semantic search, often with a high degree of similarity<\/a>, it remains limited to similarity among the query and dataset embeddings.<\/p>\n

The vector similarity search offering could not be relied upon to satisfy exact predicates. Pre-filtering aims to address exactly this gap – by searching for similar vectors only among those satisfying some filtering criteria.<\/p>\n

It\u2019s the embedding equivalent of limiting your search, whether for a job or a property, to a location. Say, you want a beachfront property in a specific state. You also want to limit your search to those with three bedrooms or more. Combing through listings without a method to filter for these criteria is almost unfeasible given a lot of listings.<\/p>\n

With pre-filtering, you can limit your search to a specific location by restricting the search space to eligible properties via geospatial and numeric queries. A vector similarity search for \u201cbeach-front\u201d, \u201cbeachside\u201d, \u201cwaterfront\u201d properties will be performed over this limited subset.<\/p>\n

\"\"<\/p>\n

Pre-filtering will allow users to specify filter queries as part of the kNN attribute in the query, only the results of which will be considered as eligible to be returned by the kNN query. Simply put, the user can now use the familiar FTS query syntax to restrict the documents over which a kNN search will be performed<\/i>.<\/p>\n

When to apply pre-filtering?<\/h2>\n

As the name suggests, the embeddings are metadata-filtered before <\/i>the similarity search. This is different from post-filtering where a kNN search is followed by metadata-filtering. Pre-filtering offers a much higher chance of returning k hits, assuming there are at least those many documents which pass the filtering stage.<\/p>\n

\"\"<\/p>\n

\"\"<\/p>\n

How this works<\/h2>\n

Before we get into the guts of pre-filtering with kNN, let\u2019s understand how a vector and full-text index are co-located in the Search service index. Each Search index has one or more partitions, each of which has one or more segments. Each of these segments is a file, and the file is divided into separate sections, with one per type of index. Viewing a segment as a self-contained unit with both text and vector content of a batch of documents<\/i> indexed will be useful for understanding how pre-filtering works at the segment level.<\/p>\n

\"SemanticA key consideration when building this was that it should be predicate-agnostic<\/i> at search time. Essentially, this means that the filtering at the vector index should work the same way, irrespective of the full-text predicate. A text predicate over a text field should be no different from a numerical predicate over another.<\/p>\n

To this end, document numbers, identifiers unique to a document, are used to demarcate which documents are eligible for the kNN query. Every FTS query, whether text, numeric, or geospatial, boils down to identifying hits by their document number. Using document numbers means we do not have to change our indexing strategy for vectors, and limit it to a search time change.<\/p>\n

Phase 1: Metadata filtering<\/h3>\n

Since a segment is essentially an immutable batch of documents, with their text and vector content indexed, the full-text metadata query returns all eligible documents at the segment level<\/i>. Their document numbers are then passed to the vector index to retrieve the nearest eligible vectors.<\/p>\n

\"Semantic<\/p>\n

Phase 2: kNN search<\/h3>\n

\"SemanticThe current algorithm which picks the nprobe<\/em> clusters closest to the query vector in an IVF index is essentially flying blind<\/i>\u00a0since it does not account for proximate clusters with few to no eligible documents. At search time, the selection of clusters to probe now has to account for distribution of metadata-filtered hits<\/i> across the index.<\/p>\n

A cluster closer<\/em>\u00a0to the query vector might have far fewer eligible hits than one much farther.<\/p>\n

Taking into account filter hit distribution while keeping recall high means that we cannot scan clusters solely on the basis of filter hit density. What it does mean is that we attempt to minimize distance computations for ineligible vectors, even as we scan as many clusters as needed to return the k nearest neighbors<\/em>.<\/p>\n

Earlier, nprobe<\/em> was defined as the absolute limit<\/i> to the number of clusters scanned. Now, it\u2019s more of a minimum number of clusters to scan assuming fewer clusters have enough eligible vectors. In cases of sparse filter hit distribution, where each cluster has relatively few eligible vectors, our quest for k nearest neighbours can lead us to scan far more than nprobe clusters<\/i>. Both filtered and unfiltered kNN scan clusters in increasing order of distance from the query vector, with the difference lying in scanning a subset of vectors across a potentially larger number of clusters.<\/p>\n

eligible_clusters: clusters with at least 1 filtered hit\r\nif eligible_clusters < nprobe :\r\n    scan all clusters\r\nelse:\r\n    total_hits(1,n): Cumulative count of eligible hits in closest 'n' eligible clusters\r\n    if total_hits(1,nprobe) >= k:\r\n        scan nprobe eligible clusters\r\n    else:\r\n        while total_hits(1,n) < k:\r\n            n++\r\n        scan n eligible clusters<\/pre>\n
Once the vector index returns the most similar vectors at a segment level, these are aggregated at a global index level similar to how it\u2019s done for kNN without filtering.<\/p>\n
How to use this<\/h3>\n
Let\u2019s take a bucket, landmarks<\/em>. This is a sample document:<\/p>\n
{\r\n\u00a0\u00a0\"title\": \"Los Angeles\/Northwest\",\r\n\u00a0\u00a0\"name\": \"El Cid Theatre\",\r\n\u00a0\u00a0\"alt\": null,\r\n\u00a0\u00a0\"address\": \"4212 W Sunset Blvd\",\r\n\u00a0\u00a0\"directions\": null,\r\n\u00a0\u00a0\"phone\": null,\r\n\u00a0\u00a0\"tollfree\": null,\r\n\u00a0\u00a0\"email\": null,\r\n\u00a0\u00a0\"url\": null,\r\n\u00a0\u00a0\"hours\": null,\r\n\u00a0\u00a0\"image\": null,\r\n\u00a0\u00a0\"price\": null,\r\n\u00a0\u00a0\"content\": \"Built around turn of the century and, after several reincarnations, offers one of the only dinner theater options left in Los Angeles. The menu is heavily Spanish and the shows differ depending on the night and range from flamenco performances to tongue-in-cheek burlesque.\",\r\n\u00a0\u00a0\"geo\": {\r\n\u00a0\u00a0\u00a0\u00a0\"accuracy\": \"ROOFTOP\",\r\n\u00a0\u00a0\u00a0\u00a0\"lat\": 34.0939,\r\n\u00a0\u00a0\u00a0\u00a0\"lon\": -118.2822\r\n\u00a0\u00a0},\r\n\u00a0\u00a0\"activity\": \"do\",\r\n\u00a0\u00a0\"type\": \"landmark\",\r\n\u00a0\u00a0\"id\": 35034,\r\n\u00a0\u00a0\"country\": \"United States\",\r\n\u00a0\u00a0\"city\": \"Los Angeles\",\r\n\u00a0\u00a0\"state\": \"California\",\r\n\u00a0\u00a0\"embedding_crc\": \"fa6edfd97ffa665b\",\r\n\u00a0\u00a0\"embedding\": [-0.003134159604087472, -0.020280055701732635,.... \u00a0 -0.014541691169142723]\r\n}<\/pre>\n
Create an index, test,<\/em> which indexes the embedding<\/em>, id<\/em> and city<\/em> fields.<\/p>\n
My first query is an embedding of the Royal Engineers Museum in Gillingham:<\/p>\n
{\r\n\u00a0\u00a0\"knn\": [\r\n\u00a0\u00a0\u00a0\u00a0{\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\"k\": 10,\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\"field\": \"embedding\",\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\"vector\": [\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a00.0022478399332612753,\r\n        ....\r\n      ]\r\n\u00a0\u00a0\u00a0\u00a0}\r\n\u00a0\u00a0],\r\n\u00a0\u00a0\"explain\": true,\r\n\u00a0\u00a0\"size\": 10,\r\n\u00a0\u00a0\"from\": 0\r\n}<\/pre>\n
The closest hits I get are the London Fire Brigade Museum, the Verulamium Museum in St Albans and the RAF Museum in London.<\/p>\n
We now want to search for similar museums in Glasgow, that is, the city field should have a value Glasgow<\/em>.<\/p>\n
Here\u2019s how the filtered query looks, with the filter<\/strong> clause added:<\/p>\n
{\r\n\u00a0\u00a0\"knn\": [\r\n\u00a0\u00a0\u00a0\u00a0{\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\"k\": 10,\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\"field\": \"embedding\",\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\"vector\": [\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a00.0022478399332612753,\r\n        ....\r\n  ],\r\n  \"filter\": {\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\"match\": \"Glasgow\",\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\"field\": \"city\"\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0}\r\n\u00a0\u00a0\u00a0\u00a0}\r\n\u00a0\u00a0],\r\n\u00a0\u00a0\"explain\": true,\r\n\u00a0\u00a0\"size\": 10,\r\n\u00a0\u00a0\"from\": 0\r\n}<\/pre>\n
As expected, the results are now limited to those in Glasgow – the closest being the Kelvingrove Art Gallery and Museum and the Riverside Museum.<\/p>\n
As seen in this example, a filtered kNN query offers the advantage of selecting the documents for a similarity search using good, ol\u2019 FTS queries.<\/p>\n
Keep learning<\/h2>\n