![\"\"](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/08\/Screen-Shot-2020-08-10-at-5.25.41-PM.png\")
Full-Text Search refers to techniques for searching text content within a document or a collection of documents that hold textual content. A Full-Text search engine examines all the textual content within documents as it tries to match a single search term or several terms, text analysis being a pivotal component.<\/p>\n
You\u2019ve probably heard of the most well-known Full-Text Search engine: Lucene<\/span><\/a><\/strong> with Elasticsearch<\/span><\/a><\/strong> built on top of it. Couchbase\u2019s Full-Text Search (FTS<\/strong>) Engine<\/span> is powered by Bleve<\/span><\/a><\/strong>, and this article will showcase the various ways to analyze text<\/strong>\u00a0within this engine.<\/p>\n Bleve is an open-sourced text indexing and search library implemented in Go, developed in-house at Couchbase.<\/p>\n Couchbase\u2019s FTS engine supports indexes that subscribe to data residing within a Couchbase Server<\/b><\/span> and indexes data that it ingests from the server. It\u2019s a distributed system – meaning it can partition data across multiple nodes in a cluster and searches involve scattering the request and gathering responses from across all nodes within the cluster before responding to the application.<\/p>\n The FTS engine distributes documents ingested for an index across a configurable number of partitions and these partitions could reside across multiple nodes within a cluster. Each partition follows the same set of rules that the FTS index is configured with – to analyze and index text into the full-text search database.<\/p>\n The text analysis<\/b> component of a Full-Text search engine is responsible for breaking down the raw text into a list of words – which we\u2019ll refer to as tokens. These tokens are more suitable for indexing in the database and searching.<\/p>\n Couchbase\u2019s FTS Engine handles text indexing for JSON documents. It builds an index for the content that is analyzed and stores into the database – the index along with all the relevant metadata needed to link the tokens generated to the original documents within which they reside.<\/p>\n An Inverted index<\/span><\/a><\/strong> is the data structure chosen to index the tokens generated from text, to make search queries faster. This index links every token generated to documents that contain the token.<\/p>\n For example, take the following documents ..<\/p>\n The inverted index for the tokens generated from the 2 documents above would resemble this..<\/p>\n Here\u2019s a diagram highlighting the components of the full-text search engine ..<\/p>\n The components of a text analyzer can broadly be classified into 2 categories:<\/p>\n Couchbase’s engine further categorizes filters into:<\/p>\n Before we dive into the function of each of these components, here\u2019s an overview of a text analyzer ..<\/p>\n A tokenizer is the first component to which the documents are subjected to. As the name suggests, it breaks the raw text into a list of tokens. This conversion will depend on a rule-set defined for the tokenizer.<\/p>\n Stock tokenizers ..<\/p>\n Take this sample text for an example: “this is my email ID: <\/i>abhi123@cb.com”<\/p>\n A couple of configurable tokenizers ..<\/p>\n For example:<\/p>\n Character filters are to remove or replace undesirable characters.<\/p>\n Stock character filters ..<\/p>\n A configurable character filter ..<\/p>\n For example:<\/p>\n Token filters accept a token stream provided by a tokenizer and make modifications to the tokens in the stream. Most common forms of token filtering are normalizing and stemming.<\/p>\n Several stock token filters, here are a few prominent ones ..<\/p>\n Configurable token filters ..<\/p>\n With Couchbase\u2019s Full-Text Search engine, the analyzers and all their components work on text that constitutes field values within JSON documents. They do not work on field names.<\/p>\n Consider the JSON document:<\/p>\n For the document, analyzers can be defined to work on “value1”, “value2”, “value3”, “value4”, “value5” and “value6”.<\/p>\n Couchbase offers several stock analyzers ..<\/p>\n <\/p>\n Here are a couple of examples ..<\/p>\n <\/p>\n Test the behavior of our stock analyzers and your custom built analyzers here ..<\/p>\n https:\/\/bleveanalysis.couchbase.com<\/a><\/span><\/p>\n Here’s a good read on best practices while using Couchbase’s Full Text Search ..<\/p>\n Full Text Search Indexing Best Practices by Use Case<\/a><\/p><\/blockquote>\n![\"\"](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/08\/Screen-Shot-2020-08-10-at-6.05.12-PM.png\")
<\/p>\n![\"Inverted](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/08\/Screen-Shot-2020-08-10-at-4.23.03-PM.png\")
<\/p>\n![\"Couchbase](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/08\/Screen-Shot-2020-08-10-at-4.40.16-PM.png\")
<\/p>\nA Text Analyzer<\/b><\/h3>\n
\n
\n
\n
\n
![\"Text](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/08\/Screen-Shot-2020-08-10-at-4.28.32-PM.png\")
<\/p>\nTokenizer<\/b><\/h4>\n
![\"Stock](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/08\/Screen-Shot-2020-08-10-at-4.20.50-PM.png\")
<\/p>\n\n
\n
![\"Regexp](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/08\/Screen-Shot-2020-08-10-at-4.24.20-PM.png\")
<\/p>\nCharacter filter<\/b><\/h4>\n
![\"Stock](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/08\/Screen-Shot-2020-08-10-at-4.25.28-PM.png\")
<\/p>\n\n
\n
![\"Regexp](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/08\/Screen-Shot-2020-08-10-at-4.26.14-PM.png\")
<\/p>\nToken filter<\/b><\/h4>\n
![\"Stock](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/08\/Screen-Shot-2020-08-10-at-4.27.20-PM.png\")
<\/p>\n![\"Configurable](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/08\/Screen-Shot-2020-08-10-at-4.27.56-PM.png\")
<\/p>\nStock Analyzers<\/b><\/h3>\n
{\r\n \"field1\": \"value1\",\r\n \"field2\": \"value2\",\r\n \"array_field3\": [\r\n \"value3\",\r\n \"value4\"\r\n ],\r\n \"object_field4\": {\r\n \"field5\": \"value5\",\r\n \"field6\": \"value6\"\r\n }\r\n}<\/pre>\n![\"\"](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/08\/Screen-Shot-2021-10-25-at-9.29.39-AM.png\")
<\/p>\n![\"Keyword](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/08\/Screen-Shot-2020-08-10-at-4.34.28-PM.png\")
<\/p>\n![\"\"](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/08\/Screen-Shot-2020-08-10-at-5.27.39-PM.png\")
<\/p>\nConfiguring a custom analyzer<\/b><\/h3>\n
\n
![\"Configuring](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/08\/Screen-Shot-2020-08-10-at-4.35.46-PM.png\")
<\/p>\nText analysis playground<\/b><\/h3>\n
![\"Bleve](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/08\/Screen-Shot-2020-08-10-at-1.58.24-PM.png\")
<\/p>\n