{"id":5816,"date":"2018-09-12T08:10:08","date_gmt":"2018-09-12T15:10:08","guid":{"rendered":"https:\/\/www.couchbase.com\/blog\/?p=5816"},"modified":"2025-06-13T20:59:13","modified_gmt":"2025-06-14T03:59:13","slug":"how-analyzers-tokenizers-filters-work-fts-part-2","status":"publish","type":"post","link":"https:\/\/www.couchbase.com\/blog\/how-analyzers-tokenizers-filters-work-fts-part-2\/","title":{"rendered":"Building a Shazam-like app to understand how Tokenizers and Filters work | FTS Part 2"},"content":{"rendered":"

In the previous blog post, we talked about why full-text search is a better solution at scale to implement a well-designed search in your application<\/a>. In this second part, we are going to deep-dive on the Inverted Index and explore how analyzers, tokenizers, and filters might shape the result of your searches.<\/p>\n

Full-text search is all about searching on the text; therefore, it does not matter if you are indexing and searching logs, genes in a DNA, your own data structure, and of course, language. They will all essentially work nearly the same way.<\/p>\n

To give you an example of how to you can use FTS even when you have your own custom structure, let\u2019s leverage the fact that Apple finally bought Shazam and build an imaginary Shazam-like app. However, instead of listening to a small fragment of music like Shazam does, we will ask for the user to whistle it.<\/p>\n

 <\/p>\n

Wait\u2026 why do I need Full-text Search for it?<\/strong><\/h2>\n

As the user might wrongly whistle some parts of the song, we will need to split it in \u201csmall blocks of melody\u201d and then try to match them against our library. Assuming that our library will have thousands or even millions of songs (Apple and Spotify libraries have over 30 million songs), a simple LIKE \u201c%melody%\u201d stands no chance of bringing results in a reasonable amount of time.<\/p>\n

An inverted index seems to be the right tool for the job as we can easily find all songs that contain a given block of melody. If you are not familiar with this concept yet, please check out my previous blog post<\/a> about it.<\/p>\n

 <\/p>\n

The Parsons Code<\/strong><\/h2>\n

The first thing we need to do is convert our songs library to text. We can achieve that by using the Parsons code<\/a>, which is a notation used to identify a piece of music\u00a0according to movements of the\u00a0pitch<\/a><\/u>\u00a0up and down:<\/p>\n

 <\/p>\n