Let’s take a quick look at the different components that we will be using throughout this blog post.<\/span><\/p>\n Couchbase<\/b><\/a>\u00a0 is an Open Source, distributed, JSON document database. It exposes a scale-out, key-value store with managed cache for sub-millisecond data operations, purpose-built indexers for efficient queries and a powerful query engine for executing SQL-like queries.<\/span><\/p>\n Elasticsearch<\/b><\/a> is a full-text, distributed NoSQL database with a powerful search and analytics engine at the heart of Elastic Stack. Elasticsearch stores documents in Index (sometimes referred to as Indices?) which are analogous to tables in sql world.<\/span><\/p>\n Ingest Node Pipeline<\/b><\/a> is a powerful tool that Elasticsearch gives you in order to pre-process your documents before they are indexed. An Ingest node pipeline consists of one or more processors that are executed in the order it is declared. Elasticsearch comes with a set of processors out of the box, you can also build a custom processor as needed, for a list of all processors visit the documentation <\/span>here<\/span><\/a>.<\/span><\/p>\n Elasticsearch Connector<\/b><\/a> is a tool built by Couchbase that enables replication of data from Couchbase to Elasticsearch.<\/span><\/p>\n Kibana<\/b><\/a> is a free and open user interface that lets you visualize your Elasticsearch data and navigate the Elastic Stack. Kibana also enables management and evaluation of Ingest node pipelines.<\/span><\/p>\n This post assumes that you have a basic understanding of all of the components as listed above and you are using a computer running macOS to walk through the illustration.<\/span><\/p>\n Launch a new terminal window and navigate to your user directory located under \/Users\/<username>. Create a new directory called Connectors<\/strong><\/em>. <\/span>We will refer to this as <\/span>\u201cBASE_DIR<\/span>\u201d for the remainder of the post.<\/span><\/p>\n <\/p>\n \u00a0 Launch a new terminal and navigate to BASE_DIR<\/span><\/span>\u00a0and then t<\/span>ype the following commands (<\/span><\/span>one at a time)\u00a0<\/span><\/p>\n Launch a new terminal window and navigate to the directory BASE_DIR<\/span><\/span>\/es<\/strong><\/span> then type the following command<\/span><\/p>\n Now open a web browser and navigate to <\/span>https:\/\/localhost:9200<\/span><\/a>.You should see a response like the one below, indicating that the Elasticsearch server is\u00a0 available.<\/span><\/p>\n Launch a new terminal window and navigate to the directory BASE_DIR<\/span><\/span>\/kibana\u00a0 <\/b>then type the following command<\/span><\/p>\n To see Kibana in action, open a web browser and navigate to <\/span>https:\/\/localhost:5601<\/span><\/a> .<\/span><\/p>\n Launch a new terminal and navigate to BASE_DIR<\/span><\/span>\u00a0 and t<\/span>ype the following commands (<\/span>one at a time)<\/span><\/span>. <\/b>When prompted enter “<\/span>A” <\/b>to uncompress the files<\/span><\/p>\n Now let’s explore the some of the important directories within <\/span>cbes<\/b><\/p>\n bin<\/b> – contains all the command line utilities required to help manage connectors.<\/span><\/p>\n config<\/b> – contains base configuration file that can be used as reference.<\/span><\/p>\n secrets<\/b> – contains credentials to connect to Couchbase and Elasticsearch servers.<\/span><\/p>\n Let’s build our pipeline. We will name it \u201c<\/span>landmark-pipeline\u201d. <\/i><\/b>Our pipeline will<\/span><\/p>\n We will be using two of the existing processors to build our ingestion pipeline.<\/span><\/p>\n Script Processor :<\/b> Executes any script defined by the <\/span>painless scripting language<\/span><\/a>.<\/span><\/p>\n Uppercase Processor<\/b> : Converts the specified field\u2019s value into UPPERCASE <\/span><\/p>\n Connector assumes the existence of a configuration file default-connector.toml\u00a0 and reads this file for any configuration. However you can also specify the configuration when deploying the connector by using a – -config command line option. For this post we will use the default configuration.<\/span><\/p>\n \u00a0If you want to skip this step, a complete modified configuration file can be found <\/span>here<\/span><\/a>.\u00a0\u00a0<\/span><\/p>\n \u00a0a ) Under the [<\/em><\/span>group]<\/span><\/em> table, set the name key to \u201clandmark-example-group\u201d<\/span><\/p>\n \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0b) Under the [elasticsearch.docStructure] <\/em>table, set the documentContentAtTopLevel key to \u201ctrue\u201d <\/span>\u00a0<\/span><\/i><\/p>\n c) Remove any existing [[elasticsearch.type]]<\/em> tables and replace\u00a0 with the following.<\/span>\u00a0 \u00a0\u00a0<\/span><\/p>\n You are all set. It’s time to see all of this in action now.<\/span><\/p>\n Elasticsearch connectors can be deployed three different modes\u00a0<\/span><\/p>\n Solo<\/b> :\u00a0 This is the simplest mode where the connector runs as a standalone process. Solo mode is preferable in a low-traffic environment or in a development environment.<\/span><\/p>\n Distributed <\/b>: In this mode multiple connectors run as different processes. In a scenario when the traffic is moderate to high, this mode is recommended. Unlike the solo mode where only one dedicated process does the whole work, in a distributed mode there is more than one process and each connector is configured independently to share the workload.<\/span><\/p>\nPrerequisites & Assumptions<\/b><\/h4>\n
\n
\n
\n
Get Started<\/strong><\/h4>\n
mkdir Connectors<\/pre>\n
Install & Launch Elasticsearch and Kibana<\/b><\/h4>\n
\n
\n
Download<\/a> the latest version of Elasticsearch and Kibana and move them into the BASE_DIR<\/span>. You should see two files Elasticsearch-<version>-darwin-x86_64.tar.gz and kibana-<version>-darwin-x86_64.tar.tz.<\/h5>\n<\/li>\n
Uncompress the downloaded files and rename the directory to es<\/strong> and kibana<\/strong><\/span><\/h5>\n<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n
tar -xzvf elasticsearch-7.8.0-darwin-x86_64.tar.gz && tar -xzvf kibana-7.8.0-darwin-x86_64.tar.gz<\/pre>\n
mv elasticsearch-7.8.0-darwin-x86_64 es && mv kibana-7.8.0-darwin-x86_64 kibana<\/pre>\n
\u00a0 \u00a0 \u00a0 \u00a03. Launch Elasticsearch<\/span><\/h5>\n
bin\/elasticsearch<\/pre>\n
![\"Elasticsearch](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/07\/es_launch.png\")
<\/p>\n4. Launch Kibana<\/span><\/h5>\n
bin\/kibana<\/pre>\n
![\"Kibana](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/07\/kibana3.png\")
<\/h4>\nInstall Elasticsearch Connector<\/b><\/h4>\n
\n
\n
Download<\/span><\/a> the latest version of Elasticsearch connector and move it to the BASE_DIR<\/span><\/span>.<\/span><\/i> At the time of writing this post, the most recent version of the connector available is 4.2.2. You should have downloaded a file that looks something like couchbase-elasticsearch-connector-<\/span><version><\/b>.zip.<\/span><\/h5>\n<\/li>\n
Uncompress the downloaded file and rename the directory to <\/span>cbes.<\/b><\/h5>\n<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n
unzip couchbase-elasticsearch-connector-4.2.2.zip<\/pre>\n
mv couchbase-elasticsearch-connector-4.2.2 cbes<\/pre>\n
![\"cbes](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/07\/cbes.png\")
<\/p>\nBuild Elasticsearch Ingest Node Pipeline<\/b><\/h4>\n
\n
\n
\n
\n
1. Launch a new terminal window and execute the following command<\/span><\/h5>\n
curl -X PUT \"localhost:9200\/_ingest\/pipeline\/landmark-pipeline?pretty\" -H 'Content-Type: application\/json' -d'\r\n{\r\n\"description\" : \"Converts the field name to uppercase and creates new field last_update_time \",\r\n\"processors\" : [ { \"uppercase\": { \"field\": \"name\" }, \"script\": { \"lang\": \"painless\", \"source\": \"ctx.last_update_time = new Date();\" } } ]\r\n}'<\/pre>\n\u00a0 \u00a0 The curl command above creates and saves a pipeline definition within Elasticsearch database.<\/span><\/i><\/span><\/strong><\/h6>\n
2.\u00a0 Open a web browser and navigate to Kibana at <\/span>https:\/\/localhost:5601<\/span><\/a>. You should be able to see the pipeline that you had just created under Discover -><\/strong>\u00a0 \u00a0 \u00a0 \u00a0 Ingest Node Pipelines.<\/span><\/h5>\n
![\"Ingest](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/07\/pipeline2.png\")
<\/p>\nConfigure Elasticsearch Connector<\/b><\/h4>\n
1. Launch a new terminal window and navigate to BASE_DIR<\/span><\/span>\/<\/span><\/i>cbes\/config<\/b><\/h5>\n
<\/b>2. Copy the file example-connector.toml<\/span> and name it as default-connector.toml<\/span><\/strong>\u00a0by executing the following command<\/span><\/h5>\n
cp example-connector.toml default-connector.toml<\/pre>\n
3. Open default-connector.toml in the editor of your choice and modify the following configuration settings.If you want further information about the Elasticsearch Connector configuration settings, go <\/span>here<\/span><\/a>.<\/span><\/h5>\n
Tip:\u00a0 If this is your first time working with the TOML config file format, check out Nate Finch\u2019s excellent Intro to TOML<\/a>, or the official specification<\/a>.\u00a0<\/em><\/span><\/strong><\/h6>\n
[group]\r\nname = 'landmark-example-group'<\/pre>\n
[elasticsearch.docStructure]\r\ndocumentContentAtTopLevel = true<\/pre>\n
[[elasticsearch.type]]\r\nprefix = '_sync:'\r\nignore = true\r\n\r\n[[elasticsearch.type]]\r\nprefix = 'landmark_'\r\nindex = 'landmark'\r\npipeline = 'landmark-pipeline'<\/pre>\n
![\"Elasticsearch](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/07\/config2.gif\")
<\/h4>\n<\/h4>\n
Deploy Elasticsearch Connector<\/b><\/h4>\n