{"id":13533,"date":"2022-07-06T13:37:00","date_gmt":"2022-07-06T20:37:00","guid":{"rendered":"https:\/\/www.couchbase.com\/blog\/?p=13533"},"modified":"2025-06-13T22:40:18","modified_gmt":"2025-06-14T05:40:18","slug":"eventing-data-consolidation-in-couchbase","status":"publish","type":"post","link":"https:\/\/www.couchbase.com\/blog\/eventing-data-consolidation-in-couchbase\/","title":{"rendered":"Using the Eventing Service to Consolidate Data from Multiple Sources"},"content":{"rendered":"

One of the many advantages of using a document-based database such as Couchbase is the ability to use a flexible data model to store data without the constraints of a rigid, predetermined schema. Many customers choose a NoSQL database to support the ingestion of data from multiple sources. But what if you need your data transformed so that it matches your existing data? Maybe you need it all in a uniform format for reporting purposes. Whatever the reason, this is a perfect use for the Eventing Service<\/a> in Couchbase.<\/span><\/p>\n

Couchbase Eventing Service<\/h2>\n

The eventing service is a built-in feature of Couchbase Enterprise Edition that allows the application of JavaScript functions, <\/span>or lambdas<\/em><\/span>, that respond to mutation events. Whenever a document is created, modified, or deleted, an action can be taken <\/span>by executing pure business logic in your JavaScript<\/em><\/span>. There are a number of actions that can be taken, such as data enrichment, calling of external workflows (via RESTful API calls), and timer-based events (resuming execution in the future). Think of events as a legacy database <\/span>post-<\/span>trigger on steroids.<\/span><\/p>\n

Data consolidation<\/h2>\n

In this simplified example, imagine you have just acquired two new data sources for customer data. This data might be coming from a normalized source, or in a flat file, or both, and you want this data to match your existing customer data stored in a denormalized format.<\/span><\/p>\n

The first data source is a normalized database (<\/span>clean and accurate) <\/span>containing customers in Canada. Each customer record points to a billing and a shipping address, and each address is normalized to eliminate redundancy of city and province data:<\/span><\/p>\n

\"Normalized
\nThe second data source is a flat file also containing Canadian customers, but that does not track separate billing and shipping addresses. As you can see, <\/span>only a single address data element<\/span> is stored within the customer record:<\/span><\/p>\n

\"Customer<\/p>\n

We need to transform this data as it comes in so that it matches our existing customer database which is denormalized such that billing and shipping addresses are attached to the customer data (<\/span>from the second source<\/span>) as sub-documents:<\/span><\/p>\n

\"Creating<\/p>\n

Let\u2019s start with <\/span>an easy<\/span> solution first, the flat file import. The process for transforming this data involves copying the customer data (name, email, phone, company name) into a new document. The address data then needs to be copied into subdocuments for both billing and mailing addresses. Because we only have one address per customer, we\u2019ll copy this address into both the billing and the mailing address subdocument:<\/span><\/p>\n

\"Importing
\nJavaScript code to consolidate the tables<\/span><\/h2>\n

The Eventing code to accomplish this is quite simple:<\/span><\/p>\n

 \u00a01 function OnUpdate(doc, meta) {\r\n\u00a0\u00a02 \u00a0 \u00a0 const new_doc = {}; \/\/ New customer document\r\n\u00a0\u00a03 \u00a0 \u00a0 const address = {}; \/\/ New address document\r\n\u00a0\u00a04\r\n\u00a0\u00a05 \u00a0 \u00a0 new_doc['first_name'] = doc.first_name;\r\n\u00a0\u00a06 \u00a0 \u00a0 new_doc['last_name'] = doc.last_name;\r\n\u00a0\u00a07 \u00a0 \u00a0 new_doc['email'] = doc.email;\r\n\u00a0\u00a08 \u00a0 \u00a0 new_doc['phone'] = doc.phone;\r\n\u00a0\u00a09 \u00a0 \u00a0 new_doc['company_name'] = doc.company_name;\r\n\u00a010\r\n\u00a011 \u00a0 \u00a0 address['street_address_1'] = doc.street_address_1;\r\n\u00a012 \u00a0 \u00a0 address['street_address_2'] = doc.street_address_2;\r\n\u00a013 \u00a0 \u00a0 address['city'] = doc.city;\r\n\u00a014 \u00a0 \u00a0 address['state'] = doc.province;\r\n\u00a015 \u00a0 \u00a0 address['zip_code'] = doc.postal_code;\r\n\u00a016\r\n\u00a017 \u00a0 \u00a0 \/\/ Add address document to customer document as subdocuments\r\n\u00a018 \u00a0 \u00a0 new_doc['billing_address'] = address;\r\n\u00a019 \u00a0 \u00a0 new_doc['mailing_address'] = address;\r\n\u00a020\r\n\u00a021 \u00a0 \u00a0 \/\/ This creates the new document in the destination bucket\r\n\u00a022 \u00a0 \u00a0 dest_bkt[doc.email] = new_doc;\r\n\u00a023\r\n\u00a024 \u00a0 \u00a0 \/\/ Delete original document - uncomment below\r\n\u00a025 \u00a0 \u00a0 \/\/ delete sch2[meta.id];\r\n\u00a026\r\n\u00a027 \u00a0 \u00a0 log(\"Doc created\/updated\", meta.id);\r\n\u00a028 }<\/pre>\n
The above code should be easy enough to understand, even for non JavaScript coders, but I\u2019ll break this down by section just to be sure.<\/span><\/p>\n