If you are familiar with both Couchbase and Eventing please feel free to skip the brief overview and skip\u00a0ahead to the examples.<\/a><\/em><\/p>\n First off, let’s look at the basic Eventing Lifecycle<\/a>.<\/p>\n The steps below show how easy it is to write and use an Eventing Function<\/p>\n That’s it you now have a distributed function responding to mutations in your data set in real-time across your entire cluster.<\/p>\n The Eventing service provides an infrastructure-less platform that can scale your Eventing Functions as your business experiences growth whether a one-time spike or a monthly increase in data stores or clients served without concern for the fact that your JavaScript based Eventing functions are running in a robust reliable parallel distributed fashion.\u00a0 To learn more about Couchbase Eventing please refer to the Eventing Overview<\/a> in our documentation.<\/p>\n The examples in this article below show that in some cases Eventing can act like a drop of oil to needed “free up” the moving parts of your applications.<\/p>\n I like to think of a Couchbase cluster as a set of scalable interoperable “micro-services”. These services can be wired together and sized to meet a specific set of operational and business needs.<\/p>\n Couchbase provides Multi-dimensional Scaling, or MDS, across six key services with minimal resource interference between them. Furthermore, each Couchbase service is independently scalable, via simply adding more nodes. This allows customers to create the ideal multi-node cluster at the possible lowest TCO for the task(s) at hand.<\/p>\n Like all product lines the newer services in Couchbase Query, Search, and Eventing, and Analytics have a few warts but taken as a whole the complete basket provides a unified suite, or a one-stop shop to solve a myriad of problems. \u00a0I mean seriously if you don’t care about a unified product and all you are going to do is use FTS you might just consider using Elasticsearch but once you need to integrate your FTS results with N1QL (SQL for JSON) you might have been much better off just starting with Couchbase.<\/p>\n Today we are going to utilize just two services 1) the primary KV service provided by Data nodes 2) the Eventing Service.\u00a0 I will highlight through a few tiny JavaScript functions how you can overcome some hard at-scale problems by leveraging the Eventing Service.<\/p>\n In this article we will be using the latest GA version, i.e. Couchbase version 6.5.1<\/p>\n However, if you are not familiar with Couchbase or the Eventing service please walk through GET STARTED and one Eventing example specifically refer to the following:<\/p>\n A live production system has stored billions of documents. A new business need has occurred where the existing data needs to be enriched.\u00a0 This requirement of additional data impacts the complete document set.\u00a0 The operational impact encompasses both old or historic data and also new or mutating data. The business needs to keep the production system running non-stop and continuously respond to new real-time data.<\/p>\n Consider a somewhat contrived example application a GeoIP lookup service. This utility needs a dataset to enable looking up countries by IPV4 address ranges. The initial implementation stored records as follows:<\/p>\n Months or later new business requirements change. Engineering needed the JSON documents to be enriched with new fields.\u00a0 The requirement was to include the numeric representations of the two existing IPV4 address.<\/p>\n A simple fourteen (14) line JavaScript (2 of which comments) Eventing Function can be written and deployed to solve the issue with minimal resources.<\/p>\n By deploying the above Function with a “Feed boundary” set to “Everything” all documents of type: “ip_country_map” are processed and enriched.<\/p>\n The Eventing Function is left “deployed” reacting to all new inserts or updates (or mutations) in real-time enriching new items and also updating existing items on changes to “ip_start_num” or “ip_end_num” to the proper “numeric” representations.<\/p>\n Because the documents are enriched (the old fields are still present) the existing production applications will still work.\u00a0 All new or changed data is updated in real-time to the new schema. \u00a0The GeoIP application components are decoupled via this simple Eventing Function such that they can be upgraded one at a time.<\/p>\n When all production components have been updated the Eventing Function can be undeployed and decommissioned.<\/p>\n A live production system has stored over 7 billion of documents.\u00a0 All documents have an automatic expiration (or TTL<\/a> for time to live) set. The production environment constantly receives new data and constantly expires old data.<\/p>\n An operational mistake was made resulting in 2 billion documents being created without an expiration.<\/p>\n The customer didn’t have the resources (nor desired to pay for the resources) to create a large index to utilize N1QL to identify select and purge the data that lacked an active (non-zero TTL) when it was no longer useful.<\/p>\n A simple six (6) line JavaScript (2 of which are comments) Eventing Function was deployed and solve the issue with minimal resources.<\/p>\n By deploying the above Function with a “Feed boundary” set to “Everything” the entire document set in the source bucket was scanned. All documents with a non-zero TTL (meaning they had no expiration) were ignored.\u00a0 Only the matching documents with a TTL greater than zero are deleted.<\/p>\n Once the source bucket was cleaned the Eventing Function was undeployed as it was used as an administrative tool.<\/p>\n Note we could replace the expiration !== 0 test in our JavaScript to filter out data for any needed purpose.<\/p>\n Pretty easy to guess what we are doing below:<\/p>\n In fact, we could easily update the above to perform a cascade archive and delete not only of the customer but of any other related information such as orders, returns and shipping addresses. Refer to the example Cascade Delete<\/a> in the Eventing Documentation.<\/p>\n A company running Couchbase on-premises in production was needed to share customer profile information (150M and growing).\u00a0 Their business partner is also running Couchbase but in a cloud provider, AWS.<\/p>\n Given a typical profile record like the following:<\/p>\n They couldn’t just share the entire profile since it contained sensitive data on user preferences and payment methods. \u00a0They only needed to share a limited subset like the following:<\/p>\n<\/div>\n The customer wanted to replace middle-wear SPARK solution required hours on failure to initialize and only provided slow batch process (hours to reflect updates) and sync up the profile information in real-time.<\/p>\n A simple nine (9) line JavaScript (3 of which are comments) Eventing Function was deployed and solve the issue with minimal resources.<\/p>\n By deploying the above Function with a “Feed boundary” set to “Everything” the entire document set in the source bucket was scanned and all documents of type: “master_profile” were processed and only a sub-document from each profile without the sensitive information was copied to the shared destination bucket.<\/p>\n The Eventing Function is always left deployed reacting to all user profile changes (or mutations) in real-time and forwarding each and every mutation to the AWS destination bucket.<\/p>\n In this use case a final bucket to bucket synchronization is performed via the Couchbase XCDR<\/a> feature at no time does the sensitive data leave the on-premises cluster and it is never transmitted to AWS.<\/p>\n Eventing Simple Yet Powerful: Eventing allows small scripts to overcome hard to solve problems. If you are familiar with both Couchbase and Eventing please feel free to skip the brief overview and skip\u00a0ahead to the examples. Overview: First off, let’s […]<\/p>\n","protected":false},"author":42711,"featured_media":8745,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[1814,1815,1821,1816,2273,9411],"tags":[1445,9271,2046],"ppma_author":[9113],"class_list":["post-8743","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-application-design","category-best-practices-and-tutorials","category-couchbase-architecture","category-couchbase-server","category-eventing","category-multi-dimensional-scaling","tag-mds","tag-multi-dimensional-scaling","tag-mutation"],"acf":[],"yoast_head":"\nOverview:<\/h2>\n
![\"Eventing](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2020\/05\/cron_implementation_1_image_0_lifecycle_overview_notitle_b-1024x398.png\")
<\/p>\n\n
Couchbase a database you build to order:<\/h2>\n
![\"\"](\"https:\/\/www.couchbase.com\/blog\/wp-content\/uploads\/2017\/07\/MDS-container-1-1024x405.png\")
<\/p>\n\n
Prerequisites \/ Learning about Eventing:<\/h2>\n
\n
Enriching Data via Eventing:<\/h2>\n
A typical customer problem<\/h4>\n
{\r\n \"type\": \"ip_country_map\",\r\n \"country\": \"RU\",\r\n \"ip_start\": \"7.12.60.1\",\r\n \"ip_end\": \"7.62.60.9\"\r\n}<\/pre>\n{\r\n \"type\": \"ip_country_map\",\r\n \"country\": \"RU\",\r\n \"ip_end\": \"7.62.60.9\",\r\n \"ip_start\": \"7.12.60.1\",\r\n \"ip_num_start\": 118242305,\r\n \"ip_num_end\": 121519113\r\n}<\/pre>\n<\/div>\nEventing to the rescue<\/h4>\n
function OnUpdate(doc, meta) {\r\n if (doc.type !== \"ip_country_map\" ) return;\r\n doc[\"ip_num_start\"] = get_numip_first_3_octets(doc[\"ip_start\"]);\r\n doc[\"ip_num_end\"] = get_numip_first_3_octets(doc[\"ip_end\"]);\r\n \/\/ src is a bucket alias to the source bucket in settings, write back to it\r\n src[meta.id]=doc;\r\n}\r\nfunction get_numip_first_3_octets(ip) {\r\n if (!ip) return 0;\r\n var parts = ip.split('.');\r\n \/\/ IP Number = A x (256*256*256) + B x (256*256) + C x 256 + D\r\n return = (parts[0]*(256*256*256)) + (parts[1]*(256*256)) +\r\n (parts[2]*256) + parseInt(parts[3]);\r\n}<\/pre>\n<\/div>\nPurging Stale Data via Eventing:<\/h2>\n
A typical customer problem<\/h4>\n
Eventing to the rescue<\/h4>\n
function OnUpdate(doc, meta) {\r\n if (meta.expiration !== 0) return;\r\n \/\/ delete all items that have TTL or expiration of 0\r\n \/\/ src is a bucket alias to the source bucket in settings, delete from it.\r\n delete src[meta.id];\r\n}<\/pre>\n<\/div>\nfunction OnUpdate(doc, meta) {\r\n if (!(doc.type === \"customer\"&& doc.active === false)) return \r\n \/\/ archive the customer to the bucket aias arc and remove from the bucket alias src\r\n arc[meta.id] = doc;\r\n delete src[meta.id];\r\n}<\/pre>\nStripping sensitive data via Eventing:<\/h2>\n
A typical customer problem<\/h4>\n
{\r\n \"type\": \"master_profile\",\r\n \"first_name\": \"Peter\",\r\n \"last_name\": \"Chang\",\r\n \"id\": 80927079070,\r\n \"basic_profile\": {\r\n \"partner_id\": 80980221,\r\n \"services\": [\r\n {\r\n \"music\": true\r\n },\r\n {\r\n \"radio\": true\r\n },\r\n {\r\n \"video\": false\r\n }\r\n ]\r\n },\r\n \"sensitive_profile\": {\r\n \"ssn\": \"111-11-1111\",\r\n \"credit_card\": {\r\n \"number\": \"3333-333-3333-3333\",\r\n \"expires\": \"01\/09\",\r\n \"ccv\": \"111\"\r\n }\r\n },\r\n \"address\": {\r\n \"home\": {\r\n \"street\": \"4032 Kenwood Drive\",\r\n \"city\": \"Boston\",\r\n \"zip\": \"02102\"\r\n },\r\n \"billing\": {\r\n \"street\": \"541 Bronx Street\",\r\n \"city\": \"Boston\",\r\n \"zip\": \"02102\"\r\n }\r\n },\r\n \"phone\": {\r\n \"home\": \"800-555-9201\",\r\n \"work\": \"877-123-8811\",\r\n \"cell\": \"878-234-8171\"\r\n },\r\n \"locale\": \"en_US\",\r\n \"timezone\": -7,\r\n \"gender\": \"M\"\r\n}<\/pre>\n{\r\n \"type\": \"shared_profile\",\r\n \"first_name\": \"Peter\",\r\n \"id\": 80927079070,\r\n \"basic_profile\": {\r\n \"partner_id\": 80980221,\r\n \"services\": [\r\n {\r\n \"music\": true\r\n },\r\n {\r\n \"radio\": true\r\n },\r\n {\r\n \"video\": false\r\n }\r\n ]\r\n },\r\n \"timezone\": -7\r\n}<\/pre>\n<\/div>\nEventing to the rescue<\/h4>\n
function OnUpdate(doc, meta) {\r\n \/\/ only process our profile documents\r\n if (doc.type !== \"master_profile\") return;\r\n \/\/ aws_bkt is a bucket alias to the target bucket to replicate to AWS via \r\n \/\/ XCDR. Write the minimal (non-sensitive) profile doc to the bucket for AWS.\r\n aws_bkt[\"shared_profile::\"+doc.id] = \r\n { \"type\": \"shared_profile\", \"first_name\": doc.first_name, \"id\": doc.id, \r\n \"basic_profile\": doc.basic_profile, \"timezone\": doc.timezone };\r\n}<\/pre>\n<\/div>\nResources<\/h3>\n
\n
References<\/h3>\n
\n
\nhttps:\/\/docs.couchbase.com\/server\/current\/eventing\/eventing-overview.html<\/a><\/li>\n
\nhttps:\/\/docs.couchbase.com\/server\/6.5\/introduction\/whats-new.html<\/a><\/li>\n
\nhttps:\/\/www.couchbase.com\/blog\/tag\/eventing\/<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"