{"id":12479,"date":"2021-12-21T08:00:29","date_gmt":"2021-12-21T16:00:29","guid":{"rendered":"https:\/\/www.couchbase.com\/blog\/?p=12479"},"modified":"2025-06-13T22:40:20","modified_gmt":"2025-06-14T05:40:20","slug":"migrating-buckets-to-collections-scopes-via-eventing-part-2","status":"publish","type":"post","link":"https:\/\/www.couchbase.com\/blog\/migrating-buckets-to-collections-scopes-via-eventing-part-2\/","title":{"rendered":"Migrating Buckets to Collections & Scopes via Eventing: Part 2"},"content":{"rendered":"

Migrating Buckets to Collections & Scopes via Eventing: Part 2<\/h2>\n

Again (as I did in Part 1) I want to point out an excellent blog written by Shivani Gupta, How to Migrate to Scopes & Collections in Couchbase 7.0<\/a>, which covers in great detail other methods of migrating bucket-based documents to Scopes and Collections in Couchbase.\u00a0 I encourage you to also read about the multiple non-Eventing methods that Shivani touches upon.<\/p>\n

Whether you\u2019re new to Couchbase or<\/strong> a seasoned vet, you\u2019ve likely heard about Scopes and Collections. If you\u2019re ready to try them, this article helps you make it happen.<\/p>\n

Scopes and Collections are a new feature introduced in Couchbase Server 7.0<\/a>\u00a0that allows you to logically organize data within Couchbase. To learn more,\u00a0read this introduction to Scopes and Collections<\/a>.<\/p>\n

You should take advantage of Scopes and Collections if you want to map your legacy RDBMS to a document database or if you\u2019re trying to consolidate hundreds of microservices and\/or tenants into a single\u00a0Couchbase<\/a>\u00a0cluster (resulting in much lower TCO).<\/p><\/blockquote>\n

\n

Using Eventing for Scopes & Collections Migration<\/h2>\n

In the prior article (Part 1), I discussed the mechanics of a high performance method to migrate from an older Couchbase version to Scopes and Collections in Couchbase 7.0 based on Eventing<\/a>.<\/span><\/p>\n

Just the Data Service (or KV) and the Eventing Service is required to migrate from buckets to collections. In a well-tuned, large Couchbase cluster, you can migrate over 1 million documents a second. Yes, no N1QL, and no index needed.<\/span><\/p>\n

In this follow up article, I will provide a simple fully automated methodology to do large migrations with dozens (or even hundreds) of data types via a simple <\/span>Perl<\/span><\/i> script.<\/span><\/p>\n

Recap of the final\u00a0 Eventing Function: ConvertBucketToCollections<\/h3>\n

In Part 1 we had the following settings for the Eventing Function.\u00a0 Note to each unique type, “beer<\/strong>” and “brewery<\/strong>” we had to add a Bucket binding alias to the target collection in “read+write” mode.\u00a0 In addition we had to create the target collections, in this case “bulk.data.beer<\/strong>” and “bulk.data.brewery<\/strong>“<\/span><\/p>\n

\"\"<\/p>\n

In Part 1 we had the following JavaScript code in our Eventing Function.\u00a0 Note to each unique type, “beer<\/strong>” and “brewery<\/strong>” we had to replicate a JavaScript code block and update reference the corresponding binding alias or target collection in the Couchbase Data Service.<\/span><\/p>\n

function OnUpdate(doc, meta) {\r\n    if (!doc.type) return;\r\n  \r\n    var type = doc.type;\r\n    if (DROP_TYPE) delete doc.type;\r\n  \r\n    if (type === 'beer') {\r\n        if (DO_COPY) beer_col[meta.id] = doc;\r\n        if (DO_DELETE) {\r\n            if(!beer_col[meta.id]) { \/\/ safety check \r\n                log(\"skip delete copy not found type=\" + doc.type + \", meta.id=\" + meta.id);\r\n            } else {\r\n                delete src_col[meta.id];\r\n            }\r\n        }\r\n    }\r\n    if (type === 'brewery') {\r\n        if (DO_COPY) brewery_col[meta.id] = doc;\r\n        if (DO_DELETE) {\r\n            if(!brewery_col[meta.id]) { \/\/ safety check\r\n                log(\"skip delete copy not found type=\" + doc.type + \", meta.id=\" + meta.id);\r\n            } else {\r\n                delete src_col[meta.id];\r\n            }\r\n        }\r\n    }\r\n}<\/pre>\n
The technique in Part 1 works but what if I have a lot of types?<\/h2>\n
Using Eventing can indeed do migrations as shown in Part 1, but it seems like a bit of work to set things up.<\/p>\n
If you have 80 different types, it would be an incredible amount of error-prone effort to use this technique (both creating the Eventing Function and creating the needed keyspaces). If I had 80 types in a bucket to migrate and split, I wouldn’t want to do all the work described above by hand for each type.<\/p>\n
Automate via CustomConvertBucketToCollections.pl<\/h2>\n
To solve this problem, I wrote a tiny Perl<\/em> script, CustomConvertBucketToCollections.pl<\/em>, that generates two files:<\/p>\n