Mirror Couchbase bucket with bulk documents

Couchbase Server
, ,
1

I want to mirror content from one Couchbase bucket to another bucket and the number of documents in one bucket is more than half million. What’s the process to do it?

2

Hi @Kratos

The Couchbase Eventing service can be used as a point tool (one and done) for a one time replication -or- if you leave the Function deployed and active it can accomplish a continuous real-time mirror of one bucket to another.

Note I am describing intra-cluster replication, if you want to replicate buckets between clusters (inter-cluster) you would use the XDCR Service.

The Eventing Function is quite simple 6 lines without comments:

function OnUpdate(doc,meta) {
    // doc is the data from the source (listen to location) insert/update 
    // and meta.id is the key
    dst_bkt[meta.id] = doc;
}

function OnDelete(meta) {
    // meta.id is the key of a document that has been deleted or expired in 
    // the source (listen to location).
    delete dst_bkt[meta.id];
}

Essentially we have three buckets:

  1. Bucket ‘primary’, what Eventing listens to all changes on - this is the Eventing source bucket.
  2. Bucket ‘mirror’ which is the destination bucket where we want to mirror our data to.
  3. Bucket ‘metadata’ a small scratchpad used by the eventing service for keeping state.

In the Eventing settings we set up the bucket ‘mirror’ with an Bucket binding or alias of ‘dst_bkt’ inread+write mode (this exposes a JavaScript map that can be manipulate KV or the data service directly).

When you deploy the Eventing (the feed boundary should be set to Everything) it will mirror the bucket this may take a while until it catches up to steady state. If you want the mirroring function to run faster (on the initial catch up) you may increase the number of workers up to the number of vCPUs in the instance running Eventing. Once in real-time you could pause the Eventing function and adjust the workers back down to a lower number if you so desire (pause/resume will not miss any mutations or changes).

To understand Eventing works in detail (how to configure and invoke) I recommend that you run through a detailed example or two at Examples: Using the Eventing Service | Couchbase Docs

Best

Jon Strabala
Principal Product Manager - Server‌

3

Note I am describing intra-cluster replication, if you want to replicate buckets between clusters (inter-cluster) you would use the XDCR Service.

It’s possible to setup XDCR between two buckets on the same cluster, so that is another option here.

A third option is to use backup and restore.