Capella App Services is a fully managed cloud backend for your mobile, desktop and IoT frontend applications. On the other end of the spectrum, the paradigm of “Edge Computing” has gained a lot of traction in recent years. Edge Computing is a distributed computing topology that aims to bring compute and storage closer to the edge, which is essentially the location where data is generated and consumed. Applications running at the edge benefit from low latency real time responses, reduced network bandwidth costs to cloud and adherence to data privacy and regulatory restrictions which prescribe that sensitive data is processed at the edge.
Edge Computing complements Cloud Computing. The cloud continues to serve as the source of truth. Data that is processed at the edge is transmitted to the cloud for eventual consistency and data integrity. So how do we enable such a deployment topology that combines the power of Capella’s managed cloud environment with self-managed, on-prem edge deployment? This is possible with the Inter-Sync Gateway Replication protocol that offers enterprise-grade scalable, secure sync between couchbase cloud and edge clusters.
In this post, we will walkthrough an example of how you can set up a hybrid deployment between a Capella managed App Services deployment and a self managed Couchbase Mobile deployment.
Hybrid Deployment Topology & Use Cases
In the context of this post, we use hybrid deployment topology to refer to a topology consisting of a fully managed Capella App Services syncing data with one or more self managed couchbase mobile clusters. A self-managed couchbase mobile cluster could be one that’s deployed and managed by a user in a public cloud, private cloud or data center or on-prem. A simple hybrid topology is shown below. In this model, data is stored and processed in Capella App Services as well as in the self-managed couchbase mobile cluster. Data is synced between the cloud and edge cluster using Inter-Sync Gateway Replication. Mobile and desktop client apps can sync data with Capella App Services or with self-managed Couchbase Mobile cluster.
There are several use cases of distributed cloud deployments as described in this whitepaper. They include resiliency to Internet disruptions and reduced latency in data processing.
Specifically, hybrid deployment topology offers the following benefits :-
-
- Data Privacy/Governance: Fulfillment of regulatory policies that dictate that sensitive data that should only be stored and processed on self managed, private or on-prem data centers
- Phased Migration to Capella: Not as obvious, but the hybrid topology with Capella will simplify the migration of existing on-prem couchbase mobile deployments to Capella. While XDCR based migration allows a one-shot migration of on-prem mobile clusters to Capella App Services, wherein all clients have to switch over from self-managed cluster to Capella. On the other hand, a hybrid topology with a bi-directional Inter-Sync Gateway Replication will allow for a phased migration – allowing for clients to migrate over time.
- Emerging Cloud Provider Edge Services: As cloud service providers continue to expand their infrastructure to the edge with offerings like AWS Local Zones, users can take advantage of these offerings by connecting their self-managed couchbase mobile clusters deployed at the edge of the cloud provider network with Capella App Services.
Configuration
Active Cluster
The Sync Gateway cluster on which the replication is initialized or scheduled is the Active Cluster. Think of it as being equivalent to a client in a classic client-server connection that is initializing a connection. In hybrid deployment, this would correspond to the self-managed Couchbase Mobile cluster. In other words, all replications (bi-directional and uni-directional) are initialized on the self-managed side.
Passive Cluster
The Sync Gateway cluster that is the target of replication is the Passive Cluster. Think of it as being equivalent to a server in a classic client-server connection that is listening for incoming connections.
Walkthrough
We will walk through a simple example that demonstrates how you can setup a Hybrid topology with Capella App Services. For brevity, we will not be walking through the specifics of how you deploy and provision Sync Gateway on-prem or on Capella App Services. If you are new to Capella App Services, please refer to the Getting Started with App Services guide. If you are new to Couchbase Mobile, refer to Getting Started with Sync Gateway guide.
Initial State
In order to support hybrid deployment with Capella App Services, the Active Cluster which is the self-managed mobile cluster MUST be running Sync Gateway versions v3.0.5 and or v2.8.4.
Self-Managed Cluster
This is the Sync Gateway database configuration on the self-managed side of the deployment as retrieved using GET db call.
Request
|
1 2 3 4 |
curl --location --request GET 'https://on-prem-syncgateway:4985/travel-sample/_config' \ --header 'Accept: application/json' \ --header 'Content-Type: application/json' \ --header 'Authorization: Basic c2d3X2FkbWluOnBhc3N3b3Jk' |
Response
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
{ "server": "couchbases://on-prem-couchbase-server", "bucket": "travel-sample", "username": "sgw_admin", "password": "xxxxx", "name": "travel-sample", "sync": "\nfunction sync(doc, oldDoc) {\n if (doc.type) {\n channel(\"channel.\"+doc.type);\n }\n else {\n channel(\"!\");\n }\n}", "import_docs": true, "cache": { "rev_cache": {}, "channel_cache": {} }, "unsupported": { "warning_thresholds": { "xattr_size_bytes": 943718, "channels_per_doc": 50, "access_and_role_grants_per_doc": 50, "channels_per_user": 50000, "channel_name_size": 250 } }, "enable_shared_bucket_access": true, "num_index_replicas": 0, "delta_sync": { "enabled": true } } |
As you can see, the configuration is straightforward.
-
- I have a Sync Gateway database named “travel-sample” backed up by a “travel-sample” bucket on Couchbase Server. The “travel-sample” bucket is a sample bucket that is loaded into the server.
- “sgw_admin” is the sync gateway user that is used for authenticating Sync Gateway to Couchbase server.
- The sync function does the following:
- All documents that have a “type” property are assigned to a corresponding channel corresponding to that document type. For example, documents that have a “type”:”airline” are assigned to a channel named “channel.airline“, a document that has a “type”:”airport” are assigned to a channel named “channel.airport” and so on.
- The documents that do not have a “type” property are assigned to a public channel.
The rest of the configuration is default.
Capella App Services
This is the configuration on the Capella App Services side.
I have an App Endpoint named “travel-sample” that is backed up by a “travel-sample” database. In my example, the “travel-sample” database is empty.
The Access Control Function is identical to the Sync Function on the self-managed side.
I have an App User named “demo@example.com” who is granted access to “channel.airline” channel (in addition to the system public “!” channel).
Setting up Inter-Sync Gateway Replication
The self-managed Couchbase mobile cluster is the “active cluster” and it is the cluster where replications MUST be initialized.
I will initialize a bi-directional, continuous replication named “pushandpull-with-target-continuous” on the self-managed Sync Gateway using the PUT _replication API.
Request
Validating Data Sync
Finally, since we have a bidirectional sync setup between Capella App Services and a self managed cluster, I can validate that document changes made on either endpoint are synced over to the other side.
Syncing from self-managed Couchbase Mobile cluster to Capella App Services
Changes to any document on self-managed are automatically synced over to the cloud as demonstrated below. This is because we do not have any filters set on the replication.
Changes to document on self-managed Couchbase Server (right) are synced to Capella (left)
Syncing from capella App Services to self-managed Couchbase Mobile cluster
In contrast, only changes made to documents that have a property “type” of “airline” on Capella App Services are synced down to the self-managed cluster. The reason for this is because the App User on Capella App Services, “demo@example.com” is granted access to only the “channel.airline”. So the App User can only read documents of “type“equal to”airline“.
Changes to document on Capella (left) are synced to self-managed Couchbase Server (right)
Monitoring Replications
Once your configurations are up and running, you can monitor it via the replicationStatus endpoint.
Resources
You can try Capella App Services for free today and deploy hybrid topologies with self-managed edge clusters.
Be sure to deploy v3.0.5 or v2.8.4 versions of Sync Gateway which you can download from our downloads page.
If you want to dive into the details, here’s where you can find more information:
The Couchbase Forums is a great place to reach out with questions. Please leave a comment below or feel free to reach out to me via Twitter or email me
Acknowledgements
I would like to thank Mark Gamble for his review and feedback on the blog post.
