Docker Service and Swarm Mode to Create Couchbase Cluster

Docker 1.12 introduced Services. A replicated, distributed and load balanced service can be easily created using docker service create command. A “desired state” of the application, such as run 3 containers of Couchbase, is provided and the self-healing Docker engine ensures that that many containers are running in the cluster. If a container goes down, another container is started.
If a node goes down, containers on that node are started on a different node. This blog will show how to setup a Couchbase cluster using Docker Services. Many thanks to @marcosnils, another fellow Docker Captain,
to help me debug the networking!

Couchbase Cluster

A cluster of Couchbase Servers is typically deployed on commodity servers. Couchbase Server has a peer-to-peer topology where all the nodes are equal and communicate to each other on demand. There is no concept of master nodes, slave nodes, config nodes,
name nodes, head nodes, etc, and all the software loaded on each node is identical. It allows the nodes to be added or removed without considering their “type”. This model works particularly well with cloud infrastructure in general.

A typical Couchbase cluster creation process looks like:

• Start Couchbase: Start n Couchbase servers
• Create cluster: Pick any server, and add all other servers to it to create the cluster
• Rebalance cluster: Rebalance the cluster so that data is distributed across the cluster

In order to automate using Docker Services, the cluster creation is split into a “master” and “worker” service.

The master service has only one replica. This provides a single reference point to start the cluster creation. This service also exposes port 8091. It allows the Couchbase Web Console to
be accessible from outside the cluster.

The worker service uses the exact same image as master service. This keeps the cluster homogenous which allows to scale the cluster easily.

Let’s get started!

Setup Swarm Mode on Ubuntu

1. Launch an Ubuntu instance on Amazon. This blog used mx4.large size for the AMI.
2. Install Docker:

3. Docker Swarm mode is an optional feature and need to be explicitly enabled. Initialize Swarm mode:

Create Couchbase “master” Service

1. Create an overlay network:

   This is required so that multiple Couchbase Docker containers in the cluster can talk to each other.

2. Create a “master” service:
   This image is created using the Dockerfile here. This Dockerfile uses a configuration script to
   configure the base Couchbase Docker image. First, it uses Couchbase REST API to setup memory quota, setup index, data and
   query services, security credentials, and loads a sample data bucket. Then, it invokes the appropriate Couchbase CLI commands to add the
   Couchbase node to the cluster or add the node and rebalance the cluster. This is based upon three environment variables:
   • TYPE: Defines whether the joining pod is worker or master
   • COUCHBASE_MASTER: Name of the master service
   • AUTO_REBALANCE: Defines whether the cluster needs to be rebalanced
   For this first configuration file, the TYPE environment variable is set to MASTER and so no additional configuration is done on the Couchbase image.

   This service also uses the previously created overlay network named couchbase. It exposes the port 8091 that makes the Couchbase Web Console accessible outside the cluster. This service contains only one replica of the container.

3. Check status of the Docker service:

   It shows that the service is running. The “desired” and “expected” number of replicas are 1, and thus are matching.

4. Check the tasks in the service:

   This shows that the container is running.

5. Access Couchbase Web Console using public IP address and it should look like:
   
   The image used in the configuration file is configured with the Administrator username and password password. Enter the credentials to see the console:
   
6. Click on Server Nodes to see how many Couchbase nodes are part of the cluster. As expected, it shows only one node:
   

Create Couchbase “worker” Service

1. Create “worker” service:

   This RC also creates a single replica of Couchbase using the same arungupta/couchbase:swarm image. The key differences here are:

   • TYPE environment variable is set to WORKER. This adds a worker Couchbase node to be added to the cluster.
   • COUCHBASE_MASTER environment variable is passed the name of the master service,  couchbase-master.couchbase in our case. This uses the service discovery mechanism built into Docker for the worker and
     the master to communicate.
2. Check service:

3. Checking the Couchbase Web Console shows the updated output:
   
   It shows that one server is pending to be rebalanced.During the worker service creation, AUTO_REBALANCE environment variable could have been set to true or false to
   enable rebalance. This ensures that the node is only added to the cluster but the cluster itself is not rebalanced. Rebalancing the cluster requires to
   re-distribute the data across multiple nodes of the cluster. The recommended way is to add multiple nodes, and then manually rebalance the cluster using the Web Console.

Add Couchbase Nodes by Scaling Docker Service

1. Scale the service: 


2. Check the service:

   This shows that 2 replicas of worker are running.

3. Check the Couchbase Web Console:
   
   As expected, two servers are now added in the cluster and pending rebalance.
4. Optionally, you can rebalance the cluster by clicking on the Rebalance button. and it will show like:
   
   After the rebalancing is complete, the Couchbase Web Console is updated to as as shown:
   
5. See all the running containers using docker ps:

   0

In addition to creating a cluster, Couchbase Server supports a range of high availability and disaster recovery (HA/DR) strategies. Most HA/DR strategies
rely on a multi-pronged approach of maximizing availability, increasing redundancy within and across data centers, and performing regular backups. Now that your Couchbase cluster is ready, you can run your first sample application.
Learn more about Couchbase and Containers:

• Couchbase on Containers
• Follow us on @couchbasedev or @couchbase
• Ask questions on Couchbase Forums