Rebalancing a cluster involves marking nodes to be added or removed from the cluster, and then starting the rebalance operation so that the data is moved around the cluster to reflect the new structure.
For information on adding nodes to your cluster, see Section 18.104.22.168, “Adding a Node to a Cluster”.
For information on adding nodes to your cluster, see Section 22.214.171.124, “Removing a Node from a Cluster”.
In the event of a failover situation, a rebalance is required to bring the cluster back to a healthy state and re-enable the configured replicas. For more information on how to handle a failover situation, see Section 5.5, “Failing Over Nodes”
The Couchbase Admin Web Console will indicate when the cluster requires a rebalance because the structure of the cluster has been changed, either through adding a node, removing a node, or due to a failover. The notification is through the count of the number of servers that require a rebalance. You can see a sample of this in the figure below, here shown on the Manage Server Nodes page
To rebalance the cluster, you must initiate the rebalance process, detailed in Section 126.96.36.199, “Starting a Rebalance”.
There are a number of methods available for adding a node to a cluster. The result is the same in each case, the node is marked to be added to the cluster, but the node is not an active member until you have performed a rebalance operation.
The methods are:
Web Console — During Installation
When you are performing the Setup of a new Couchbase Server installation (see Section 2.3, “Initial Server Setup”), you have the option of joining the new node to an existing cluster.
During the first step, you can select the Join a cluster now radio button, as shown in the figure below:
You are prompted for three pieces of information:
The IP address of any existing node within the cluster you want to join.
The username of the administrator of the target cluster.
The password of the administrator of the target cluster.
The node will be created as a new cluster, but the pending status of the node within the new cluster will be indicated on the Cluster Overview page, as seen in the example below:
Web Console — After Installation
You can add a new node to an existing cluster after installation by clicking the Manage Server Nodes area of the Admin Console. You can see the button in the figure below.button within the
You will be presented with a dialog box, as shown below. Couchbase Server should be installed, and should have been configured as per the normal setup procedures. You can also add a server that has previously been part of this or another cluster using this method. The Couchbase Server must be running.
You need to fill in the requested information:
Server IP Address
The IP address of the server that you want to add.
The username of the administrator of the target node.
The password of the administrator of the target node.
You will be provided with a warning notifying you that the operation is destructive on the destination server. Any data currently stored on the server will be deleted, and if the server is currently part of another cluster, it will be removed and marked as failed over in that cluster.
Once the information has been entered successfully, the node will be marked as ready to be added to the cluster, and the servers pending rebalance count will be updated.
Using the REST API
Using the REST API, you can add nodes to the cluster by providing the IP address, administrator username and password as part of the data payload. For example, using curl you could add a new node:
shell> curl -u
cluster-password\ localhost:8091/controller/addNode \ -d "hostname=
For more information, see Section 8.7.2, “Adding a Node to a Cluster”.
Using the Command-line
You can use the couchbase-cli command-line tool to add one or more nodes to an existing cluster. The new nodes must have Couchbase Server installed, and Couchbase Server must be running on each node.
To add, run the command:
shell> couchbase-cli server-add \ --cluster=
|The IP address of a node in the existing cluster.|
|The username for the existing cluster.|
|The password for the existing cluster.|
|The IP address of the node to be added to the cluster.|
|The username of the node to be added.|
|The password of the node to be added.|
If the add process is successful, you will see the following response:
SUCCESS: server-add 192.168.0.72:8091
If you receive a failure message, you will be notified of the type of failure.
You can add multiple nodes in one command by supplying
options to the command.
Once a server has been successfully added, the Couchbase Server cluster will indicate that a rebalance is required to complete the operation.
You can cancel the addition of a node to a cluster without having to perform a rebalance operation. Canceling the operation will remove the server from the cluster without having transferred or exchanged any data, since no rebalance operation took place. You can cancel the operation through the web interface.
Removing a node marks the node for removal from the cluster, and will completely disable the node from serving any requests across the cluster. Once removed, a node is no longer part of the cluster in any way and can be switched off, or can be updated or upgraded.
Before you remove a node from the cluster, you should ensure that you have the capacity within the remaining nodes of your cluster to handle your workload. For more information on the considerations, see Choosing when to shrink your cluster. For the best results, use swap rebalance to swap the node you want to remove out, and swap in a replacement node. For more information on swap rebalance, see Section 5.7.3, “Swap Rebalance”.
Like adding nodes, there are a number of solutions for removing a node:
You can remove a node from the cluster from within the Manage Server Nodes section of the Web Console, as shown in the figure below.
To remove a node, click thebutton next to the node you want to remove. You will be provided with a warning to confirm that you want to remove the node. Click to mark the node for removal.
Using the Command-line
You cannot mark a node for removal from the command-line
without also initiating a rebalance operation. The
rebalance command accepts one or more
--server-remove options. This adds or
removes the server from the cluster, and immediately
initiates a rebalance operation.
For example, to remove a node during a rebalance operation:
shell> couchbase-cli rebalance --cluster=127.0.0.1:8091 \ -u Administrator -p Password \ --server-remove=192.168.0.73
For more information on the rebalance operation, see Section 188.8.131.52, “Starting a Rebalance”.
Removing a node does not stop the node from servicing requests. Instead, it only marks the node ready for removal from the cluster. You must perform a rebalance operation to complete the removal process.
Once you have configured the nodes that you want to add or remove from your cluster, you must perform a rebalance operation. This moves the data around the cluster so that the data is distributed across the entire cluster, removing and adding data to different nodes in the process.
If Couchbase Server identifies that a rebalance is required, either through explicit addition or removal, or through a failover, then the cluster is in a pending rebalance state. This does not affect the cluster operation, it merely indicates that a rebalance operation is required to move the cluster into its configured state.
To initiate a rebalance operation:
Using the Web Console
Within the Manage Server Nodes area of the Couchbase Administration Web Console, a cluster pending a rebalance operation will have enabled the button.
Clicking this button will immediately initiate a rebalance operation. You can monitor the progress of the rebalance operation through the web console. The progress of the movement of vBuckets is provided for each server by showing the movement progress as a percentage.
You can stop a rebalance operation at any time during the process by clicking thebutton. This only stops the rebalance operation, it does not cancel the operation. You should complete the rebalance operation.
Using the Command-line
You can initiate a rebalance using the
couchbase-cli and the
shell> couchbase-cli rebalance -c 127.0.0.1:8091 -u Administrator -p Password INFO: rebalancing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . SUCCESS: rebalanced cluster
You can also use this method to add and remove nodes and
initiate the rebalance operation using a single command.
You can specify nodes to be added using the
--server-add option, and nodes to be
removed using the
can use multiple options of each type. For example, to add
two nodes, and remove two nodes, and immediately initiate
a rebalance operation:
shell> couchbase-cli rebalance -c 127.0.0.1:8091 \ -u Administrator -p Password \ --server-add=192.168.0.72 \ --server-add=192.168.0.73 \ --server-remove=192.168.0.70 \ --server-remove=192.168.0.69
The command-line provides an active view of the progress and will only return once the rebalance operation has either completed successfully, or in the event of a failure.
You can stop the rebalance operation by using the
stop-rebalance command to
The time taken for a rebalance operation is entirely dependent on the number of servers, quantity of data, cluster performance and any existing cluster activity, and is therefore impossible to accurately predict or estimate.
Throughout any rebalance operation you should monitor the process to ensure that it completes successfully, see Section 5.7.4, “Monitoring During Rebalance”.