Swap Rebalance functionality was added to Couchbase Server 1.8.1.
Swap Rebalance is an automatic feature that optimizes the movement of data when you are adding and removing the same number of nodes within the same operation. The swap rebalance optimizes the rebalance operation by moving data directly from the nodes being removed to the nodes being added. This is more efficient than standard rebalancing which would normally move data across the entire cluster.
Swap rebalance can be used during an upgrade process to upgrade all the nodes of the cluster, without requiring data to be moved around the cluster degrading the performance. For more information, Section 2.3.1.1, “Upgrading to Couchbase Server 1.8.1 Using Swap Rebalance”.
Swap rebalance only occurs if the following are true:
You are removing and adding the same number of nodes during rebalance. For example, if you have marked two nodes to be removed, and added another two nodes to the cluster.
There must be at least one Couchbase Server 1.8.1 node in the cluster.
Swap rebalance occurs automatically if the number of nodes being added and removed are identical. There is no configuration or selection mechanism to force a swap rebalance. If a swap rebalance cannot take place, then a normal rebalance operation will be used instead.
When Couchbase Server 1.8.1 identifies that a rebalance is taking place and that there are an even number of nodes being removed and added to the cluster, the swap rebalance method is used to perform the rebalance operation.
When a swap rebalance takes place, the rebalance operates as follows:
Data will be moved directly from a node being removed to a node being added on a one-to-one basis. This eliminates the need to restructure the entire vBucket map.
Active vBuckets are moved, one at a time, from a source node to a destination node.
Replica vBuckets are created on the new node and populated with existing data before being activated as the live replica bucket. This ensures that if there is a failure during the rebalance operation, that your replicas are still in place.
For example, if you have a cluster with 20 nodes in it, and configure two nodes (X and Y) to be added, and two nodes to be removed (A and B):
vBuckets from node A will be moved to node X.
vBuckets from node B will be moved to node Y.
The benefits of swap rebalance are:
Reduced rebalance duration. Since the move takes place directly from the nodes being removed to the nodes being added.
Reduced load on the cluster during rebalance.
Reduced network overhead during the rebalance.
Reduced chance of a rebalance failure if a failover occurs during the rebalance operation, since replicas are created in tandem on the new hosts while the old host replicas still remain available.
Because data on the nodes are swapped, rather than performing a full rebalance, the capacity of the cluster remains unchanged during the rebalance operation, helping to ensure performance and failover support.
The behaviour of the cluster during a failover and rebalance operation with the swap rebalance functionality affects the following situations:
Stopping a rebalance
If rebalance fails, or has been deliberately stopped, the active and replica vBuckets that have been transitioned will be part of the active vBucket map. Any transfers still in progress will be canceled. Restarting the rebalance operation will continue the rebalance from where it left off.
Adding back a failed node
When a node has failed, removing it and adding a replacement node, or adding the node back, will be treated as swap rebalance.
With swap rebalance functionality, after a node has failed over, you should either clean up and re-add the failed over node, or add a new node and perform a rebalance as normal. The rebalance will be handled as a swap rebalance which will minimize the data movements without affecting the overall capacity of the cluster.