You should monitor the system during and immediately after a rebalance operation until you are confident that replication has completed successfully.
There are essentially two stages to replication:
Backfilling
The first stage of replication involves reading all data for a given active vBucket and sending it to the server that is responsible for the replica. This can put increased load on the disk subsystem as well as network bandwidth but is not designed to impact any client activity.
You can monitor the progress of this task by watching for ongoing TAP disk fetches and/or watching cbstats tap, or
shell> cbstats <node_IP>:11210 -b bucket_name -p bucket_password tap | grep backfillBoth will return a list of TAP backfill processes and whether they are still running (true) or done (false).
When all have completed, you should see the Total Item count
(curr_items_tot) be equal to the number
of active items multiplied by replica count.
If you are continuously adding data to the system, these values may not line up exactly at a given instant in time. However, you should be able to determine whether there is a significant difference between the two figures.
Until this is completed, you should avoid using the "failover" functionality since that may result in loss of the data that has not yet been replicated.
Draining
After the backfill process is completed, all nodes that had replicas materialized on them will then need to persist those items to disk. It is important to continue monitoring the disk write queue and memory usage until the rebalancing operation has been completed, to ensure that your cluster is able to keep up with the write load and required disk I/O.