When using Couchbase Server version 4.6.1 we noticed a significant increase in XDCR traffic and fluctuating bucket record counts when a failover and rebalance was applied on one of the clusters. This failover/rebalance was invoke because we needed to take a cluster member out of service temporarily. This increase was visible on both the local cluster where the member was taken out of service as well as on the remote cluster.
Some time after the rebalance had completed, the XDCR traffic reduced again. End result is that the proper/correct record counters are back.
During this whole process, no records were added to the buckets.
Is this expected behavior? What explains the fluctuating record counts? What explains the increased XDCR traffic? Would our application be affected in the sense that it might find records that are there only as a result of the XDCR activity (afterall records counts were increasing and reducing during this process so i suppose some records were only present temporarily during the process)?
In general, in case you need to do a planned failover and rebalance of any cluster node, would you recommend to pause XDCR?