[MB-7290] Rebalance-in operation failed twice with "bulk_set_vbucket_state" failing with heavy front end load on an XDCR set up and with system in DGM (~65% resident ratio)

Wed, 10 Apr 2013 15:12:09 -0500

At the time of the rebalance failure:

+ 5 nodes rebalance in on each cluster
Cluster setup: c1:c2::10:10
biXDCR_bucket: c1 <---> c2
uniXDCR_src: c1 ---> c2 :uniXDCR_dest
Front end loads on c1 and c2 for biXDCR_bucket, and on c1 for uniXDCR_src.
c1: http://ec2-177-71-230-72.sa-east-1.compute.amazonaws.com:8091/
c2: http://ec2-175-41-186-167.ap-southeast-1.compute.amazonaws.com:8091/

On C1, Rebalance operation failed with this reason on the UI logs:

Rebalance exited with reason {{bulk_set_vbucket_state_failed,
[{'ns_1@ec2-177-71-170-44.sa-east-1.compute.amazonaws.com',
{'EXIT',
{{timeout,
{gen_server,call,
['ns_memcached-biXDCR_bucket',
{set_vbucket,544,replica},
180000]}},
{gen_server,call,
[{'janitor_agent-biXDCR_bucket',
'ns_1@ec2-177-71-170-44.sa-east-1.compute.amazonaws.com'},
{if_rebalance,<0.10136.88>,
{update_vbucket_state,544,replica,
undefined,undefined}},
infinity]}}}}]},
[{janitor_agent,bulk_set_vbucket_state,4},
{ns_vbucket_mover,
update_replication_post_move,3},
{ns_vbucket_mover,handle_info,2},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]}

The second time, rebalance failed with the following UI log message:

Rebalance exited with reason {{timeout,
{gen_server,call,
['ns_memcached-biXDCR_bucket',
{set_vbucket,849,active},
180000]}},
{gen_server,call,
[{'janitor_agent-biXDCR_bucket',
'ns_1@ec2-177-71-230-72.sa-east-1.compute.amazonaws.com'},
{if_rebalance,<0.21090.114>,
{update_vbucket_state,849,active,paused,
undefined}},
infinity]}}

After giving it some time, the third rebalance did complete successfully.

Will attach the grabbed diags from one of the nodes at C1 in a bit.

Couchbase

[MB-7290] Rebalance-in operation failed twice with "bulk_set_vbucket_state" failing with heavy front end load on an XDCR set up and with system in DGM (~65% resident ratio)