Couchbase cluster failing to rebalance
Hi,
We're using Couchbase Version: 1.8.0 enterprise edition (build-55). We have 3 servers in the cluster, two real machines S1 and S2 and one virtual machine S3 running in S1 (OS is Ubuntu 10.04.3 LTS on all 3 servers). While we were testing failover capabilities we were turning S3 on and off. At some point rebalance started to fail (I've replaced IP addresses with server names):
--------------
Starting rebalance, KeepNodes = ['ns_1@S2','ns_1@S1',
'ns_1@S3'], EjectNodes = []
ns_orchestrator004 ns_1@S1 16:24:40 - Tue Apr 17, 2012
--------------
Shutting down bucket "default" on 'ns_1@S2' for server shutdown ns_memcached002 ns_1@S2 16:24:45 - Tue Apr 17, 2012
--------------
Rebalance exited with reason {noproc,
{gen_server,call,
[{'ns_memcached-default','ns_1@S2'},
{get_vbucket,83},
30000]}}
ns_orchestrator002 ns_1@S1 16:24:45 - Tue Apr 17, 2012
--------------
Bucket "default" loaded on node 'ns_1@S2' in 0 seconds. ns_memcached001 ns_1@S2 16:24:47 - Tue Apr 17, 2012
--------------
Thanks in advance,
Kirill
UPDATE
Sometimes fail reason is
Rebalance exited with reason {noproc,
{gen_server,call,
[{'ns_vbm_sup-default','ns_1@S2'},
which_children,infinity]}}
Besides, argument of get_vbucket call (83 in {get_vbucket,83} in my post above) is different every time.