consequent rebalance fails when adding 4th node to a cluster after one node is warmed up

Farshid Ghods added a comment - 17/Aug/12 7:20 PM diags : http://www.couchbase.com/issues/secure/attachment/14374/10.3.121.16-8091-diag.txt.gz http://www.couchbase.com/issues/secure/attachment/14373/10.3.121.15-8091-diag.txt.gz http://www.couchbase.com/issues/secure/attachment/14372/10.3.121.14-8091-diag.txt.gz http://www.couchbase.com/issues/secure/attachment/14371/10.3.121.13-8091-diag.txt.gz

Farshid Ghods added a comment - 17/Aug/12 7:20 PM diags : http://www.couchbase.com/issues/secure/attachment/14374/10.3.121.16-8091-diag.txt.gz http://www.couchbase.com/issues/secure/attachment/14373/10.3.121.15-8091-diag.txt.gz http://www.couchbase.com/issues/secure/attachment/14372/10.3.121.14-8091-diag.txt.gz http://www.couchbase.com/issues/secure/attachment/14371/10.3.121.13-8091-diag.txt.gz

Andrei Baranouski added a comment - 17/Aug/12 9:22 PM - edited after the same steps I tried to repeat the rebalance again. so, rebalance was failed in the second time, but was passed in third. new diags are attached also noticed that several items had been lost: sasl bucket: before- 3198589, after- 3198310 default: before - 4401961, after - 4401508

Peter Wansch added a comment - 18/Aug/12 12:34 PM Alk, I believe this is the most important one to address next.

Aleksey Kondratenko added a comment - 18/Aug/12 8:12 PM I'd like us to start gathering memcached logs for: *) all bugreports (manually by QE at least for now) *) as part of cbcollect_info

Farshid Ghods added a comment - 18/Aug/12 8:28 PM notified the team about this requirements. about this bug cbcollect info is uploaded but i am going to ssh and see if i can extract memcached.log and attach it seperately now

Farshid Ghods added a comment - 18/Aug/12 8:36 PM rebalance succeeded after three attempts oin this cluster i grabbed memcached log files from all nodes anyways

Aleksey Kondratenko added a comment - 18/Aug/12 11:07 PM Sorry folks, I was not in my best shape today. That's 100% duplicate of that bug Tony filed. You may need 3-4, up to <number of nodes> - 1 rebalances in order to clean up those dead connections. See MB-6216

Farshid Ghods added a comment - 18/Aug/12 11:38 PM Thanks Alk Is there another workaround like running a command or rebooting that node twice instead of once to kill those dead connections or the only way is to rebalance again

Thuan Nguyen added a comment - 19/Aug/12 4:18 AM Following these steps 1. 3 nodes 10.3.121.23, 24, 25 2. reboot node 24, and wait while ep_warmup_thread: complete 3. add 4-th node (26) and rebalance Successful reblance in first run. It took more than 2 hrs 34 minutes to finish. During rebalance, node 24 become unstable (yellow). I monitor atop in node 24 and see disk use 100%. Then I disable compaction in UI both data and view. Node 24 back to green. Eventhough compaction was disable, compaction in view5 production still running. When ever view5 production running compaction, rebalance process was paused (look at vbucket moving between nodes). That made rebalance took more than 2 hours to finish. Cluster has 16 millions items with resident ratio 55%, 12 dev views, 2 pro views, load about 3K ops

Aleksey Kondratenko added a comment - 19/Aug/12 2:11 PM Tony, if you think that slowness is a problem, or something else is a problem, please file new bug.

Thuan Nguyen added a comment - 19/Aug/12 2:19 PM I will try on latest build 705. If I can repro this slowness during rebalance, I will file new bug.

Farshid Ghods added a comment - 19/Aug/12 8:18 PM all, please retest with build 709

Ketaki Gangal added a comment - 19/Aug/12 10:42 PM Verified on 709. Rebalance successful.

Andrei Baranouski added a comment - 13/Sep/12 5:53 PM build 1717 - rebalance successful.

Andrei Baranouski added a comment - 13/Sep/12 5:53 PM verified on 1717