http://www.couchbase.com/issues This file is an XML representation of an issue en-us 5.2.4 845 26-12-2012 http://www.couchbase.com/issues/browse/MB-7382 Couchbase Server - 2 node cluster <br/> - 2 buckets <br/> - Bucket &#39;bkt&#39; had a very high percentage of sets in its front end load. <br/> - Failed over ec2-54-252-25-132.ap-southeast-2.compute.amazonaws.com, added back, rebalance. <br/> - Rebalance froze at around 98%. <br/> - Stopped front end loads, disk write queue drained. <br/> - Mem used for both nodes, greater than higher water mark. <br/> - Restarted couchbase server, waited for warm up to complete, retried rebalance, rebalance remained frozen at 50%. <br/> - Rebooted nodes, waited for warm up to complete, retried rebalance, rebalance remained frozen at 50%. <br/> <br/> Cluster diags: <br/> 1 <a href="https://s3.amazonaws.com/bugdb/MB-7382/ec2-54-252-25-132.ap-southeast-2.compute.amazonaws.com-8091-diag.txt.gz">https://s3.amazonaws.com/bugdb/MB-7382/ec2-54-252-25-132.ap-southeast-2.compute.amazonaws.com-8091-diag.txt.gz</a> <br/> <br/> 2 <a href="https://s3.amazonaws.com/bugdb/MB-7382/ec2-54-252-20-171.ap-southeast-2.compute.amazonaws.com-8091-diag.txt.gz">https://s3.amazonaws.com/bugdb/MB-7382/ec2-54-252-20-171.ap-southeast-2.compute.amazonaws.com-8091-diag.txt.gz</a> <br/> <br/> Attached the cbstats of all, raw memory for both nodes. <a href="http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1976-rel.deb.manifest.xml">http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1976-rel.deb.manifest.xml</a> <br/> 12.04 Ubuntu LTS ec2 MB-7382

rebalance froze when node failed over and added back (observed mem used > high water mark for bucket)

Bug Major Resolved Won't Fix Public Abhinav Dangeti Abhinav Dangeti Sun, 9 Dec 2012 01:55:47 -0600 Mon, 10 Dec 2012 19:48:27 -0600 Mon, 10 Dec 2012 18:49:42 -0600 2.0 2.0.1 couchbase-bucket 0 0 The load rate from clients was too high, which caused the cluster to be highly overloaded during rebalance. There were lots of backlogs in the replication queues, which caused the bucket &quot;bkt&quot; to have memory usage more than 90% of bucket quota. If memory usage is above 90% of bucket quota, the replication or vbucket takeover would stop.<br/> <br/> If we don&#39;t set up the cluster with the enough capacity, we could have the rebalance issues.<br/> <br/> Please set up the cluster with the enough capacity. Rebalance tests with two nodes wouldn&#39;t be good for system tests. All of our customers use three node cluster at least. Not part of a system test, this was the cluster where I was checking the deleted items&#39; status, I just tried failing over and adding back one of the nodes. Date of First Response Sun, 9 Dec 2012 03:17:08 -0600 Rank 3456 Time In Status