http://www.couchbase.com/issues This file is an XML representation of an issue en-us 5.2.4 845 26-12-2012 http://www.couchbase.com/issues/browse/MB-6706 Couchbase Server Cluster information: <br/> - 8 centos 6.2 64bit server with 4 cores CPU <br/> - Each server has 32 GB RAM and 400 GB SSD disk. <br/> - SSD disk format ext4 on /data <br/> - Each server has its own drive, no disk sharing with other server. <br/> - Load 15 million items to both buckets <br/> - Cluster has 2 buckets, default (11GB) and saslbucket (11GB) with consistent view enable. For 2 buckets, we use only 68% total RAM of system. <br/> - Each bucket has one doc and 2 views for each doc (default d1 and saslbucket d11) <br/> <br/> * Create cluster with 4 nodes installed couchbase server 2.0.0-1746 <br/> <br/> 10.6.2.37 <br/> 10.6.2.38 <br/> 10.6.2.39 <br/> 10.6.2.40 <br/> <br/> * Data path /data <br/> * View path /data <br/> <br/> * Add 4 nodes to cluster and rebalance <br/> 10.6.2.42 <br/> 10.6.2.43 <br/> 10.6.2.44 <br/> 10.6.2.45 <br/> <br/> * rebalance hang <br/> <br/> * Link to collect info of all nodes <a href="https://s3.amazonaws.com/packages.couchbase/collect_info/orange/2_0_0/201209/8nodes-col-info-1746-reb-hang-20120920.tgz">https://s3.amazonaws.com/packages.couchbase/collect_info/orange/2_0_0/201209/8nodes-col-info-1746-reb-hang-20120920.tgz</a> <br/> <br/> Link to atop file of all nodes <a href="https://s3.amazonaws.com/packages.couchbase/atop-files/orange/201209/atop-8nodes-1746-reb-hang-20120920.tgz">https://s3.amazonaws.com/packages.couchbase/atop-files/orange/201209/atop-8nodes-1746-reb-hang-20120920.tgz</a> centos 6.2 64bit build 2.0.0-1746 MB-6706

[system test] rebalance hang when add nodes to cluster

Bug Blocker Closed Fixed Public Filipe Manana Thuan Nguyen system-test Thu, 20 Sep 2012 20:58:58 -0500 Thu, 10 Jan 2013 01:21:35 -0600 Wed, 3 Oct 2012 13:36:47 -0500 2.0 2.0 ns_server 0 0 I think this bug the same as bug <a href="http://www.couchbase.com/issues/browse/MB-6707" title="[system test] rebalance does not stop when click on &quot;Stop Rebalance&quot; button"><strike>MB-6707</strike></a> Filipe, there are no crashes and you can see in logs of .38 that we&#39;re waiting for index update (there&#39;s just 1 simple index) and this does not happen.<br/> <br/> <br/> [ns_server:debug,2012-09-20T17:20:30.431,<a href='mailto:ns_1@10.6.2.38'>ns_1@10.6.2.38</a>:&lt;0.21339.32&gt;:capi_set_view_manager:do_wait_index_updated:596]References to wait: [#Ref&lt;0.0.309.194865&gt;] (&quot;saslbucket&quot;, 531)<br/> <br/> I advise you to take a quick look at source of do_wait_index_updated in capi_set_view_manager. Maybe you will spot something that I&#39;m not doing right. Hit this bug again in build 2.0.0-1777 with swap rebalance Add node 44, 45 and remove node 39, 40 <br/> Rebalance hang after moving some items to new added nodes <br/> <br/> Cluster information: <br/> - 8 centos 6.2 64bit server with 4 cores CPU <br/> - Each server has 32 GB RAM and 400 GB SSD disk. <br/> - 24.8 GB RAM for couchbase server at each node <br/> - SSD disk format ext4 on /data <br/> - Each server has its own SSD drive, no disk sharing with other server. <br/> - Create cluster with 6 nodes installed couchbase server 2.0.0-1777 <br/> - Cluster has 2 buckets, default (12GB) and saslbucket (12GB). <br/> - Each bucket has one doc and 2 views for each doc (default d1 and saslbucket d11) <br/> <br/> <br/> 10.6.2.37 <br/> 10.6.2.38 <br/> 10.6.2.39 <br/> 10.6.2.40 <br/> 10.6.2.42 <br/> 10.6.2.43 <br/> <br/> * Load 18 million items to both bucket. Each key has size from 512 bytes to 1024 bytes <br/> * Queries all 4 views from 2 docs <br/> <br/> <br/> 10.6.2.44 <br/> 10.6.2.45 <br/> <br/> * Data path /data <br/> * View path /data <br/> <br/> Link to collect info of all nodes <a href="https://s3.amazonaws.com/packages.couchbase/collect_info/orange/2_0_0/201209/8nodes-col-info-1777-swap-reb-hang-20120927-155552.tgz">https://s3.amazonaws.com/packages.couchbase/collect_info/orange/2_0_0/201209/8nodes-col-info-1777-swap-reb-hang-20120927-155552.tgz</a> <br/> <br/> Link to atop of all nodes <a href="https://s3.amazonaws.com/packages.couchbase/atop-files/orange/201209/atop-8nodes-1777-swap-reb-hang-20120927-155750.tgz">https://s3.amazonaws.com/packages.couchbase/atop-files/orange/201209/atop-8nodes-1777-swap-reb-hang-20120927-155750.tgz</a> <br/> <br/> Hit this bug again in build 2.0.0-1781 in system test. <br/> <br/> * Add 2 nodes: 39 and 40 and rebalance. During rebalance, reboot node 42 and 43. Rebalance failed as expected. <br/> * After node finished warmup, rebalance again. Rebalance failed with bug <a href="http://www.couchbase.com/issues/browse/MB-6490" title="Rebalance failed with reason &quot;Partition 687 not in active nor passive set&quot; in add in node rebalance"><strike>MB-6490</strike></a> on node 44. <br/> * Failover node 44 and rebalance. <br/> * Cluster rebalance saslbucket first. Rebalance was done after 17 hrs <br/> <br/> Started rebalancing bucket saslbucket ns_rebalancer000 <a href='mailto:ns_1@10.6.2.37'>ns_1@10.6.2.37</a> 14:44:27 - Mon Oct 1, 2012 <br/> Started rebalancing bucket default ns_rebalancer000 <a href='mailto:ns_1@10.6.2.37'>ns_1@10.6.2.37</a> 08:14:08 - Tue Oct 2, 2012 <br/> <br/> ** Rebalance of default bucket hang around 10:00AM Tue Oct 2, 2012 as in capture screen <br/> <br/> <br/> Cluster information: <br/> - 8 centos 6.2 64bit server with 4 cores CPU <br/> - Each server has 32 GB RAM and 400 GB SSD disk. <br/> - 24.8 GB RAM for couchbase server at each node <br/> - SSD disk format ext4 on /data <br/> - Each server has its own SSD drive, no disk sharing with other server. <br/> - Create cluster with 6 nodes installed couchbase server 2.0.0-1781 <br/> - Cluster has 2 buckets, default (12GB) and saslbucket (12GB). <br/> - Each bucket has one doc and 2 views for each doc (default d1 and saslbucket d11) <br/> - Enable consistent view on cluster (by default) <br/> <br/> 10.6.2.37 <br/> 10.6.2.38 <br/> 10.6.2.44 <br/> 10.6.2.45 <br/> 10.6.2.42 <br/> 10.6.2.43 <br/> <br/> * Load 14 million items to each bucket. Each key has size from 512 bytes to 1024 bytes <br/> * Mutate 14 million items to each bucket with size of each key from 1024 to 1500 bytes <br/> * Load running about 8K to 10K ops on both buckets <br/> * Queries all 4 views from 2 docs <br/> <br/> <br/> 10.6.2.39 <br/> 10.6.2.40 <br/> <br/> * Data path /data <br/> * View path /data <br/> <br/> Manifest info from build 1781 <br/> <a href="http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1781-rel.rpm.manifest.xml">http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1781-rel.rpm.manifest.xml</a> <br/> <br/> Link to collect info of all nodes <a href="https://s3.amazonaws.com/packages.couchbase/collect_info/orange/2_0_0/201210/8nodes-col-info-1781-rebalance-hang-20121002-114333.tgz">https://s3.amazonaws.com/packages.couchbase/collect_info/orange/2_0_0/201210/8nodes-col-info-1781-rebalance-hang-20121002-114333.tgz</a> <br/> <br/> Link to tap stats from all nodes <a href="https://friendpaste.com/6JqjtMOwZLvmlx5h9fxt6L">https://friendpaste.com/6JqjtMOwZLvmlx5h9fxt6L</a> <br/> Promote it to blocker since we hit it often in system test. I killed loads on default bucket (currently rebalancing but hang), rebalance started after five minutes all loads stopped. Few minutes later, restart half loads on default bucket, rebalance continues running Integrated in github-couchdb-preview #509 (See [<a href="http://qa.hq.northscale.net/job/github-couchdb-preview/509/">http://qa.hq.northscale.net/job/github-couchdb-preview/509/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.couchbase.com/issues/browse/MB-6706" title="[system test] rebalance hang when add nodes to cluster"><strike>MB-6706</strike></a> Trigger update after defining indexable partitions (Revision 9098ff069968247556da72a2be1bbfd944b1d30e)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> pwansch : <br/> Files : <br/> * src/couch_set_view/src/couch_set_view_group.erl<br/> Date of First Response Mon, 24 Sep 2012 01:46:17 -0500 Rank 3923 Time In Status