http://www.couchbase.com/issues This file is an XML representation of an issue en-us 5.2.4 845 26-12-2012 http://www.couchbase.com/issues/browse/MB-6481 Couchbase Server Install couchbase server 2.0.0-1645 on 11 nodes <br/> 10.3.121.13 <br/> 10.3.121.14 <br/> 10.3.121.15 <br/> 10.3.121.16 <br/> 10.3.121.17 <br/> 10.3.121.20 <br/> 10.3.121.22 <br/> 10.3.121.26 <br/> 10.3.121.24 <br/> 10.3.121.25 <br/> 10.3.121.23 <br/> Create cluster 10 nodes <br/> Create 2 buckets, default and saslbucket <br/> Create 1 docs with 2 views per doc for each bucket <br/> Load 7+ million items to default bucket and 1+ million items to saslbucket <br/> Maintain load about 12 K ops on default bucket and 3 K ops on saslbucket <br/> Continuously querying views on both buckets <br/> Rebalance in and out node <br/> Then do swap rebalance by add 3 nodes ( 24, 25 and 26) to cluster and remove 3 nodes (20, 22 and 23) out of cluster <br/> After 3 minutes rebalance, rebalance failed with error <br/> <br/> <br/> 2012-08-29 01:03:09.578 ns_orchestrator:4:info:message(<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>) - Starting rebalance, KeepNodes = [&#39;<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>&#39;,&#39;<a href='mailto:ns_1@10.3.121.14'>ns_1@10.3.121.14</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.3.121.15'>ns_1@10.3.121.15</a>&#39;,&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.3.121.17'>ns_1@10.3.121.17</a>&#39;,&#39;<a href='mailto:ns_1@10.3.121.24'>ns_1@10.3.121.24</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>&#39;,&#39;<a href='mailto:ns_1@10.3.121.26'>ns_1@10.3.121.26</a>&#39;], EjectNodes = [&#39;<a href='mailto:ns_1@10.3.121.20'>ns_1@10.3.121.20</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.3.121.22'>ns_1@10.3.121.22</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.3.121.23'>ns_1@10.3.121.23</a>&#39;] <br/> <br/> 2012-08-29 01:03:09.893 ns_storage_conf:0:info:message(<a href='mailto:ns_1@10.3.121.24'>ns_1@10.3.121.24</a>) - Deleting old data files of bucket &quot;saslbucket&quot; <br/> 2012-08-29 01:03:09.893 ns_storage_conf:0:info:message(<a href='mailto:ns_1@10.3.121.24'>ns_1@10.3.121.24</a>) - Deleting old data files of bucket &quot;default&quot; <br/> 2012-08-29 01:03:09.896 ns_storage_conf:0:info:message(<a href='mailto:ns_1@10.3.121.26'>ns_1@10.3.121.26</a>) - Deleting old data files of bucket &quot;saslbucket&quot; <br/> 2012-08-29 01:03:09.898 ns_storage_conf:0:info:message(<a href='mailto:ns_1@10.3.121.26'>ns_1@10.3.121.26</a>) - Deleting old data files of bucket &quot;default&quot; <br/> 2012-08-29 01:03:09.942 ns_storage_conf:0:info:message(<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>) - Deleting old data files of bucket &quot;saslbucket&quot; <br/> 2012-08-29 01:03:09.942 ns_storage_conf:0:info:message(<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>) - Deleting old data files of bucket &quot;default&quot; <br/> 2012-08-29 01:03:13.771 ns_rebalancer:0:info:message(<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>) - Started rebalancing bucket saslbucket <br/> 2012-08-29 01:03:14.794 ns_memcached:1:info:message(<a href='mailto:ns_1@10.3.121.26'>ns_1@10.3.121.26</a>) - Bucket &quot;saslbucket&quot; loaded on node &#39;<a href='mailto:ns_1@10.3.121.26'>ns_1@10.3.121.26</a>&#39; in 0 seconds. <br/> 2012-08-29 01:03:14.820 ns_memcached:1:info:message(<a href='mailto:ns_1@10.3.121.24'>ns_1@10.3.121.24</a>) - Bucket &quot;saslbucket&quot; loaded on node &#39;<a href='mailto:ns_1@10.3.121.24'>ns_1@10.3.121.24</a>&#39; in 0 seconds. <br/> 2012-08-29 01:03:14.831 ns_memcached:1:info:message(<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>) - Bucket &quot;saslbucket&quot; loaded on node &#39;<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>&#39; in 0 seconds. <br/> 2012-08-29 01:03:16.674 ns_vbucket_mover:0:info:message(<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>) - Bucket &quot;saslbucket&quot; rebalance appears to be swap rebalance <br/> 2012-08-29 01:06:21.275 ns_orchestrator:2:info:message(<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>) - Rebalance exited with reason {{bulk_set_vbucket_state_failed, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{&#39;<a href='mailto:ns_1@10.3.121.20'>ns_1@10.3.121.20</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&#39;EXIT&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{{timeout, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server,call, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[timeout_diag_logger, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{diag, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{timeout, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server,call,[ns_config,get]}}}]}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server,call, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{&#39;janitor_agent-saslbucket&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.3.121.20'>ns_1@10.3.121.20</a>&#39;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{if_rebalance,&lt;0.12846.432&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{update_vbucket_state,294,replica, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;undefined,undefined}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;infinity]}}}}]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{janitor_agent,bulk_set_vbucket_state,4}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{ns_vbucket_mover, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;update_replication_post_move,3}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{ns_vbucket_mover,handle_info,2}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server,handle_msg,5}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{proc_lib,init_p_do_apply,3}]} <br/> <br/> <br/> <br/> Look at log on node 20 around 2012-08-29 01:06:21 before rebalance failed, see memcached connection closed by force <br/> <br/> [ns_server:info,2012-08-29T1:05:49.620,<a href='mailto:ns_1@10.3.121.20'>ns_1@10.3.121.20</a>:ns_config_rep:ns_config_rep:do_pull:341]Pulling config from: &#39;<a href='mailto:ns_1@10.3.121.22'>ns_1@10.3.121.22</a>&#39; <br/> <br/> [ns_server:debug,2012-08-29T1:05:50.500,<a href='mailto:ns_1@10.3.121.20'>ns_1@10.3.121.20</a>:&#39;capi_ddoc_replication_srv-default&#39;:cb_generic_replication_srv:handle_info:144]doing replicate_newnodes_docs <br/> [ns_server:debug,2012-08-29T1:05:50.656,<a href='mailto:ns_1@10.3.121.20'>ns_1@10.3.121.20</a>:ns_config_rep:ns_config_rep:handle_call:113]Got full synchronization request from &#39;<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>&#39; <br/> [ns_server:debug,2012-08-29T1:05:50.660,<a href='mailto:ns_1@10.3.121.20'>ns_1@10.3.121.20</a>:ns_config_log:ns_config_log:log_common:111]config change: <br/> buckets -&gt; <br/> [{configs,[{&quot;default&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{map,[]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{fastForwardMap,[]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{uuid,&lt;&lt;&quot;cdf4999f6920e2d4c3526e17f36c843f&quot;&gt;&gt;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{sasl_password,[]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{num_replicas,2}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{replica_index,true}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{ram_quota,3145728000}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{auth_type,sasl}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{autocompaction,false}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{type,membase}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{num_vbuckets,1024}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{servers,[&#39;<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>&#39;,&#39;<a href='mailto:ns_1@10.3.121.14'>ns_1@10.3.121.14</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.3.121.15'>ns_1@10.3.121.15</a>&#39;,&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.3.121.17'>ns_1@10.3.121.17</a>&#39;,&#39;<a href='mailto:ns_1@10.3.121.20'>ns_1@10.3.121.20</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.3.121.22'>ns_1@10.3.121.22</a>&#39;,&#39;<a href='mailto:ns_1@10.3.121.23'>ns_1@10.3.121.23</a>&#39;]}]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&quot;saslbucket&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{map,[{202, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.3.121.15'>ns_1@10.3.121.15</a>&#39;,&#39;<a href='mailto:ns_1@10.3.121.22'>ns_1@10.3.121.22</a>&#39;,&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39;], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.3.121.15'>ns_1@10.3.121.15</a>&#39;,&#39;<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39;]}]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{fastForwardMap,[]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{uuid,&lt;&lt;&quot;9f1fdb211536936b49c8a392e079ecb0&quot;&gt;&gt;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{num_replicas,2}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{replica_index,true}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{ram_quota,3145728000}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{auth_type,sasl}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{sasl_password,&quot;password&quot;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{autocompaction,false}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{type,membase}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{num_vbuckets,1024}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{servers,[&#39;<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>&#39;,&#39;<a href='mailto:ns_1@10.3.121.14'>ns_1@10.3.121.14</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.3.121.15'>ns_1@10.3.121.15</a>&#39;,&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.3.121.17'>ns_1@10.3.121.17</a>&#39;,&#39;<a href='mailto:ns_1@10.3.121.24'>ns_1@10.3.121.24</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.3.121.25'>ns_1@10.3.121.25</a>&#39;,&#39;<a href='mailto:ns_1@10.3.121.26'>ns_1@10.3.121.26</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.3.121.20'>ns_1@10.3.121.20</a>&#39;,&#39;<a href='mailto:ns_1@10.3.121.22'>ns_1@10.3.121.22</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.3.121.23'>ns_1@10.3.121.23</a>&#39;]}]}]}] <br/> [ns_server:debug,2012-08-29T1:05:50.660,<a href='mailto:ns_1@10.3.121.20'>ns_1@10.3.121.20</a>:ns_config_rep:ns_config_rep:handle_call:119]Fully synchronized config in 458 us <br/> [ns_server:info,2012-08-29T1:05:50.857,<a href='mailto:ns_1@10.3.121.20'>ns_1@10.3.121.20</a>:ns_port_memcached:ns_port_server:log:169]memcached&lt;0.506.0&gt;: Wed Aug 29 08:05:50.552220 3: TAP (Producer) eq_tapq:replication_building_272_&#39;<a href='mailto:ns_1@10.3.121.17'>ns_1@10.3.121.17</a>&#39; - disconnected, keep alive for 300 seconds <br/> memcached&lt;0.506.0&gt;: Wed Aug 29 08:05:50.623620 3: TAP (Producer) eq_tapq:replication_building_272_&#39;<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>&#39; - disconnected, keep alive for 300 seconds <br/> <br/> [ns_server:debug,2012-08-29T1:05:50.902,<a href='mailto:ns_1@10.3.121.20'>ns_1@10.3.121.20</a>:&#39;capi_ddoc_replication_srv-default&#39;:cb_generic_replication_srv:handle_info:144]doing replicate_newnodes_docs <br/> [ns_server:error,2012-08-29T1:05:51.136,<a href='mailto:ns_1@10.3.121.20'>ns_1@10.3.121.20</a>:&lt;0.16154.0&gt;:ns_memcached:verify_report_long_call:274]call {stats,&lt;&lt;&gt;&gt;} took too long: 1276102 us <br/> [ns_server:error,2012-08-29T1:05:51.136,<a href='mailto:ns_1@10.3.121.20'>ns_1@10.3.121.20</a>:&#39;ns_memcached-saslbucket&#39;:ns_memcached:handle_info:594]handle_info(ensure_bucket,..) took too long: 809713 us <br/> [stats:warn,2012-08-29T1:05:51.387,<a href='mailto:ns_1@10.3.121.20'>ns_1@10.3.121.20</a>:&lt;0.16160.0&gt;:stats_collector:latest_tick:201]Dropped 1 ticks <br/> [ns_server:info,2012-08-29T1:05:52.779,<a href='mailto:ns_1@10.3.121.20'>ns_1@10.3.121.20</a>:ns_port_memcached:ns_port_server:log:169]memcached&lt;0.506.0&gt;: Wed Aug 29 08:05:52.492646 3: TAP (Producer) eq_tapq:replication_building_272_&#39;<a href='mailto:ns_1@10.3.121.24'>ns_1@10.3.121.24</a>&#39; - Connection is closed by force. <br/> <br/> <br/> Link to diags of all nodes <br/> <a href="https://s3.amazonaws.com/packages.couchbase/diag-logs/large_cluster_2_0/11nodes-1645-memchaced-close-connection-by-force-20120829.tgz">https://s3.amazonaws.com/packages.couchbase/diag-logs/large_cluster_2_0/11nodes-1645-memchaced-close-connection-by-force-20120829.tgz</a> <br/> centos 6.2 64bit MB-6481

Rebalance failed due to the timeout in bulk_set_vbucket_state operation with continuous view queries

Bug Major Closed Duplicate Public Thuan Nguyen Thuan Nguyen 2.0-beta-release-notes system-test Wed, 29 Aug 2012 16:55:24 -0500 Thu, 10 Jan 2013 01:21:37 -0600 Fri, 5 Oct 2012 15:45:05 -0500 2.0-beta 2.0 couchbase-bucket 0 0 4 minutes later after this rebalance failed, I did swap rebalance 3 nodes again. Rebalance failed after 13 minutes with error <br/> Rebalance exited with reason {ns_node_disco_events_exited, <br/> {&#39;EXIT&#39;,&lt;0.26298.432&gt;,normal}} <br/> ns_orchestrator002 <a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a> 01:23:41 - Wed Aug 29, 2012 <br/> <br/> Got exit from node disco events subscription ns_vbucket_mover000 <a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a> 01:23:39 - Wed Aug 29, 2012 <br/> <br/> I looked at diags on node 13, I saw the same error of this bug, memcached connection is closed by force <br/> <br/> ** At node 13: <br/> [ns_server:info,2012-08-29T1:23:40.678,<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>:&lt;0.20579.433&gt;:ns_replicas_builder_utils:kill_a_bunch_of_tap_names:59]Killed the following tap names on &#39;<a href='mailto:ns_1@10.3.121.20'>ns_1@10.3.121.20</a>&#39;: [&lt;&lt;&quot;replication_building_548_&#39;<a href='mailto:ns_1@10.3.121.24'>ns_1@10.3.121.24</a>&#39;&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;replication_building_548_&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39;&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;replication_building_548_&#39;<a href='mailto:ns_1@10.3.121.17'>ns_1@10.3.121.17</a>&#39;&quot;&gt;&gt;] <br/> [error_logger:error,2012-08-29T1:23:40.713,<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>:error_logger:ale_error_logger_handler:log_report:72] <br/> =========================CRASH REPORT========================= <br/> &nbsp;&nbsp;crasher: <br/> &nbsp;&nbsp;&nbsp;&nbsp;initial call: ns_single_vbucket_mover:mover/6 <br/> &nbsp;&nbsp;&nbsp;&nbsp;pid: &lt;0.20570.433&gt; <br/> &nbsp;&nbsp;&nbsp;&nbsp;registered_name: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;exception exit: {exited, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&#39;EXIT&#39;,&lt;0.26285.432&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{ns_node_disco_events_exited, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&#39;EXIT&#39;,&lt;0.26298.432&gt;,normal}}}} <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in function ns_single_vbucket_mover:mover_inner_old_style/6 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in call from misc:try_with_maybe_ignorant_after/2 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in call from ns_single_vbucket_mover:mover/6 <br/> &nbsp;&nbsp;&nbsp;&nbsp;ancestors: [&lt;0.26285.432&gt;,&lt;0.26141.432&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;messages: [{&#39;EXIT&#39;,&lt;0.26285.432&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{ns_node_disco_events_exited, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&#39;EXIT&#39;,&lt;0.26298.432&gt;,normal}}}] <br/> &nbsp;&nbsp;&nbsp;&nbsp;links: [&lt;0.20578.433&gt;,&lt;0.26285.432&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;dictionary: [{cleanup_list,[&lt;0.20579.433&gt;]}] <br/> &nbsp;&nbsp;&nbsp;&nbsp;trap_exit: true <br/> &nbsp;&nbsp;&nbsp;&nbsp;status: running <br/> &nbsp;&nbsp;&nbsp;&nbsp;heap_size: 1597 <br/> &nbsp;&nbsp;&nbsp;&nbsp;stack_size: 24 <br/> &nbsp;&nbsp;&nbsp;&nbsp;reductions: 3010 <br/> &nbsp;&nbsp;neighbours: <br/> <br/> [couchdb:info,2012-08-29T1:23:40.760,<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>:&lt;0.29969.390&gt;:couch_log:info:39]10.3.121.14 - - POST /_view_merge/?stale=ok&amp;limit=10 200 <br/> [ns_server:debug,2012-08-29T1:23:40.933,<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>:&lt;0.20580.433&gt;:ns_pubsub:do_subscribe_link:149]Deleting {ns_stats_event,&lt;0.15525.374&gt;} event handler: ok <br/> [ns_server:info,2012-08-29T1:23:38.869,<a href='mailto:ns_1@10.3.121.13'>ns_1@10.3.121.13</a>:&lt;0.20467.433&gt;:diag_handler:log_all_tap_and_checkpoint_stats:128]checkpoint:saslbucket: <br/> [{&lt;&lt;&quot;vb_1017:persisted_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;65&quot;&gt;&gt;}, <br/> <br/> ** At node 20: <br/> [couchdb:info,2012-08-29T1:23:40.724,<a href='mailto:ns_1@10.3.121.20'>ns_1@10.3.121.20</a>:&lt;0.13706.405&gt;:couch_log:info:39]Updater reading changes from passive partitions to update replica set view group `_design/d1` from set `default` <br/> [couchdb:info,2012-08-29T1:23:40.734,<a href='mailto:ns_1@10.3.121.20'>ns_1@10.3.121.20</a>:&lt;0.31308.390&gt;:couch_log:info:39]10.3.121.14 - - POST /_view_merge/?stale=ok&amp;limit=10 200 <br/> [ns_server:info,2012-08-29T1:23:40.737,<a href='mailto:ns_1@10.3.121.20'>ns_1@10.3.121.20</a>:&lt;0.16149.0&gt;:mc_connection:do_delete_vbucket:123]Deleting vbucket: saslbucket/547 <br/> [couchdb:info,2012-08-29T1:23:40.759,<a href='mailto:ns_1@10.3.121.20'>ns_1@10.3.121.20</a>:couch_server:couch_log:info:39]Deleting database saslbucket/547 <br/> [couchdb:info,2012-08-29T1:23:40.808,<a href='mailto:ns_1@10.3.121.20'>ns_1@10.3.121.20</a>:couch_server:couch_log:info:39]Deleting file /data/saslbucket/547.couch.313 <br/> [ns_server:info,2012-08-29T1:23:40.883,<a href='mailto:ns_1@10.3.121.20'>ns_1@10.3.121.20</a>:ns_port_memcached:ns_port_server:log:169]memcached&lt;0.506.0&gt;: Wed Aug 29 08:23:40.677138 3: TAP (Producer) eq_tapq:replication_building_548_&#39;<a href='mailto:ns_1@10.3.121.24'>ns_1@10.3.121.24</a>&#39; - Connection is closed by force. <br/> <br/> We looked at the memcached logs. <br/> @Tony is going to attach the memcached logs from node 13. <br/> memcached log file named memcached-log-1663-20120905.tgz in attached location above. <br/> <br/> On the node 13. Rebalance failure around this time. <br/> Rebalance exited with reason {{bulk_set_vbucket_state_failed, <br/> [{&#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39;, <br/> {&#39;EXIT&#39;, <br/> {{normal, <br/> {gen_server,call, <br/> [&lt;15201.5655.244&gt;, <br/> {start_vbucket_filter_change, <br/> [171,197,224,225,226,227,228,475,476, <br/> 477,478,479,531,532,533,655,656,657]}, <br/> 30000]}}, <br/> {gen_server,call, <br/> [{&#39;janitor_agent-saslbucket&#39;, <br/> &#39;<a href='mailto:ns_1@10.3.121.16'>ns_1@10.3.121.16</a>&#39;}, <br/> {if_rebalance,&lt;0.6019.435&gt;, <br/> {update_vbucket_state,657,replica, <br/> undefined,&#39;<a href='mailto:ns_1@10.3.121.23'>ns_1@10.3.121.23</a>&#39;}}, <br/> infinity]}}}}]}, <br/> [{janitor_agent,bulk_set_vbucket_state,4}, <br/> {ns_vbucket_mover, <br/> update_replication_post_move,3}, <br/> {ns_vbucket_mover,handle_info,2}, <br/> {gen_server,handle_msg,5}, <br/> {proc_lib,init_p_do_apply,3}]} <br/> <br/> Node 13 memcached logs:- <br/> Wed Sep 5 16:30:35.608213 3: 101 Closing connection due to read error: Connection reset by peer <br/> <br/> From memcached.c code:- <br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if (res == -1) { <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;if (errno == EAGAIN || errno == EWOULDBLOCK) { <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;break; <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;} <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;settings.extensions.logger-&gt;log(EXTENSION_LOG_WARNING, c, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;%d Closing connection due to read error: %s&quot;, c-&gt;sfd, strerror(errno)); <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;return READ_ERROR; <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;} <br/> <br/> &quot;connection is closed by force&quot; is the normal log from ep-engine whenever ep-engine receives the connection close request from the ns-server after completing the replication building phase. In other words, it&#39;s not an error case. <br/> <br/> The rebalance originally failed due to the timeout from 10.3.121.20 in processing {update_vbucket_state,294,replica} on the sasl bucket. We were continuously querying views on both buckets during rebalance. <br/> <br/> It would be more useful to grab any paging and swap activities during rebalance, but I think we already had the bugs opened for the very similar timeout issues with continuous view updates and queries. <br/> Looked at the ns_server code. <br/> <br/> -define(SET_VBUCKET_STATE_TIMEOUT, infinity). <br/> <br/> The vbucket_state_timeout is set to infinity. Not sure, what we can update more to? Alk, the timeout was set to inifinity and we still received a timeout. Who should next look at this? <a href="http://www.couchbase.com/issues/browse/MB-6595">http://www.couchbase.com/issues/browse/MB-6595</a> Date of First Response Wed, 5 Sep 2012 18:41:05 -0500 Rank 4129 Time In Status