http://www.couchbase.com/issues This file is an XML representation of an issue en-us 5.2.4 845 26-12-2012 http://www.couchbase.com/issues/browse/MB-7168 Couchbase Server version=2.0.0-1947-rel <br/> <a href="http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/448/consoleFull">http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/448/consoleFull</a> <br/> /testrunner -i resources/jenkins/centos-64-5node-failover.ini get-logs=True,GROUP=BAT -t failovertests.FailoverTests.test_failover_firewall,replica=1,keys_count=100000,dgm_run=True,GROUP=BAT <br/> <br/> steps: <br/> 1.3 nodes in cluster:10.1.3.114, 10.1.3.118,10.1.3.116 <br/> 2.enabled firewall on ip:10.1.3.118 : /sbin/iptables -A INPUT -p tcp -i eth0 --dport 1000:60000 -j REJECT <br/> 3 <br/> [2012-11-12 11:28:37,102] - [failovertests:260] INFO - node 10.1.3.118:8091 is &#39;unhealthy&#39; as expected <br/> [2012-11-12 11:29:07,544] - [rest_client:849] INFO - fail_over successful <br/> [2012-11-12 11:29:07,545] - [failovertests:278] INFO - failed over node : <a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a> <br/> [2012-11-12 11:29:07,545] - [failovertests:292] INFO - 10 seconds sleep after failover before invoking rebalance... <br/> 4.[2012-11-12 11:29:17,545] - [rest_client:883] INFO - rebalance params : password=password&amp;ejectedNodes=ns_1%4010.1.3.118&amp;user=Administrator&amp;knownNodes=ns_1%4010.1.3.114%2Cns_1%4010.1.3.118%2Cns_1%4010.1.3.116 <br/> <br/> result: <br/> [2012-11-12 11:29:17,562] - [rest_client:890] INFO - rebalance operation started <br/> [2012-11-12 11:29:17,569] - [rest_client:986] INFO - rebalance percentage : 0 % <br/> [2012-11-12 11:29:19,573] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:21,577] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:23,584] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:25,589] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:27,593] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:29,599] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:31,603] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:33,607] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:35,612] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:37,616] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:39,621] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:41,626] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:43,630] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:45,635] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:47,640] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:49,644] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:51,648] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:53,652] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:55,657] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:57,662] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:59,667] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:30:01,671] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:30:03,676] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:30:05,682] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:30:07,698] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:30:09,703] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:30:11,708] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:30:13,712] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:30:15,716] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:30:17,721] - [rest_client:971] ERROR - {u&#39;status&#39;: u&#39;none&#39;, u&#39;errorMessage&#39;: u&#39;Rebalance failed. See logs for detailed reason. You can try rebalance again.&#39;} - rebalance failed <br/> <br/> [rebalance:info,2012-11-12T11:49:03.581,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:&lt;0.20240.1&gt;:ns_rebalancer:rebalance:258]Waiting for bucket &quot;default&quot; to be ready on [&#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39;] <br/> [ns_server:info,2012-11-12T11:49:09.386,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:&lt;0.769.0&gt;:ns_orchestrator:handle_info:282]Skipping janitor in state rebalancing: {rebalancing_state,&lt;0.20215.1&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{dict,3,16,16,8,80,48, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{[],[],[],[],[],[],[],[],[],[],[],[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[],[],[],[]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{{[],[],[],[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[],[],[],[],[],[],[]}}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39;,&#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39;], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39;]} <br/> [ns_server:info,2012-11-12T11:49:19.386,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:&lt;0.769.0&gt;:ns_orchestrator:handle_info:282]Skipping janitor in state rebalancing: {rebalancing_state,&lt;0.20215.1&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{dict,3,16,16,8,80,48, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{[],[],[],[],[],[],[],[],[],[],[],[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[],[],[],[]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{{[],[],[],[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[],[],[],[],[],[],[]}}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39;,&#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39;], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39;]} <br/> [ns_server:info,2012-11-12T11:49:23.120,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:&lt;0.20319.1&gt;:compaction_daemon:try_to_cleanup_indexes:439]Cleaning up indexes for bucket `default` <br/> [ns_server:info,2012-11-12T11:49:23.121,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:&lt;0.20319.1&gt;:compaction_daemon:spawn_bucket_compactor:404]Compacting bucket default with config: <br/> [{database_fragmentation_threshold,{30,undefined}}, <br/> &nbsp;{view_fragmentation_threshold,{30,undefined}}] <br/> [ns_server:info,2012-11-12T11:49:24.348,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:ns_config_rep&lt;0.358.0&gt;:ns_config_rep:do_pull:341]Pulling config from: &#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39; <br/> <br/> [ns_server:info,2012-11-12T11:49:29.386,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:&lt;0.769.0&gt;:ns_orchestrator:handle_info:282]Skipping janitor in state rebalancing: {rebalancing_state,&lt;0.20215.1&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{dict,3,16,16,8,80,48, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{[],[],[],[],[],[],[],[],[],[],[],[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[],[],[],[]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{{[],[],[],[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[],[],[],[],[],[],[]}}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39;,&#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39;], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39;]} <br/> [ns_server:warn,2012-11-12T11:49:29.546,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:capi_set_view_manager-default&lt;0.15508.1&gt;:capi_set_view_manager:handle_info:345]Remote server node {&#39;capi_ddoc_replication_srv-default&#39;,&#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39;} process down: noconnection <br/> [ns_server:warn,2012-11-12T11:49:29.546,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:xdc_rdoc_replication_srv&lt;0.470.0&gt;:xdc_rdoc_replication_srv:handle_info:128]Remote server node {xdc_rdoc_replication_srv,&#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39;} process down: noconnection <br/> [user:warn,2012-11-12T11:49:29.546,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:ns_node_disco&lt;0.351.0&gt;:ns_node_disco:handle_info:168]Node &#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39; saw that node &#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39; went down. <br/> [error_logger:error,2012-11-12T11:49:29.546,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:error_logger&lt;0.5.0&gt;:ale_error_logger_handler:log_msg:76]** Node &#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39; not responding ** <br/> ** Removing (timedout) connection ** <br/> <br/> [ns_server:info,2012-11-12T11:49:30.536,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:ns_config_rep&lt;0.358.0&gt;:ns_config_rep:do_pull:341]Pulling config from: &#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39; <br/> <br/> [ns_server:info,2012-11-12T11:49:39.386,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:&lt;0.769.0&gt;:ns_orchestrator:handle_info:282]Skipping janitor in state rebalancing: {rebalancing_state,&lt;0.20215.1&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{dict,3,16,16,8,80,48, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{[],[],[],[],[],[],[],[],[],[],[],[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[],[],[],[]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{{[],[],[],[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[],[],[],[],[],[],[]}}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39;,&#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39;], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39;]} <br/> [ns_server:info,2012-11-12T11:49:49.386,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:&lt;0.769.0&gt;:ns_orchestrator:handle_info:282]Skipping janitor in state rebalancing: {rebalancing_state,&lt;0.20215.1&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{dict,3,16,16,8,80,48, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{[],[],[],[],[],[],[],[],[],[],[],[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[],[],[],[]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{{[],[],[],[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[],[],[],[],[],[],[]}}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39;,&#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39;], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39;]} <br/> [ns_server:info,2012-11-12T11:49:53.135,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:&lt;0.20445.1&gt;:compaction_daemon:try_to_cleanup_indexes:439]Cleaning up indexes for bucket `default` <br/> [ns_server:info,2012-11-12T11:49:53.136,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:&lt;0.20445.1&gt;:compaction_daemon:spawn_bucket_compactor:404]Compacting bucket default with config: <br/> [{database_fragmentation_threshold,{30,undefined}}, <br/> &nbsp;{view_fragmentation_threshold,{30,undefined}}] <br/> [ns_server:info,2012-11-12T11:49:59.386,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:&lt;0.769.0&gt;:ns_orchestrator:handle_info:282]Skipping janitor in state rebalancing: {rebalancing_state,&lt;0.20215.1&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{dict,3,16,16,8,80,48, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{[],[],[],[],[],[],[],[],[],[],[],[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[],[],[],[]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{{[],[],[],[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[],[],[],[],[],[],[]}}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39;,&#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39;], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39;]} <br/> [user:info,2012-11-12T11:50:03.582,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:&lt;0.769.0&gt;:ns_orchestrator:handle_info:319]Rebalance exited with reason {not_all_nodes_are_ready_yet, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39;,&#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39;]} <br/> <br/> <br/> <br/> <br/> MB-7168

[Doc'd] failover of node that's completely down is still not quick (was: Rebalance exited with reason {not_all_nodes_are_ready_yet after failover node)

Bug Blocker Reopened Unresolved Public Aleksey Kondratenko Andrei Baranouski 2.0.1-release-notes 2.0.2-release-notes Tue, 13 Nov 2012 05:16:46 -0600 Mon, 20 May 2013 12:46:14 -0500 2.0 2.0.1 2.0.2 2.1 0 9 almost all test fail due to rebalance failure after failover nodes on version=2.0.0-1949-rel <br/> <a href="http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/449/consoleFull">http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/449/consoleFull</a> if this is an automated test please rerun and confrm the behavior twice per bug-scrub2 3 tests are constantly falling with the same reason <a href="http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/457/consoleFull">http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/457/consoleFull</a> ( 1954 build) can be easily reproduced for windows - more info is in duplicate bug <a href="http://www.couchbase.com/issues/browse/MB-7205" title="Rebalance exited with reason {not_all_nodes_are_ready_yet,[&#39;ns_1@10.1.3.147&#39;]}"><strike>MB-7205</strike></a> (reproduced manually) so this failure occurs when the other node is unavailable ? <br/> <br/> Iryna , when you reproduce this on windows manually does the rebalance succeed second time ? <br/> do you see any data loss ? does rebalance fail immediately ? it not succeed after retrying during some time window, after about 15 minutes it succeed <br/> there is no data loss, <br/> rebalance fails in first 1-5 mins of rebalancing I cannot see any clear evidence of what caused this. But it looks bad enough. 60 seconds timeout to query vbucket states on both nodes was hit here. <br/> <br/> I also see that janitor pass immediately preceding rebalance also failed but in different phase, where it was waiting for vbucket change requests to be complete also failed on both nodes. And this timeout is 30 seconds. <br/> <br/> <br/> I&#39;d like diag to be grabbed _immediately_ after rebalance fails so that I could see what is the state of janitor_ageng and ns_memcached on each node. I.e. without doing any cleanup please. May I have that ? See above logs immediately after rebalance failure rebalance is successful in second time It&#39;s not quite immediately, sadly. Few seconds after problem. And backtraces on .118 (node where we failed waiting) don&#39;t have anything bad sadly. given that rebalance succeeded second time moving this to 2.0.1 Andrei,<br/> <br/> is this test fialing with the latest 2.0.1 build ? yes, 3 tests constantly fall in failover job: <a href="http://qa.hq.northscale.net/view/2.0.1/job/centos-64-2.0-failover-tests/501/consoleFull">http://qa.hq.northscale.net/view/2.0.1/job/centos-64-2.0-failover-tests/501/consoleFull</a> 2.0.1-120-rel per bug scrub <br/> <br/> retrying rebalance after a few minutes should work. Aliaksey, <br/> <br/> i know this was discussed before but we want to confirm what the test does with your conclusion <br/> <br/> we put the node behind the firewall , then wait until node is marked as unhealthy by NS_SERVER <br/> then failover this node and click on rebalance to eject the node ( ejectedNodes = 10.1.3.118 ) <br/> <br/> we want to know why when the node was already failed over from the cluster we wait for it be ready as part of rebalance. ( or the timeouts happened on existing nodes ? ) <br/> <br/> This is quite subtle to explain fully, but main problem is we have to wait until failover actually completes internally and that is subject to timeouts in certain areas.<br/> <br/> So if you keep trying for 2-3 minutes it should eventually work. &gt;&gt;but main problem is we have to wait until failover actually completes internally and that is subject to timeouts in certain areas. <br/> so when failover REST api returns it does not mean that failover process internally is completed. <br/> is failover a synchronous call or asynchronous ? in case its asyncronous is there a way for a test to check before initiating a rebalance so that we have a test that is more deterministic It is best-effort sync. If it fails to be sync with reasonably short timeout, it&#39;ll silently become async.<br/> <br/> There&#39;s no way to detect that right now. per bug scrub - assigning to Jin A few things came out of the engineering talk regarding this issue: <br/> 1) This is a good catch in that it is confirming we need a better way of handling the api since it cannot %100 warranty the completion of failover. <br/> 2) However, it is not a critical or blocker since the symptom is more obvious (highly probable) while running an automated testing case. <br/> 3) Only feasible approach (per ns server team) for now is to wait and retry. <br/> <br/> Based on this and the fact the fix would require changes across components (ep engine, etc) we may want to consider to put this into a future enhancement. <br/> Assign this to Yaseen for his input here. Pelase assign it back to Jin or Dipti afterwards. Thanks. Please see the above comment. Thanks Jin for confirming this. I think this expected behavior can be included in the release notes as well so that users and support team can be aware of the issue and the suggested workaround. <br/> <br/> Andrei, <br/> can you then modify the test accordingly. <br/> <br/> per bug scrub : <br/> <br/> please revise the test accordingly and after running the test for few times can you propose how long customer should wait before kicking the rebalance again Please assign it back to Jin after having enough information for recommendation. Will follow up with the doc team. <br/> with timeout 90 sec after failover befor rebalance - tests passed <br/> will launch with 60 sec can you try with lower timeout ( 10 seconds for instance ) ? 10 secs was in initial state for ticket. <br/> <br/> [2012-11-12 11:29:07,545] - [failovertests:292] INFO - 10 seconds sleep after failover before invoking rebalance... <br/> <br/> 60 secs tests passed <br/> <br/> will try with 30 sec any update on the test w/ 30 sec? 30 sec, build 151, tests passed <a href="http://qa.hq.northscale.net/view/2.0.1/job/centos-64-2.0-failover-tests/536/">http://qa.hq.northscale.net/view/2.0.1/job/centos-64-2.0-failover-tests/536/</a> Karen,<br/> <br/> could you please add this to the release notes that the user needs to wait for 30 seconds before they attempt to run rebalance operation after failing over a node.<br/> comment son this bug should explain under what conditions this 30 seconds delay is neeeded Jin, <br/> <br/> <br/> Can you summarize the situation when someone should wait 30 seconds to reattempt rebalance? <br/> <br/> <br/> Thanks! Failover REST api is sync operation with timeout. When it fails to complete the failover process within the timeout period, it internally switches to async operation (continues the failover to completion) and immediately returns. Subsequent rebalance in this case would fail because the failover process is still running. User can wait between 30 seconds upto a minute and reattempt rebalance. The failover REST api timeout is 30 second. &lt;para&gt; <br/> A cluster rebalance may exit and produce the error <br/> {not_all_nodes_are_ready_yet} if you perform the rebalance right <br/> after failing over a node in the cluster. You may need to <br/> wait 30 seconds after the node failover before you <br/> attempt the cluster rebalance. <br/> &lt;/para&gt; <br/> &lt;para&gt;This is because the failover REST API is a synchronous operation with a timeout. If it fails to <br/> complete the failover process by the timeout, the operation internally switches into a <br/> asynchronous operation. It will immediately return and re-attempt failover in the background which will cause <br/> rebalance to fail since the failover operation is still running.&lt;/para&gt; Added to RN as : <br/> <br/> &lt;para&gt; <br/> A cluster rebalance may exit and produce the error <br/> {not_all_nodes_are_ready_yet} if you perform the rebalance right <br/> after failing over a node in the cluster. You may need to <br/> wait 30 seconds after the node failover before you <br/> attempt the cluster rebalance. <br/> &lt;/para&gt; <br/> &lt;para&gt;This is because the failover REST API is a synchronous operation with a timeout. If it fails to <br/> complete the failover process by the timeout, the operation internally switches into a <br/> asynchronous operation. It will immediately return and re-attempt failover in the background which will cause <br/> rebalance to fail since the failover operation is still running.&lt;/para&gt; Jin, I got the same error with 30 seconds waiting after failover on build 2.0.2-761-rel<br/> Should we update RN for 2.0.1/next 2.0.2 release?<br/> <a href="http://qa.hq.northscale.net/view/2.0.1/job/centos-64-2.0-failover-tests/583/consoleFull">http://qa.hq.northscale.net/view/2.0.1/job/centos-64-2.0-failover-tests/583/consoleFull</a><br/> <br/> [2013-04-08 19:41:13,981] - [failovertests:148] INFO - failed over node : <a href='mailto:ns_1@10.1.3.115'>ns_1@10.1.3.115</a><br/> [2013-04-08 19:41:13,981] - [failovertests:163] INFO - 30 seconds sleep after failover before invoking rebalance...<br/> [2013-04-08 19:41:43,981] - [rest_client:834] INFO - rebalance params : password=password&amp;ejectedNodes=ns_1%4010.1.3.115&amp;user=Administrator&amp;knownNodes=ns_1%4010.1.3.114%2Cns_1%4010.1.3.115%2Cns_1%4010.1.3.118%2Cns_1%4010.1.3.116<br/> [2013-04-08 19:41:43,991] - [rest_client:838] INFO - rebalance operation started<br/> [2013-04-08 19:41:44,004] - [rest_client:940] INFO - rebalance percentage : 0 %<br/> [2013-04-08 19:41:46,009] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:41:48,014] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:41:50,018] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:41:52,023] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:41:54,029] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:41:56,033] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:41:58,039] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:00,043] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:02,048] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:04,056] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:06,061] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:08,066] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:10,071] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:12,077] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:14,084] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:16,089] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:18,096] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:20,102] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:22,108] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:24,113] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:26,119] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:28,125] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:30,130] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:32,137] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:34,143] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:36,152] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:38,157] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:40,163] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:42,168] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:44,174] - [rest_client:923] ERROR - {u&#39;status&#39;: u&#39;none&#39;, u&#39;errorMessage&#39;: u&#39;Rebalance failed. See logs for detailed reason. You can try rebalance again.&#39;} - rebalance failed<br/> [2013-04-08 19:42:44,175] - [rest_client:924] INFO - Latest logs from UI:<br/> [2013-04-08 19:42:44,273] - [rest_client:925] ERROR - {u&#39;node&#39;: u&#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39;, u&#39;code&#39;: 2, u&#39;text&#39;: u&quot;Rebalance exited with reason {not_all_nodes_are_ready_yet,\n it&#39;s happened twice in this run Thanks for the update, Andrei. Before we update the RN first let&#39;s figure out how long a user should wait before retrying the rebalance. Can we upgrade the wait period to 1 minute and see if that is long enough? Thanks. set timeout in 1 min <a href="http://review.couchbase.org/#/c/25587/">http://review.couchbase.org/#/c/25587/</a><br/> before that we had a timeout for 60 sec, the tests do not fall<br/> <a href="http://www.couchbase.com/issues/browse/MB-7168?focusedCommentId=49128&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-49128">http://www.couchbase.com/issues/browse/MB-7168?focusedCommentId=49128&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-49128</a> Integrated in ui-testing #42 (See [<a href="http://qa.hq.northscale.net/job/ui-testing/42/">http://qa.hq.northscale.net/job/ui-testing/42/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS tested with timeout for 60 sec, all tests passed Jin, I think to add in RN waiting for 60 seconds after failover should be okay Karen, please update the release notes to suggest 60 sec (not 30). Also lowering the priority from Blocker to Critical as it&#39;s not a 2.0.2 release blocking issue. Ok, changed to : A cluster rebalance may exit and produce the error <br/> {not_all_nodes_are_ready_yet} if you perform the rebalance right <br/> after failing over a node in the cluster. You may need to <br/> wait 60 seconds after the node failover before you <br/> attempt the cluster rebalance.<br/> <br/> in RN 2.0.2 added to RN 2.0.2 Integrated in win-ui-testing-P0 #56 (See [<a href="http://qa.hq.northscale.net/job/win-ui-testing-P0/56/">http://qa.hq.northscale.net/job/win-ui-testing-P0/56/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.couchbase.com/issues/browse/MB-7168" title="[Doc&#39;d] failover of node that&#39;s completely down is still not quick (was: Rebalance exited with reason {not_all_nodes_are_ready_yet after failover node)">MB-7168</a>: sleep 30 sec after reb falied when killed memcached (Revision 44f755b962b7987d5b972caf9a283baa95edaed1)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> andrei : <br/> Files : <br/> * pytests/swaprebalance.py<br/> Verified that the doc change has also gone to 2.0.1 RN. <br/> <br/> <a href="http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-server-rn_2-0-0l.html">http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-server-rn_2-0-0l.html</a> <br/> <br/> Assigning to PM for the next step. discussed with ALK, this bug will be fixed in 2.1. talked about having UI alert but since this happens only when node dies completely for now Release Note as Known Issue should fine. thanks Updated ticket to reflect badness of this and must-have-ness for 2.1 added flag to include 2.0.1 release note to 2.0.2. Integrated in windows32_sanity_P0 #29 (See [<a href="http://qa.hq.northscale.net/job/windows32_sanity_P0/29/">http://qa.hq.northscale.net/job/windows32_sanity_P0/29/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = UNSTABLE Integrated in windows32_view_P0 #6 (See [<a href="http://qa.hq.northscale.net/job/windows32_view_P0/6/">http://qa.hq.northscale.net/job/windows32_view_P0/6/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = UNSTABLE Date of First Response Mon, 19 Nov 2012 13:26:06 -0600 Flagged Rank 506