http://www.couchbase.com/issues This file is an XML representation of an issue en-us 5.2.4 845 26-12-2012 http://www.couchbase.com/issues/browse/MB-5279 Couchbase Server Have not seen this issue previously ever on 181. Seems to be something new. <br/> <br/> Test case that failed <br/> swaprebalance.SwapRebalanceFailedTests.test_add_back_failed_node,replica=3,num-buckets=5,num-swap=2,swap-orchestrator=True,keys-count=800000 <br/> <br/> 1) Failover 2 nodes, including the orchestrator <br/> 2) Rebalance them out <br/> 3) Add them back and rebalance in, without cleaning them. <br/> <br/> [rebalance:error] [2012-05-10 8:48:59] [<a href='mailto:ns_1@10.1.3.74'>ns_1@10.1.3.74</a>:&lt;0.27930.16&gt;:ns_rebalancer:do_wait_buckets_shutdown:170] Failed to wait deletion of some buckets on some nodes: [{&#39;<a href='mailto:ns_1@10.1.3.77'>ns_1@10.1.3.77</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&#39;EXIT&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{old_buckets_shutdown_wait_failed, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&quot;bucket-2&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;bucket-1&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;bucket-0&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;bucket-4&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;bucket-3&quot;]}}}] <br/> <br/> [user:info] [2012-05-10 8:48:59] [<a href='mailto:ns_1@10.1.3.74'>ns_1@10.1.3.74</a>:&lt;0.27893.16&gt;:ns_orchestrator:handle_info:246] Rebalance exited with reason {buckets_shutdown_wait_failed, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{&#39;<a href='mailto:ns_1@10.1.3.77'>ns_1@10.1.3.77</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&#39;EXIT&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{old_buckets_shutdown_wait_failed, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&quot;bucket-2&quot;,&quot;bucket-1&quot;,&quot;bucket-0&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;bucket-4&quot;,&quot;bucket-3&quot;]}}}]} CentOS 64 bit <br/> <br/> 1.8.1-807 MB-5279

user is unable to add the node back to the cluster because the failed over node is stuck in warmup phase

Bug Blocker Closed Fixed Public Aleksey Kondratenko Karan Kumar customer Thu, 10 May 2012 15:20:21 -0500 Wed, 9 Jan 2013 22:59:21 -0600 Tue, 29 May 2012 17:21:11 -0500 1.8.1-release-candidate 1.8.1 ns_server 0 2 Found that indeed shutting down bucket-2 took minutes even though we (according to logs) passed force flag: <br/> <br/> <br/> [user:info] [2012-05-10 8:45:32] [<a href='mailto:ns_1@10.1.3.77'>ns_1@10.1.3.77</a>:&#39;ns_memcached-bucket-0&#39;:ns_memcached:terminate:296] Shutting down bucket &quot;bucket-0&quot; on &#39;<a href='mailto:ns_1@10.1.3.77'>ns_1@10.1.3.77</a>&#39; for deletion <br/> ... (no other messages from this erlang process in between) ... <br/> [ns_server:debug] [2012-05-10 8:49:38] [<a href='mailto:ns_1@10.1.3.77'>ns_1@10.1.3.77</a>:&#39;ns_memcached-bucket-0&#39;:ns_storage_conf:delete_bucket_db_directory:384] Result of deleting db directory: &quot;/opt/couchbase/var/lib/couchbase/data/bucket-0-data&quot;: ok <br/> <br/> That&#39;s what rebalance failure actually mentioned. <br/> <br/> I&#39;ve seen some memcached crash on this .77 node shortly before we&#39;ve started bucket deletion. So my guess is that warmup prevents force shutdown to be quick. I.e.: <br/> <br/> <br/> [error_logger:info] [2012-05-10 8:45:04] [<a href='mailto:ns_1@10.1.3.77'>ns_1@10.1.3.77</a>:error_logger:ale_error_logger_handler:log_report:72] <br/> =========================PROGRESS REPORT========================= <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;supervisor: {local,ns_port_sup} <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;started: [{pid,&lt;0.29206.11&gt;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{name, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{memcached,&quot;/opt/couchbase/bin/memcached&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&quot;-X&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;/opt/couchbase/lib/memcached/stdin_term_handler.so&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;-l&quot;,&quot;0.0.0.0:11210,0.0.0.0:11209:1000&quot;,&quot;-E&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;/opt/couchbase/lib/memcached/bucket_engine.so&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;-B&quot;,&quot;binary&quot;,&quot;-r&quot;,&quot;-c&quot;,&quot;10000&quot;,&quot;-e&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;admin=_admin;default_bucket_name=default;auto_create=false&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{env, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{&quot;EVENT_NOSELECT&quot;,&quot;1&quot;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&quot;MEMCACHED_TOP_KEYS&quot;,&quot;100&quot;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&quot;ISASL_PWFILE&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;/opt/couchbase/var/lib/couchbase/data/isasl.pw&quot;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&quot;ISASL_DB_CHECK_TIME&quot;,&quot;1&quot;}]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;use_stdio,stderr_to_stdout,exit_status, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;port_server_send_eol,stream]}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{mfargs, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{supervisor_cushion,start_link, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[memcached,5000,ns_port_server,start_link, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[memcached,&quot;/opt/couchbase/bin/memcached&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&quot;-X&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;/opt/couchbase/lib/memcached/stdin_term_handler.so&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;-l&quot;,&quot;0.0.0.0:11210,0.0.0.0:11209:1000&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;-E&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;/opt/couchbase/lib/memcached/bucket_engine.so&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;-B&quot;,&quot;binary&quot;,&quot;-r&quot;,&quot;-c&quot;,&quot;10000&quot;,&quot;-e&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;admin=_admin;default_bucket_name=default;auto_create=false&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{env, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{&quot;EVENT_NOSELECT&quot;,&quot;1&quot;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&quot;MEMCACHED_TOP_KEYS&quot;,&quot;100&quot;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&quot;ISASL_PWFILE&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;/opt/couchbase/var/lib/couchbase/data/isasl.pw&quot;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&quot;ISASL_DB_CHECK_TIME&quot;,&quot;1&quot;}]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;use_stdio,stderr_to_stdout,exit_status, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;port_server_send_eol,stream]]]}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{restart_type,permanent}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{shutdown,10000}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{child_type,worker}] <br/> <br/> and prior to that: <br/> <br/> <br/> [error_logger:error] [2012-05-10 8:45:03] [<a href='mailto:ns_1@10.1.3.77'>ns_1@10.1.3.77</a>:error_logger:ale_error_logger_handler:log_report:72] <br/> =========================CRASH REPORT========================= <br/> &nbsp;&nbsp;crasher: <br/> &nbsp;&nbsp;&nbsp;&nbsp;initial call: ns_port_server:init/1 <br/> &nbsp;&nbsp;&nbsp;&nbsp;pid: &lt;0.15086.9&gt; <br/> &nbsp;&nbsp;&nbsp;&nbsp;registered_name: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;exception exit: {abnormal,134} <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in function gen_server:terminate/6 <br/> &nbsp;&nbsp;&nbsp;&nbsp;ancestors: [&lt;0.15085.9&gt;,ns_port_sup,ns_server_sup,ns_server_cluster_sup, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;0.60.0&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;messages: [{&#39;EXIT&#39;,#Port&lt;0.174933&gt;,normal}] <br/> &nbsp;&nbsp;&nbsp;&nbsp;links: [&lt;0.15085.9&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;dictionary: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;trap_exit: true <br/> &nbsp;&nbsp;&nbsp;&nbsp;status: running <br/> &nbsp;&nbsp;&nbsp;&nbsp;heap_size: 121393 <br/> &nbsp;&nbsp;&nbsp;&nbsp;stack_size: 24 <br/> &nbsp;&nbsp;&nbsp;&nbsp;reductions: 1661845 <br/> &nbsp;&nbsp;neighbours: <br/> <br/> <br/> and even more prior to that: <br/> <br/> <br/> [ns_server:info] [2012-05-10 8:44:34] [<a href='mailto:ns_1@10.1.3.77'>ns_1@10.1.3.77</a>:&lt;0.15086.9&gt;:ns_port_server:log:161] memcached&lt;0.15086.9&gt;: [warn] Epoll ADD(5) on fd 247 failed. Old events were 0; read change was 1 (add); write change was 1 (add): Bad file descriptor <br/> memcached&lt;0.15086.9&gt;: Failed to add connection to libevent: Bad file descriptor <br/> memcached&lt;0.15086.9&gt;: memcached: bucket_engine.c:997: handle_disconnect: Assertion `es&#39; failed. <br/> Given all that and especially that most likely this is possibly not a regression (i.e. warmup likely caused bucket deletion to be delayed on 1.8.0 as well) assigning to Frank so that we consider if this is really blocker. From Farshid: doesn&#39;t seem like a regression and is a common use case that we should fix in 1.8.1. we need a fix for this. Perhaps the behavior here needs to be changed? don&#39;t allow &quot;rebalance&quot; to start if a node is warming up ? Is there something else going on under the covers? Something else.<br/> <br/> We failed over node.<br/> <br/> This means we requested deletion of bucket on this node. This deletion of bucket may take some time and we normally don&#39;t care. But we don&#39;t expect it to take ages.<br/> <br/> What happens next is node is attempted to be added back while this deletion is still in progress and we refuse because we&#39;re going to re-create this bucket. But we need to complete deletion of it first.<br/> <br/> This is something we long wanted from ep-engine. We want bucket deletion to be reasonably quick. Always.<br/> <br/> If you still think we need this logic &quot;don&#39;t start rebalance unless you can be reasonably sure it will succeed&quot; we can work on it. It doesn&#39;t look like very trivial task though.<br/> Correction &quot;dont allow user to start rebalance unless you can reasonably sure it will succeed&quot;. Let me note that current behavior of performing all checks at the start of rebalance (after user has requested rebalance) is not very different. The difference is some potential confusion on behalf of user if rebalance fails immediately. Is it easy to make it more obvious then as to why rebalance failed? Isn&#39;t the following good enough?: <br/> <br/> Rebalance exited with reason {buckets_shutdown_wait_failed, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{&#39;<a href='mailto:ns_1@10.1.3.77'>ns_1@10.1.3.77</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&#39;EXIT&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{old_buckets_shutdown_wait_failed, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&quot;bucket-2&quot;,&quot;bucket-1&quot;,&quot;bucket-0&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;bucket-4&quot;,&quot;bucket-3&quot;]}}}]} Thanks Aliaksey. I understand this issue better now. Let&#39;s see what can be done in ep_engine. ns-server should send bucket-deletion request to the memcached which is waming up and ep-engine should abort warmup process and continue with bucket deletion Chiyoung, can you estimate the level of change needed to support this? How risky are these changes? BTW, do we have reliable evidence that bucket deletion was delayed because of warmup in this particular case? I.e. I know warmup is _likely_ in progress but that&#39;s all I know about this bucket deletion. yes we have confirmed that warmup process was going on and bucket deletion causes the whole thing to hang. <br/> <br/> if it helps we can get gdb backtrace when this is happening but given that this was seen many times by QE/customers i think we can safely assume thats the case. Can the ns_server move the database files or delete them directly before the failedover node is added back? we do delete files. But after bucket shutdown is complete we have patch for 1.8 that will make memcached exit immediately without shutting down buckets when failover happens. That could be easy treatment for this problem I&#39;m a little confused now. Why the failed over node is warmed up again? <br/> <br/> Anyway, if the ns-server already has the patch for this issue, can I close this bug? memcached crashed before node was failed over. Then ns_server restarted memcached and recreated buckets. And they started warming up. <br/> <br/> Then node was failed over. ns_server deletes all buckets (with force flag set to true) but that takes long time seemingly because of warm up. And that prevents rebalance back in. Because before starting rebalance we now wait for pending bucket deletions to complete. <br/> <br/> The patch I mentioned is against memcached. The idea is when we detect failover we ask memcached to just exit without any cleanup. <br/> <br/> I don&#39;t know why but Farshid and Dipti wanted ep-engine to be fixed. Given 2.0 and 1.8.1 warmup is very different I question whether we really need that btw. &gt;&gt;The idea is when we detect failover we ask memcached to just exit without any cleanup. <br/> <br/> is this like kill -9 ? yes. But works on windows too ok so if this is a force kill then we dont have to fix this on ep-engine. <br/> <br/> for this specific issue all we care about it the end user experience and if user will be able to add the node back to the cluster without any issues and your patch takes care of it so we can close this bug Patches backported and uploaded to gerrit. I can rubberstamp ns_server&#39;s patches. But not memcached. <a href="http://review.couchbase.org/16432">http://review.couchbase.org/16432</a> <br/> <a href="http://review.couchbase.org/16436">http://review.couchbase.org/16436</a> <br/> <a href="http://review.couchbase.org/16437">http://review.couchbase.org/16437</a> <br/> <a href="http://review.couchbase.org/16438">http://review.couchbase.org/16438</a> <br/> Should be fixed now with memcached &amp; ns_server side merged Integrated in github-ns-server-2-0 #357 (See [<a href="http://qa.hq.northscale.net/job/github-ns-server-2-0/357/">http://qa.hq.northscale.net/job/github-ns-server-2-0/357/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.couchbase.com/issues/browse/MB-5279" title="user is unable to add the node back to the cluster because the failed over node is stuck in warmup phase"><strike>MB-5279</strike></a>:bp: allow port_servers to send arbitrary data to port (Revision ab2b8ecf77658fda7266a9eaed1b474f29741e43)<br/> <a href="http://www.couchbase.com/issues/browse/MB-5279" title="user is unable to add the node back to the cluster because the failed over node is stuck in warmup phase"><strike>MB-5279</strike></a>:bp:made supervisor_cushion exit with same reason as child (Revision 5ba3d6e1634c613ff3b6c1678f3eb04449434834)<br/> <a href="http://www.couchbase.com/issues/browse/MB-5279" title="user is unable to add the node back to the cluster because the failed over node is stuck in warmup phase"><strike>MB-5279</strike></a>:bp: ask memcached to die real quick on failover (Revision 2f14d3a4107b6c7fccc4c479832048bb259f469c)<br/> <a href="http://www.couchbase.com/issues/browse/MB-5279" title="user is unable to add the node back to the cluster because the failed over node is stuck in warmup phase"><strike>MB-5279</strike></a>: send shutdown command instead of empty line to memcached (Revision 6118863758b88da374868d2adf7a890096067394)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> Aliaksey Kandratsenka : <br/> Files : <br/> * src/ns_port_server.erl<br/> <br/> Aliaksey Kandratsenka : <br/> Files : <br/> * src/supervisor_cushion.erl<br/> <br/> Aliaksey Kandratsenka : <br/> Files : <br/> * src/ns_port_sup.erl<br/> * src/ns_server_sup.erl<br/> <br/> Aliaksey Kandratsenka : <br/> Files : <br/> * src/ns_port_server.erl<br/> Date of First Response Thu, 10 May 2012 19:34:50 -0500 Rank 5154 Time In Status