http://www.couchbase.com/issues This file is an XML representation of an issue en-us 5.2.4 845 26-12-2012 http://www.couchbase.com/issues/browse/MB-4375 Couchbase Server when rebalance fails or is stopped by the user the vbucket state for those rebalance operations which were making progress are still pending. <br/> ns_server janitor runs every few seconds which will change the vbucket state from pending-&gt;dead <br/> <br/> now when the user restarts the rebalance sooner than 5 minutes ep-engine will try to reuse that tap stream and will not send TAP_VBUCKET_SET when restarting the takeover and since the vbucket state is dead now ep-engine will not start the vbucket transfer and this will result in rebalance getting stuck. MB-4375

rebalance failing with retry_not_ready_vbuckets error if ns_server janitors sets the vbucket state from pending to dead when rebalance fails or stops

Bug Major Closed Incomplete Public Farshid Ghods Farshid Ghods Mon, 24 Oct 2011 02:15:01 -0500 Wed, 9 Jan 2013 22:59:22 -0600 Tue, 24 Jul 2012 22:40:30 -0500 1.7 GA 1.7.1.1 1.8.0 1.7.2 ns_server 0 0 <a href="https://github.com/membase/ep-engine/commit/4af628082a4407c37b33dd16196c5a47a51332aa">https://github.com/membase/ep-engine/commit/4af628082a4407c37b33dd16196c5a47a51332aa</a> A user of 1.8.0 has run into this issue it seems. The specific message in the log is:<br/> <br/> CRASH REPORT &lt;0.2530.13&gt; 2012-03-01 09:40:29<br/> ===============================================================================<br/> Crashing process <br/> &nbsp;&nbsp;&nbsp;initial_call {ebucketmigrator_srv,init,[&#39;Argument__1&#39;]}<br/> &nbsp;&nbsp;&nbsp;pid &lt;0.2530.13&gt;<br/> &nbsp;&nbsp;&nbsp;registered_name []<br/> &nbsp;&nbsp;&nbsp;error_info<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{exit,retry_not_ready_vbuckets,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{ebucketmigrator_srv,init,1},{proc_lib,init_p_do_apply,3}]}<br/> &nbsp;&nbsp;&nbsp;ancestors [&#39;ns_vbm_sup-default&#39;,&#39;single_bucket_sup-default&#39;,&lt;0.1025.0&gt;]<br/> &nbsp;&nbsp;&nbsp;messages []<br/> &nbsp;&nbsp;&nbsp;links [&lt;0.1063.0&gt;]<br/> &nbsp;&nbsp;&nbsp;dictionary []<br/> &nbsp;&nbsp;&nbsp;trap_exit false<br/> &nbsp;&nbsp;&nbsp;status running<br/> &nbsp;&nbsp;&nbsp;heap_size 4181<br/> &nbsp;&nbsp;&nbsp;stack_size 24<br/> &nbsp;&nbsp;&nbsp;reductions 218220 Hi Matt, <br/> <br/> do you have access to the user to grab diags from their cluster ? Folks, retry_not_ready_vbucket is actually a &quot;voluntarily crash&quot;. We do that in order to restart replication later. I.e. when replicating from some node if some of vbuckets we need to replicate from are not ready yet (i.e. we&#39;re second replica and 1st is not yet ready to be replicated from) we just don&#39;t replicate those vbuckets, but after 30 seconds we perform harakiri so that supervisor restarts us and we check again. This was quick fix in few days before 1.7.0 and I&#39;m really sorry for not making log messages clearer that it&#39;s not a problem at all. 1.8.1 will fix that. <br/> <br/> So this message has nothing at all to do with rebalance failing. May we ask logs from master node ? Master node can be identified by looking at user visible logs. Server that logs &quot;rebalance failed&quot; message is the master node for that failed rebalance. Date of First Response Mon, 5 Mar 2012 12:11:02 -0600 Rank 6069 Time In Status