http://www.couchbase.com/issues This file is an XML representation of an issue en-us 5.2.4 845 26-12-2012 http://www.couchbase.com/issues/browse/MB-5298 Couchbase Server Failing testcase <br/> failovertests.FailoverTests.test_failover_firewall,replica=3,load_ratio=10 <br/> <br/> [ns_server:info] [2012-05-14 13:26:46] [<a href='mailto:ns_1@10.1.3.55'>ns_1@10.1.3.55</a>:&lt;0.3846.2&gt;:ns_janitor:wait_for_memcached:286] Waiting for &quot;default&quot; on [&#39;<a href='mailto:ns_1@10.1.3.50'>ns_1@10.1.3.50</a>&#39;,&#39;<a href='mailto:ns_1@10.1.3.51'>ns_1@10.1.3.51</a>&#39;,&#39;<a href='mailto:ns_1@10.1.3.52'>ns_1@10.1.3.52</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.1.3.54'>ns_1@10.1.3.54</a>&#39;] <br/> [ns_server:debug] [2012-05-14 13:26:46] [<a href='mailto:ns_1@10.1.3.55'>ns_1@10.1.3.55</a>:ns_bucket_worker:ns_bucket_sup:update_childs:91] Stopping child for dead bucket: {{per_bucket_sup,&quot;default&quot;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;0.23773.0&gt;,supervisor, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[single_bucket_sup]} <br/> <br/> [ns_server:debug] [2012-05-14 13:26:46] [<a href='mailto:ns_1@10.1.3.55'>ns_1@10.1.3.55</a>:&lt;0.23773.0&gt;:single_bucket_sup:top_loop:28] Delegating exit {&#39;EXIT&#39;,&lt;0.23699.0&gt;,shutdown} to child supervisor: &lt;0.23774.0&gt; <br/> <br/> <br/> [error_logger:error] [2012-05-14 13:26:03] [<a href='mailto:ns_1@10.1.3.50'>ns_1@10.1.3.50</a>:error_logger:ale_error_logger_handler:log_report:72] <br/> =========================CRASH REPORT========================= <br/> &nbsp;&nbsp;crasher: <br/> &nbsp;&nbsp;&nbsp;&nbsp;initial call: ns_vbucket_mover:init/1 <br/> &nbsp;&nbsp;&nbsp;&nbsp;pid: &lt;0.29733.2&gt; <br/> &nbsp;&nbsp;&nbsp;&nbsp;registered_name: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;exception exit: {case_clause,{{ok,replica},{ok,replica}}} <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in function gen_server:terminate/6 <br/> &nbsp;&nbsp;&nbsp;&nbsp;ancestors: [&lt;0.28176.2&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;messages: [{&#39;EXIT&#39;,&lt;0.19829.3&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{exited, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&#39;EXIT&#39;,&lt;0.29733.2&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{case_clause,{{ok,replica},{ok,replica}}}}}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&#39;EXIT&#39;,&lt;0.19792.3&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{exited, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&#39;EXIT&#39;,&lt;0.29733.2&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{case_clause,{{ok,replica},{ok,replica}}}}}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&#39;EXIT&#39;,&lt;0.15598.3&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{exited, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&#39;EXIT&#39;,&lt;0.29733.2&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{case_clause,{{ok,replica},{ok,replica}}}}}}] 1.8.1-815-rel MB-5298

Rebalance failed with reason {case_clause,{{ok,replica},{ok,replica}}} when rebalancing out a node which was failed over due to netwrok connectivity issues but it re-appears while rebalancing

Bug Critical Closed Fixed Public Aleksey Kondratenko Karan Kumar 1.8.1-release-notes Mon, 14 May 2012 15:34:40 -0500 Wed, 9 Jan 2013 23:13:18 -0600 Mon, 17 Sep 2012 13:25:20 -0500 1.8.1-release-candidate 2.0 ns_server 0 2 I&#39;ve added more logging in affected area(s). Please retest with latest stuff Adding the latest logs from 181-832-rel Failing test:- <br/> failovertests.FailoverTests.test_failover_firewall,replica=3,load_ratio=10 Good find. Now we have evidence that one of previously firewalled and failed over nodes is being (seemingly gradually) un-firewalled in the middle of rebalance. When rest of cluster actually already ejected this node. But this node discovers it was ejected 1 minute after it&#39;s replicator was able to push-replicate to one of existing nodes. We&#39;re starting to really hit limits of our naive cluster orchestration approach. <br/> I think reasonably simple treatment (still partial and naive) is to never restart replication automatically until janitor restarts it. It has some potential data safety implications though. I.e. janitor being really conservative in some cases will not restart replications that previously were automagically restarted. So not sure. Pull-based replication which is part of branch-18 would help here as well. Sadly we were not allowed to have it on branch-181. Need PM decision here May I ask for testing if this is a regression? My understanding no it&#39;s not. should we run this against 1.8.0 ? If you can, please do. I&#39;m pretty sure you&#39;ll hit this in 1.8.0 because failover &amp; replication logic (hint: no quick failover was allowed for 1.8.1) is same. We are pretty much hitting this consistently on 181. <br/> <br/> Just stop doing this questionable practice of abusing firewall. what are our options here to fix on 181? Given the likely hood of hitting this is more, we should try to fix. My understanding is that probability of hitting this in practice approaches zero. We had this issue since 1.6.0 yet nobody seen reported this problem. this is not about abusing the firewall . its about node coming back up or re-appearing after its failed over. <br/> <br/> if this is purely due to firewall then its ok to defer this your specific way of using firewall makes this problem probable to observe. That&#39;s my understanding. firewall is our way of simulating a node disappearing and re-appearing. we can simulate that by also shutting down the network interface or pulling the network cable if it helps I have evidence that you&#39;re enabling firewall back in some very specific way. Particularly memcached traffic is re-enabled first. And then _minutes_ later you re-enable erlang traffic. will defer to later release. Cannot happen on replicator on destination. Date of First Response Mon, 14 May 2012 22:52:30 -0500 Rank 5376 Time In Status