http://www.couchbase.com/issues This file is an XML representation of an issue en-us 5.2.4 845 26-12-2012 http://www.couchbase.com/issues/browse/MB-5108 Couchbase Server Failing test <br/> upgradetests.MultipleNodeUpgradeTests.test_upgrade,initial_version=1.7.2,create_buckets=True,insert_data=True,start_upgraded_first=False,load_ratio=10,online_upgrade=True <br/> <br/> <br/> 2012-04-18 05:05:23,369 - root - INFO - adding node : 10.3.121.92:8091 to the cluster <br/> 2012-04-18 05:05:23,370 - root - INFO - adding remote node : 10.3.121.92 to this cluster @ : 10.3.121.98 <br/> 2012-04-18 05:05:24,121 - root - INFO - added node : <a href='mailto:ns_1@10.3.121.92'>ns_1@10.3.121.92</a> to the cluster <br/> 2012-04-18 05:05:24,134 - root - INFO - rebalance params : password=password&amp;ejectedNodes=&amp;user=Administrator&amp;knownNodes=ns_1%4010.3.121.94%2Cns_1%4010.3.121.92%2Cns_1%4010.3.121.98%2Cns_1%4010.3.121.93%2Cns_1%4010.3.121.97%2Cns_1%4010.3.121.95 <br/> 2012-04-18 05:05:24,140 - root - ERROR - <a href="http://10.3.121.98:8091/controller/rebalance">http://10.3.121.98:8091/controller/rebalance</a> error 500 reason: unknown [&quot;Unexpected server error, request logged.&quot;] <br/> 2012-04-18 05:05:24,140 - root - ERROR - rebalance operation failed <br/> <br/> <br/> <br/> <br/> INFO REPORT &lt;0.6440.0&gt; 2012-04-18 05:06:53 <br/> =============================================================================== <br/> <br/> ns_log: logging menelaus_web:19:Server error during processing: [&quot;web request failed&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{path,&quot;/controller/rebalance&quot;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{type,exit}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{what, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{noproc, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_fsm,sync_send_event, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{global,ns_orchestrator}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{start_rebalance, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.3.121.94'>ns_1@10.3.121.94</a>&#39;,&#39;<a href='mailto:ns_1@10.3.121.92'>ns_1@10.3.121.92</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.3.121.98'>ns_1@10.3.121.98</a>&#39;,&#39;<a href='mailto:ns_1@10.3.121.93'>ns_1@10.3.121.93</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.3.121.97'>ns_1@10.3.121.97</a>&#39;,&#39;<a href='mailto:ns_1@10.3.121.95'>ns_1@10.3.121.95</a>&#39;], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[],[]}]}}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{trace, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{gen_fsm,sync_send_event,2}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{menelaus_web,do_handle_rebalance,3}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{menelaus_web,loop,3}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{mochiweb_http,headers,5}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{proc_lib,init_p_do_apply,3}]}] <br/> <br/> <br/> Ubuntu 10.04 MB-5108

Rolling upgrade from 172 to latest 181 fails with failed rebalance {type,exit}, {what,{noproc, {gen_fsm,sync_send_event,}}}

Bug Blocker Closed Fixed Public Aleksey Kondratenko Karan Kumar 1.8.1-release-notes Wed, 18 Apr 2012 13:47:57 -0500 Wed, 9 Jan 2013 22:59:32 -0600 Fri, 27 Apr 2012 15:53:35 -0500 1.8.1-release-candidate 1.8.1 ns_server 0 1 This looks to be regression in ns_server Found this to be issue in old ns_server. Rebalance requests needs to either be sent to node running new version or work around this issue by waiting and retrying. Commit that fixed it (for 1.8.0) is: <br/> <br/> commit d45ccaab92158d4a4fc882d3216d1557b7b39816 <br/> Author: Aliaksey Kandratsenka &lt;<a href='mailto:alk@tut.by'>alk@tut.by</a>&gt; <br/> Date: Tue Nov 29 15:10:09 2011 +0300 <br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;wait for orchestrator presense for key operations. <a href="http://www.couchbase.com/issues/browse/MB-4214" title="add and rebalancing a node fails in the first attempt in 2.0 ({gen_fsm,sync_send_event,2}, {menelaus_web,do_handle_rebalance,3}"><strike>MB-4214</strike></a> <a href="http://www.couchbase.com/issues/browse/MB-4559" title="Rolling upgrade from 172 to 180r-38 failed, unable to determine master node"><strike>MB-4559</strike></a> <br/> Not a &quot;bug&quot;. Still failing. The suggested workaround does not work. <br/> <br/> In the test we are issuing rebalance call to the newly upgraded 181 node.. <br/> <br/> Results in the rebalance failure.<br/> <br/> Rebalance exited with reason {{case_clause,<br/> {badrpc,<br/> {&#39;EXIT&#39;,<br/> {{badfun,#Fun&lt;erl_eval.4.88154533&gt;},<br/> [{erlang,apply,2},<br/> {rpc,&#39;-handle_call_call/6-fun-0-&#39;,5}]}}}},<br/> [{ns_vbm_sup,change_vbucket_filter,4},<br/> {ns_vbm_sup,&#39;-set_replicas/3-fun-2-&#39;,5},<br/> {lists,foldl,3},<br/> {ns_vbm_sup,set_replicas,3},<br/> {ns_vbm_sup,&#39;-set_replicas/2-fun-1-&#39;,3},<br/> {lists,foreach,2},<br/> {ns_vbm_sup,apply_changes,2},<br/> {ns_vbucket_mover,sync_replicas,0}]} Neither does waiting for the newly added node to become orchestrator solves this issue. That&#39;s different failure. Thanks for finding it. Fixed in <a href="http://review.couchbase.org/15366">http://review.couchbase.org/15366</a> Integrated in github-ns-server-2-0 #342 (See [<a href="http://qa.hq.northscale.net/job/github-ns-server-2-0/342/">http://qa.hq.northscale.net/job/github-ns-server-2-0/342/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;reimplemented backwards-compat for change_vbucket_filter. <a href="http://www.couchbase.com/issues/browse/MB-5108" title="Rolling upgrade from 172 to latest 181 fails with failed rebalance {type,exit}, {what,{noproc, {gen_fsm,sync_send_event,}}}"><strike>MB-5108</strike></a> (Revision 023a90b14d2530823602f9c0c1c03dc86c33013e)<br/> forward-ported new change_filter code (023a90b14). <a href="http://www.couchbase.com/issues/browse/MB-5108" title="Rolling upgrade from 172 to latest 181 fails with failed rebalance {type,exit}, {what,{noproc, {gen_fsm,sync_send_event,}}}"><strike>MB-5108</strike></a> (Revision f6d217bec4b036b617f6ccf19404ac5ef8b0b793)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> Aliaksey Kandratsenka : <br/> Files : <br/> * src/ns_vbm_sup.erl<br/> <br/> Aliaksey Kandratsenka : <br/> Files : <br/> * src/ns_vbm_sup.erl<br/> * src/cb_gen_vbm_sup.erl<br/> Date of First Response Wed, 18 Apr 2012 18:02:23 -0500 Rank 5166 Time In Status