http://www.couchbase.com/issues This file is an XML representation of an issue en-us 5.2.4 845 26-12-2012 http://www.couchbase.com/issues/browse/MB-7572 Couchbase Server build 139 <br/> <a href="http://qa.hq.northscale.net/job/centos-64-2.0-rebalance-regressions/170/consoleFull">http://qa.hq.northscale.net/job/centos-64-2.0-rebalance-regressions/170/consoleFull</a> <br/> ./testrunner -i /tmp/rebalance_regression.ini get-logs=True,wait_timeout=100 -t swaprebalance.SwapRebalanceFailedTests.test_add_back_failed_node,replica=2,num-buckets=3,num-swap=1,keys-count=1000000,swap-orchestrator=False <br/> <br/> <br/> steps: <br/> [2013-01-21 08:18:48,775] - [swaprebalance:472] INFO - failover node <a href='mailto:ns_1@10.3.121.94'>ns_1@10.3.121.94</a> and rebalance afterwards <br/> [2013-01-21 08:19:19,849] - [rest_client:767] INFO - fail_over successful <br/> [2013-01-21 08:19:20,071] - [rest_client:795] INFO - rebalance params : password=password&amp;ejectedNodes=ns_1%4010.3.121.94&amp;user=Administrator&amp;knownNodes=ns_1%4010.3.121.94%2Cns_1%4010.3.121.92%2Cns_1%4010.3.121.98%2Cns_1%4010.3.121.96%2Cns_1%4010.3.121.93%2Cns_1%4010.3.121.97%2Cns_1%4010.3.121.95 <br/> [2013-01-21 08:19:20,161] - [rest_client:799] INFO - rebalance operation started <br/> [2013-01-21 08:19:20,337] - [rest_client:894] INFO - rebalance percentage : 0 % <br/> ... <br/> [2013-01-21 09:11:14,394] - [rest_client:894] INFO - rebalance percentage : 68.451909784 % <br/> [2013-01-21 09:11:16,519] - [rest_client:879] ERROR - {u&#39;status&#39;: u&#39;none&#39;, u&#39;errorMessage&#39;: u&#39;Rebalance failed. See logs for detailed reason. You can try rebalance again.&#39;} - rebalance failed <br/> <br/> <br/> <br/> <br/> 2013-01-21 09:38:12.596 ns_vbucket_mover:0:critical:message(<a href='mailto:ns_1@10.3.121.93'>ns_1@10.3.121.93</a>) - &lt;0.25863.52&gt; exited with {badmatch, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{&lt;0.25911.52&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{{badmatch,[{&lt;20234.16808.17&gt;,killed}]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{misc, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;sync_shutdown_many_i_am_trapping_exits,1}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{misc,try_with_maybe_ignorant_after,2}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server,terminate,6}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{proc_lib,init_p_do_apply,3}]}}]} <br/> 2013-01-21 09:38:12.774 ns_orchestrator:2:info:message(<a href='mailto:ns_1@10.3.121.93'>ns_1@10.3.121.93</a>) - Rebalance exited with reason {badmatch, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{&lt;0.25911.52&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{{badmatch,[{&lt;20234.16808.17&gt;,killed}]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{misc, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;sync_shutdown_many_i_am_trapping_exits,1}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{misc,try_with_maybe_ignorant_after,2}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server,terminate,6}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{proc_lib,init_p_do_apply,3}]}}]} <br/> MB-7572

test_add_back_failed_node: Rebalance exited with reason {badmatch, [{<0.25911.52>, {{badmatch,[{<20234.16808.17>,killed}]}, [{misc,sync_shutdown_many_i_am_trapping_exits,1

Bug Major Closed Cannot Reproduce Public Andrei Baranouski Andrei Baranouski Tue, 22 Jan 2013 04:55:12 -0600 Mon, 22 Apr 2013 04:50:20 -0500 Mon, 28 Jan 2013 15:32:20 -0600 2.0.1 2.0.1 couchbase-bucket 0 1 <a href="https://s3.amazonaws.com/bugdb/jira/MB-7572/10.3.121.92-8091-diag.txt.gz">https://s3.amazonaws.com/bugdb/jira/MB-7572/10.3.121.92-8091-diag.txt.gz</a> <br/> <a href="https://s3.amazonaws.com/bugdb/jira/MB-7572/10.3.121.93-8091-diag.txt.gz">https://s3.amazonaws.com/bugdb/jira/MB-7572/10.3.121.93-8091-diag.txt.gz</a> <br/> <a href="https://s3.amazonaws.com/bugdb/jira/MB-7572/10.3.121.94-8091-diag.txt.gz">https://s3.amazonaws.com/bugdb/jira/MB-7572/10.3.121.94-8091-diag.txt.gz</a> <br/> <a href="https://s3.amazonaws.com/bugdb/jira/MB-7572/10.3.121.95-8091-diag.txt.gz">https://s3.amazonaws.com/bugdb/jira/MB-7572/10.3.121.95-8091-diag.txt.gz</a> <br/> <a href="https://s3.amazonaws.com/bugdb/jira/MB-7572/10.3.121.96-8091-diag.txt.gz">https://s3.amazonaws.com/bugdb/jira/MB-7572/10.3.121.96-8091-diag.txt.gz</a> <br/> <a href="https://s3.amazonaws.com/bugdb/jira/MB-7572/10.3.121.97-8091-diag.txt.gz">https://s3.amazonaws.com/bugdb/jira/MB-7572/10.3.121.97-8091-diag.txt.gz</a> <br/> <a href="https://s3.amazonaws.com/bugdb/jira/MB-7572/10.3.121.98-8091-diag.txt.gz">https://s3.amazonaws.com/bugdb/jira/MB-7572/10.3.121.98-8091-diag.txt.gz</a> <br/> <br/> andrei, <br/> <br/> does this exact same test pass against 2.0 ? yes, sure Log file was rotated long past the rebalance failure. Please reproduce and upload diags grabbed shortly after the failure. sorry, Aliaksey <br/> <br/> following logs should be ok <br/> <br/> <a href="https://s3.amazonaws.com/bugdb/jira/MB-7572/b17838ff-4d37-4768-b8fa-cd6a0bcfa8fe-10.3.121.92-diag.txt.gz">https://s3.amazonaws.com/bugdb/jira/MB-7572/b17838ff-4d37-4768-b8fa-cd6a0bcfa8fe-10.3.121.92-diag.txt.gz</a> <br/> <a href="https://s3.amazonaws.com/bugdb/jira/MB-7572/b17838ff-4d37-4768-b8fa-cd6a0bcfa8fe-10.3.121-93-diag.txt.gz">https://s3.amazonaws.com/bugdb/jira/MB-7572/b17838ff-4d37-4768-b8fa-cd6a0bcfa8fe-10.3.121-93-diag.txt.gz</a> <br/> <a href="https://s3.amazonaws.com/bugdb/jira/MB-7572/b17838ff-4d37-4768-b8fa-cd6a0bcfa8fe-10.3.121-94-diag.txt.gz">https://s3.amazonaws.com/bugdb/jira/MB-7572/b17838ff-4d37-4768-b8fa-cd6a0bcfa8fe-10.3.121-94-diag.txt.gz</a> <br/> <a href="https://s3.amazonaws.com/bugdb/jira/MB-7572/b17838ff-4d37-4768-b8fa-cd6a0bcfa8fe-10.3.121-95-diag.txt.gz">https://s3.amazonaws.com/bugdb/jira/MB-7572/b17838ff-4d37-4768-b8fa-cd6a0bcfa8fe-10.3.121-95-diag.txt.gz</a> <br/> <a href="https://s3.amazonaws.com/bugdb/jira/MB-7572/b17838ff-4d37-4768-b8fa-cd6a0bcfa8fe-10.3.121-96-diag.txt.gz">https://s3.amazonaws.com/bugdb/jira/MB-7572/b17838ff-4d37-4768-b8fa-cd6a0bcfa8fe-10.3.121-96-diag.txt.gz</a> <br/> <a href="https://s3.amazonaws.com/bugdb/jira/MB-7572/b17838ff-4d37-4768-b8fa-cd6a0bcfa8fe-10.3.121-97-diag.txt.gz">https://s3.amazonaws.com/bugdb/jira/MB-7572/b17838ff-4d37-4768-b8fa-cd6a0bcfa8fe-10.3.121-97-diag.txt.gz</a> <br/> <a href="https://s3.amazonaws.com/bugdb/jira/MB-7572/b17838ff-4d37-4768-b8fa-cd6a0bcfa8fe-10.3.121-98-diag.txt.gz">https://s3.amazonaws.com/bugdb/jira/MB-7572/b17838ff-4d37-4768-b8fa-cd6a0bcfa8fe-10.3.121-98-diag.txt.gz</a> <br/> <br/> So here what happened. <br/> <br/> Node .93 builds a replica for vbucket 635 on node .95: <br/> <br/> [ns_server:debug,2013-01-21T9:37:00.815,<a href='mailto:ns_1@10.3.121.93'>ns_1@10.3.121.93</a>:&lt;0.25911.52&gt;:ns_replicas_builder_utils:spawn_replica_builder:86]Replica building ebucketmigrator for vbucket 635 into &#39;<a href='mailto:ns_1@10.3.121.95'>ns_1@10.3.121.95</a>&#39; is &lt;20234.16808.17&gt; <br/> <br/> Relevant messages from the .95: <br/> <br/> [ns_server:info,2013-01-21T9:37:00.723,<a href='mailto:ns_1@10.3.121.95'>ns_1@10.3.121.95</a>:&lt;0.16808.17&gt;:ebucketmigrator_srv:init:509]Setting {&quot;10.3.121.95&quot;,11209} vbucket 635 to state replica <br/> [ns_server:info,2013-01-21T9:37:00.815,<a href='mailto:ns_1@10.3.121.95'>ns_1@10.3.121.95</a>:ns_port_memcached&lt;0.15884.7&gt;:ns_port_server:log:171]memcached&lt;0.15884.7&gt;: Mon Jan 21 09:37:00.613175 PST 3: TAP (Producer) eq_tapq:<a href='mailto:replication_ns_1@10.3.121.93'>replication_ns_1@10.3.121.93</a> - Sending TAP_OPAQUE with command &quot;complete_vb_filter_change&quot; and vbucket 0 <br/> <br/> [ns_server:debug,2013-01-21T9:37:00.869,<a href='mailto:ns_1@10.3.121.95'>ns_1@10.3.121.95</a>:&lt;0.16808.17&gt;:ebucketmigrator_srv:kill_tapname:1005]killing tap named: replication_building_635_&#39;<a href='mailto:ns_1@10.3.121.95'>ns_1@10.3.121.95</a>&#39; <br/> [rebalance:info,2013-01-21T9:37:00.876,<a href='mailto:ns_1@10.3.121.95'>ns_1@10.3.121.95</a>:&lt;0.16808.17&gt;:ebucketmigrator_srv:init:568]Starting tap stream: <br/> [{vbuckets,[635]}, <br/> &nbsp;{checkpoints,[{635,0}]}, <br/> &nbsp;{name,&lt;&lt;&quot;replication_building_635_&#39;<a href='mailto:ns_1@10.3.121.95'>ns_1@10.3.121.95</a>&#39;&quot;&gt;&gt;}, <br/> &nbsp;{takeover,false}] <br/> {{&quot;10.3.121.96&quot;,11209}, <br/> &nbsp;{&quot;10.3.121.95&quot;,11209}, <br/> &nbsp;[{vbuckets,[635]}, <br/> &nbsp;&nbsp;{takeover,false}, <br/> &nbsp;&nbsp;{suffix,&quot;building_635_&#39;<a href='mailto:ns_1@10.3.121.95'>ns_1@10.3.121.95</a>&#39;&quot;}, <br/> &nbsp;&nbsp;{username,&quot;bucket-0&quot;}, <br/> &nbsp;&nbsp;{password,&quot;password&quot;}]} <br/> [rebalance:debug,2013-01-21T9:37:00.887,<a href='mailto:ns_1@10.3.121.95'>ns_1@10.3.121.95</a>:&lt;0.16808.17&gt;:ebucketmigrator_srv:init:605]upstream_sender pid: &lt;0.16809.17&gt; <br/> [rebalance:debug,2013-01-21T9:37:00.888,<a href='mailto:ns_1@10.3.121.95'>ns_1@10.3.121.95</a>:&lt;0.16808.17&gt;:ebucketmigrator_srv:handle_call:336]Suspended had_backfill waiter <br/> {had_backfill,undefined,undefined, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{&lt;18136.25927.52&gt;,#Ref&lt;18136.0.225.131688&gt;}]} <br/> [rebalance:info,2013-01-21T9:37:00.889,<a href='mailto:ns_1@10.3.121.95'>ns_1@10.3.121.95</a>:&lt;0.16808.17&gt;:ebucketmigrator_srv:process_upstream:932]Initial stream for vbucket 635 <br/> [rebalance:debug,2013-01-21T9:37:00.890,<a href='mailto:ns_1@10.3.121.95'>ns_1@10.3.121.95</a>:&lt;0.16808.17&gt;:ebucketmigrator_srv:process_downstream:896]Replied had_backfill: true to [{&lt;18136.25927.52&gt;,#Ref&lt;18136.0.225.131688&gt;}] <br/> [ns_server:info,2013-01-21T9:37:00.925,<a href='mailto:ns_1@10.3.121.95'>ns_1@10.3.121.95</a>:&lt;0.16810.17&gt;:ebucketmigrator_srv:init:509]Setting {&quot;10.3.121.95&quot;,11209} vbucket 811 to state replica <br/> [ns_server:info,2013-01-21T9:37:01.094,<a href='mailto:ns_1@10.3.121.95'>ns_1@10.3.121.95</a>:ns_port_memcached&lt;0.15884.7&gt;:ns_port_server:log:171]memcached&lt;0.15884.7&gt;: Mon Jan 21 09:37:00.890055 PST 3: TAP (Consumer) eq_tapq:anon_3574 - Reset vbucket 635 was completed succecssfully. <br/> <br/> Then we see a TAP_OPAQUE_CLOSE_BACKFILL: <br/> <br/> [ns_server:debug,2013-01-21T9:37:11.320,<a href='mailto:ns_1@10.3.121.95'>ns_1@10.3.121.95</a>:&lt;0.16808.17&gt;:ebucketmigrator_srv:process_upstream:946]seen backfill-close message <br/> <br/> Then we attempt to shutdown the ebucketmigrator. As part of this, to confirm all the messages that we&#39;ve sent to downstream we send an opaque message with (last seen sequence number + 1) and wait until downstream replies. We set timeout to 30 seconds for the entire sequence of actions. <br/> <br/> [ns_server:debug,2013-01-21T9:37:22.270,<a href='mailto:ns_1@10.3.121.95'>ns_1@10.3.121.95</a>:&lt;0.16808.17&gt;:ebucketmigrator_srv:confirm_sent_messages:727]Going to wait for reception of opaque message ack <br/> [ns_server:debug,2013-01-21T9:37:22.270,<a href='mailto:ns_1@10.3.121.95'>ns_1@10.3.121.95</a>:&lt;0.17215.17&gt;:ebucketmigrator_srv:confirm_sent_messages:722]Sending opaque message to confirm downstream reception <br/> [ns_server:debug,2013-01-21T9:37:22.270,<a href='mailto:ns_1@10.3.121.95'>ns_1@10.3.121.95</a>:&lt;0.17215.17&gt;:ebucketmigrator_srv:confirm_sent_messages:724]Sent fine <br/> <br/> And exactly 30 seconds later we see that the ebucketmigrator got killed since timeout expired: <br/> <br/> [ns_server:error,2013-01-21T9:37:52.271,<a href='mailto:ns_1@10.3.121.93'>ns_1@10.3.121.93</a>:&lt;0.25911.52&gt;:misc:sync_shutdown_many_i_am_trapping_exits:1416]Shutdown of the following failed: [{&lt;20234.16808.17&gt;,killed}] <br/> <br/> So the problem seems to be that memcached on .95 didn&#39;t manage to reply to us in 30 seconds. <br/> Please see my comment above Chiyoung. per bug-scrub: is this a regression? memcached log was truncated unfortunately. Can you upload the separated memcached log file from the node .95? don&#39;t have logs anymore. Cluster has been cleaned Andrei, <br/> <br/> did test pass in the next iteration ? this test passed against 144 please reopen if test fails again Date of First Response Tue, 22 Jan 2013 10:28:31 -0600 Rank 3287 Time In Status