http://www.couchbase.com/issues This file is an XML representation of an issue en-us 5.2.4 845 26-12-2012 http://www.couchbase.com/issues/browse/MB-5343 Couchbase Server Failing test is:- <br/> rebalancetests.RebalanceInOutWithParallelLoad.test_load,get-logs:True,replica:2,num_nodes:7 <br/> <br/> [ns_server:info] [2012-05-21 8:17:31] [<a href='mailto:ns_1@10.1.3.109'>ns_1@10.1.3.109</a>:&lt;0.15105.17&gt;:ns_replicas_builder:kill_a_bunch_of_tap_names:209] Killed the following tap names on &#39;<a href='mailto:ns_1@10.1.3.109'>ns_1@10.1.3.109</a>&#39;: [&lt;&lt;&quot;replication_building_62_&#39;<a href='mailto:ns_1@10.1.3.112'>ns_1@10.1.3.112</a>&#39;&quot;&gt;&gt;] <br/> [ns_server:info] [2012-05-21 8:17:31] [<a href='mailto:ns_1@10.1.3.109'>ns_1@10.1.3.109</a>:&lt;0.15104.17&gt;:ns_single_vbucket_mover:mover_inner:88] Got exit message (parent is &lt;0.13796.17&gt;). Exiting... <br/> {&#39;EXIT&#39;,&lt;0.15105.17&gt;,{replicator_died,{&#39;EXIT&#39;,&lt;16541.21868.10&gt;,normal}}} <br/> [ns_server:error] [2012-05-21 8:17:31] [<a href='mailto:ns_1@10.1.3.109'>ns_1@10.1.3.109</a>:&lt;0.15104.17&gt;:ns_replicas_builder:sync_shutdown_many:147] Shutdown of the following failed: [{&lt;0.15105.17&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{replicator_died, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&#39;EXIT&#39;,&lt;16541.21868.10&gt;,normal}}}] <br/> [error_logger:error] [2012-05-21 8:17:31] [<a href='mailto:ns_1@10.1.3.109'>ns_1@10.1.3.109</a>:error_logger:ale_error_logger_handler:log_report:72] <br/> =========================CRASH REPORT========================= <br/> &nbsp;&nbsp;crasher: <br/> &nbsp;&nbsp;&nbsp;&nbsp;initial call: erlang:apply/2 <br/> &nbsp;&nbsp;&nbsp;&nbsp;pid: &lt;0.15105.17&gt; <br/> &nbsp;&nbsp;&nbsp;&nbsp;registered_name: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;exception exit: {replicator_died,{&#39;EXIT&#39;,&lt;16541.21868.10&gt;,normal}} <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in function ns_replicas_builder:&#39;-build_replicas_main/6-fun-0-&#39;/1 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in call from ns_replicas_builder:observe_wait_all_done_tail/5 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in call from ns_replicas_builder:observe_wait_all_done/5 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in call from ns_replicas_builder:&#39;-build_replicas_main/6-fun-1-&#39;/8 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in call from ns_replicas_builder:try_with_maybe_ignorant_after/2 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in call from ns_replicas_builder:build_replicas_main/6 <br/> &nbsp;&nbsp;&nbsp;&nbsp;ancestors: [&lt;0.15104.17&gt;,&lt;0.13796.17&gt;,&lt;0.13763.17&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;messages: [{&#39;EXIT&#39;,&lt;16541.21868.10&gt;,normal}] <br/> &nbsp;&nbsp;&nbsp;&nbsp;links: [&lt;0.15104.17&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;dictionary: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;trap_exit: true <br/> &nbsp;&nbsp;&nbsp;&nbsp;status: running <br/> &nbsp;&nbsp;&nbsp;&nbsp;heap_size: 121393 <br/> &nbsp;&nbsp;&nbsp;&nbsp;stack_size: 24 <br/> &nbsp;&nbsp;&nbsp;&nbsp;reductions: 12423 <br/> &nbsp;&nbsp;neighbours: <br/> <br/> [ns_server:error] [2012-05-21 8:17:31] [<a href='mailto:ns_1@10.1.3.109'>ns_1@10.1.3.109</a>:&lt;0.15104.17&gt;:ns_replicas_builder:try_with_maybe_ignorant_after:68] Eating exception from ignorant after-block: <br/> {error,{badmatch,[{&lt;0.15105.17&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{replicator_died,{&#39;EXIT&#39;,&lt;16541.21868.10&gt;,normal}}}]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{ns_replicas_builder,sync_shutdown_many,1}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{ns_replicas_builder,try_with_maybe_ignorant_after,2}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{ns_single_vbucket_mover,mover,6}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{proc_lib,init_p_do_apply,3}]} <br/> [rebalance:error] [2012-05-21 8:17:31] [<a href='mailto:ns_1@10.1.3.109'>ns_1@10.1.3.109</a>:&lt;0.13796.17&gt;:ns_vbucket_mover:handle_info:158] &lt;0.15104.17&gt; exited with {exited, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&#39;EXIT&#39;,&lt;0.15105.17&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{replicator_died, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&#39;EXIT&#39;,&lt;16541.21868.10&gt;,normal}}}} <br/> [ns_server:info] [2012-05-21 8:17:31] [<a href='mailto:ns_1@10.1.3.109'>ns_1@10.1.3.109</a>:&lt;0.9348.1&gt;:ns_port_server:log:161] memcached&lt;0.9348.1&gt;: TAP (Producer) eq_tapq:rebalance_61 - Schedule the backfill for vbucket 61 <br/> memcached&lt;0.9348.1&gt;: TAP (Producer) eq_tapq:rebalance_61 - Sending TAP_OPAQUE with command &quot;opaque_enable_auto_nack&quot; and vbucket 0 <br/> memcached&lt;0.9348.1&gt;: TAP (Producer) eq_tapq:rebalance_61 - Sending TAP_OPAQUE with command &quot;enable_checkpoint_sync&quot; and vbucket 0 <br/> memcached&lt;0.9348.1&gt;: TAP (Producer) eq_tapq:rebalance_61 - Sending TAP_VBUCKET_SET with vbucket 61 and state &quot;pending&quot; <br/> memcached&lt;0.9348.1&gt;: TAP (Producer) eq_tapq:rebalance_61 - Sending TAP_OPAQUE with command &quot;initial_vbucket_stream&quot; and vbucket 61 <br/> memcached&lt;0.9348.1&gt;: TAP (Producer) eq_tapq:rebalance_61 - Backfill is completed with VBuckets 61, <br/> memcached&lt;0.9348.1&gt;: TAP (Producer) eq_tapq:rebalance_61 - Sending TAP_OPAQUE with command &quot;close_backfill&quot; and vbucket 61 <br/> memcached&lt;0.9348.1&gt;: Vbucket &lt;61&gt; is going dead. <br/> memcached&lt;0.9348.1&gt;: TAP (Producer) eq_tapq:rebalance_61 - Sending TAP_VBUCKET_SET with vbucket 61 and state &quot;active&quot; <br/> memcached&lt;0.9348.1&gt;: TAP takeover is completed. Disconnecting tap stream &lt;eq_tapq:rebalance_61&gt; <br/> memcached&lt;0.9348.1&gt;: TAP (Producer) eq_tapq:replication_building_62_&#39;<a href='mailto:ns_1@10.1.3.112'>ns_1@10.1.3.112</a>&#39; - Schedule the backfill for vbucket 62 <br/> memcached&lt;0.9348.1&gt;: TAP (Producer) eq_tapq:replication_building_62_&#39;<a href='mailto:ns_1@10.1.3.112'>ns_1@10.1.3.112</a>&#39; - Sending TAP_OPAQUE with command &quot;opaque_enable_auto_nack&quot; and vbucket 0 <br/> memcached&lt;0.9348.1&gt;: TAP (Producer) eq_tapq:replication_building_62_&#39;<a href='mailto:ns_1@10.1.3.112'>ns_1@10.1.3.112</a>&#39; - Sending TAP_OPAQUE with command &quot;enable_checkpoint_sync&quot; and vbucket 0 <br/> memcached&lt;0.9348.1&gt;: TAP (Producer) eq_tapq:replication_building_62_&#39;<a href='mailto:ns_1@10.1.3.112'>ns_1@10.1.3.112</a>&#39; - Sending TAP_OPAQUE with command &quot;initial_vbucket_stream&quot; and vbucket 62 <br/> <br/> [error_logger:error] [2012-05-21 8:17:31] [<a href='mailto:ns_1@10.1.3.109'>ns_1@10.1.3.109</a>:error_logger:ale_error_logger_handler:log_report:72] <br/> =========================CRASH REPORT========================= <br/> &nbsp;&nbsp;crasher: <br/> &nbsp;&nbsp;&nbsp;&nbsp;initial call: ns_single_vbucket_mover:mover/6 <br/> &nbsp;&nbsp;&nbsp;&nbsp;pid: &lt;0.15104.17&gt; <br/> &nbsp;&nbsp;&nbsp;&nbsp;registered_name: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;exception exit: {exited, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&#39;EXIT&#39;,&lt;0.15105.17&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{replicator_died, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&#39;EXIT&#39;,&lt;16541.21868.10&gt;,normal}}}} <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in function ns_single_vbucket_mover:mover_inner/6 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in call from ns_replicas_builder:try_with_maybe_ignorant_after/2 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in call from ns_single_vbucket_mover:mover/6 <br/> &nbsp;&nbsp;&nbsp;&nbsp;ancestors: [&lt;0.13796.17&gt;,&lt;0.13763.17&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;messages: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;links: [&lt;0.13796.17&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;dictionary: [{cleanup_list,[&lt;0.15105.17&gt;]}] <br/> &nbsp;&nbsp;&nbsp;&nbsp;trap_exit: true <br/> &nbsp;&nbsp;&nbsp;&nbsp;status: running <br/> &nbsp;&nbsp;&nbsp;&nbsp;heap_size: 987 <br/> &nbsp;&nbsp;&nbsp;&nbsp;stack_size: 24 <br/> &nbsp;&nbsp;&nbsp;&nbsp;reductions: 4014 <br/> <br/> <br/> <br/> Ubuntu 64 bit <br/> 181-831-rel MB-5343

memcached dropped connections: Rebalance failed due to replicator_died: exited (ns_single_vbucket_mover)

Bug Critical Resolved Fixed Public Frank Weigel Karan Kumar 1.8.1-release-notes Mon, 21 May 2012 13:43:25 -0500 Mon, 28 Jan 2013 23:49:28 -0600 Mon, 28 Jan 2013 23:49:28 -0600 1.8.1-release-candidate 2.0 couchbase-bucket 0 4 ep-engine closed tap-producer side of socket. We don&#39;t expect this for replica building sockets. Here are relevant logs from ebucketmigrator (run on destination node): <br/> <br/> [ns_server:info] [2012-05-21 8:17:31] [<a href='mailto:ns_1@10.1.3.112'>ns_1@10.1.3.112</a>:&lt;0.21868.10&gt;:ebucketmigrator_srv:init:181] Setting {&quot;10.1.3.112&quot;,11209} vbucket 62 to state replica <br/> [rebalance:debug] [2012-05-21 8:17:31] [<a href='mailto:ns_1@10.1.3.112'>ns_1@10.1.3.112</a>:&lt;0.21868.10&gt;:ebucketmigrator_srv:init:186] CheckpointIdsDict: <br/> {dict,64,16,16,8,80,48, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{{[[0|1],[16|1],[32|1],[48|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[3|1],[19|1],[35|1],[51|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[6|1],[22|1],[38|1],[54|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[9|1],[25|1],[41|1],[57|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[12|1],[28|1],[44|1],[60|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[15|1],[31|1],[47|1],[63|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[2|1],[18|1],[34|1],[50|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[5|1],[21|1],[37|1],[53|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[8|1],[24|1],[40|1],[56|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[11|1],[27|1],[43|1],[59|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[14|1],[30|1],[46|1],[62|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[1|1],[17|1],[33|1],[49|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[4|1],[20|1],[36|1],[52|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[7|1],[23|1],[39|1],[55|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[10|1],[26|1],[42|1],[58|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[13|1],[29|1],[45|1],[61|1]]}}} <br/> [ns_server:debug] [2012-05-21 8:17:31] [<a href='mailto:ns_1@10.1.3.112'>ns_1@10.1.3.112</a>:&lt;0.21868.10&gt;:ebucketmigrator_srv:init:209] killing tap named: replication_building_62_&#39;<a href='mailto:ns_1@10.1.3.112'>ns_1@10.1.3.112</a>&#39; <br/> [rebalance:debug] [2012-05-21 8:17:31] [<a href='mailto:ns_1@10.1.3.112'>ns_1@10.1.3.112</a>:&lt;0.21868.10&gt;:ebucketmigrator_srv:init:247] upstream_sender pid: &lt;0.21869.10&gt; <br/> [rebalance:info] [2012-05-21 8:17:31] [<a href='mailto:ns_1@10.1.3.112'>ns_1@10.1.3.112</a>:&lt;0.21868.10&gt;:ebucketmigrator_srv:process_upstream:447] Initial stream for vbucket 62 <br/> [rebalance:info] [2012-05-21 8:17:31] [<a href='mailto:ns_1@10.1.3.112'>ns_1@10.1.3.112</a>:&lt;0.21868.10&gt;:ebucketmigrator_srv:do_confirm_sent_messages:315] Got close ack! <br/> <br/> And this is logs from memcached on source node (.109) <br/> <br/> memcached&lt;0.9348.1&gt;: TAP (Producer) eq_tapq:replication_building_62_&#39;<a href='mailto:ns_1@10.1.3.112'>ns_1@10.1.3.112</a>&#39; - Schedule the backfill for vbucket 62 <br/> memcached&lt;0.9348.1&gt;: TAP (Producer) eq_tapq:replication_building_62_&#39;<a href='mailto:ns_1@10.1.3.112'>ns_1@10.1.3.112</a>&#39; - Sending TAP_OPAQUE with command &quot;opaque_enable_auto_nack&quot; and vbucket 0 <br/> memcached&lt;0.9348.1&gt;: TAP (Producer) eq_tapq:replication_building_62_&#39;<a href='mailto:ns_1@10.1.3.112'>ns_1@10.1.3.112</a>&#39; - Sending TAP_OPAQUE with command &quot;enable_checkpoint_sync&quot; and vbucket 0 <br/> memcached&lt;0.9348.1&gt;: TAP (Producer) eq_tapq:replication_building_62_&#39;<a href='mailto:ns_1@10.1.3.112'>ns_1@10.1.3.112</a>&#39; - Sending TAP_OPAQUE with command &quot;initial_vbucket_stream&quot; and vbucket 62 <br/> Passing down to Chiyoung. See message above Karan, <br/> <br/> Did you test it with 64 or 1024 vbuckets? <br/> <br/> I&#39;m running the same regression tests on my 6VM cluster, but would like to know if I can set it to 64 vbuckets to reproduce this issue more quickly. 1024 I was trying to reproduce this issue. Is the following command correct? <br/> <br/> python testrunner -i my_vm.ini -t rebalancetests.RebalanceInOutWithParallelLoad.test_load,get-logs=True,replica=2,num_nodes=6 -p skip-cleanup=True yes. Are you having issues? <br/> Let me know. Hitting the issue on windows as well. 181-871-rel <br/> Attaching the logs. [ns_server:error] [2012-06-07 1:44:48] [<a href='mailto:ns_1@10.1.3.82'>ns_1@10.1.3.82</a>:&lt;0.30696.89&gt;:ns_replicas_builder:build_replicas_main:99] Got premature exit from one of ebucketmigrators: {&#39;EXIT&#39;,&lt;16944.21895.27&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;normal} <br/> [ns_server:info] [2012-06-07 1:44:48] [<a href='mailto:ns_1@10.1.3.82'>ns_1@10.1.3.82</a>:&lt;0.30696.89&gt;:ns_replicas_builder:kill_a_bunch_of_tap_names:209] Killed the following tap names on &#39;<a href='mailto:ns_1@10.1.3.86'>ns_1@10.1.3.86</a>&#39;: [&lt;&lt;&quot;replication_building_291_&#39;<a href='mailto:ns_1@10.1.3.87'>ns_1@10.1.3.87</a>&#39;&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;replication_building_291_&#39;<a href='mailto:ns_1@10.1.3.85'>ns_1@10.1.3.85</a>&#39;&quot;&gt;&gt;] <br/> [ns_server:info] [2012-06-07 1:44:48] [<a href='mailto:ns_1@10.1.3.82'>ns_1@10.1.3.82</a>:&lt;0.30695.89&gt;:ns_single_vbucket_mover:mover_inner:88] Got exit message (parent is &lt;0.25817.89&gt;). Exiting... <br/> {&#39;EXIT&#39;,&lt;0.30696.89&gt;,{replicator_died,{&#39;EXIT&#39;,&lt;16944.21895.27&gt;,normal}}} <br/> [ns_server:error] [2012-06-07 1:44:48] [<a href='mailto:ns_1@10.1.3.82'>ns_1@10.1.3.82</a>:&lt;0.30695.89&gt;:ns_replicas_builder:sync_shutdown_many:147] Shutdown of the following failed: [{&lt;0.30696.89&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{replicator_died, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&#39;EXIT&#39;,&lt;16944.21895.27&gt;,normal}}}] <br/> [ns_server:error] [2012-06-07 1:44:48] [<a href='mailto:ns_1@10.1.3.82'>ns_1@10.1.3.82</a>:&lt;0.30695.89&gt;:ns_replicas_builder:try_with_maybe_ignorant_after:68] Eating exception from ignorant after-block: <br/> {error,{badmatch,[{&lt;0.30696.89&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{replicator_died,{&#39;EXIT&#39;,&lt;16944.21895.27&gt;,normal}}}]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{ns_replicas_builder,sync_shutdown_many,1}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{ns_replicas_builder,try_with_maybe_ignorant_after,2}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{ns_single_vbucket_mover,mover,6}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{proc_lib,init_p_do_apply,3}]} <br/> <br/> <br/> =========================CRASH REPORT========================= <br/> &nbsp;&nbsp;crasher: <br/> &nbsp;&nbsp;&nbsp;&nbsp;initial call: erlang:apply/2 <br/> &nbsp;&nbsp;&nbsp;&nbsp;pid: &lt;0.30696.89&gt; <br/> &nbsp;&nbsp;&nbsp;&nbsp;registered_name: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;exception exit: {replicator_died,{&#39;EXIT&#39;,&lt;16944.21895.27&gt;,normal}} <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in function ns_replicas_builder:&#39;-build_replicas_main/6-fun-0-&#39;/1 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in call from ns_replicas_builder:observe_wait_all_done_tail/5 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in call from ns_replicas_builder:observe_wait_all_done/5 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in call from ns_replicas_builder:&#39;-build_replicas_main/6-fun-1-&#39;/8 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in call from ns_replicas_builder:try_with_maybe_ignorant_after/2 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in call from ns_replicas_builder:build_replicas_main/6 <br/> &nbsp;&nbsp;&nbsp;&nbsp;ancestors: [&lt;0.30695.89&gt;,&lt;0.25817.89&gt;,&lt;0.25765.89&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;messages: [{&#39;EXIT&#39;,&lt;16944.21895.27&gt;,normal}] <br/> &nbsp;&nbsp;&nbsp;&nbsp;links: [&lt;0.30695.89&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;dictionary: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;trap_exit: true <br/> &nbsp;&nbsp;&nbsp;&nbsp;status: running <br/> &nbsp;&nbsp;&nbsp;&nbsp;heap_size: 514229 <br/> &nbsp;&nbsp;&nbsp;&nbsp;stack_size: 24 <br/> &nbsp;&nbsp;&nbsp;&nbsp;reductions: 53630 <br/> &nbsp;&nbsp;neighbours: <br/> <br/> =========================CRASH REPORT========================= <br/> &nbsp;&nbsp;crasher: <br/> &nbsp;&nbsp;&nbsp;&nbsp;initial call: ns_single_vbucket_mover:mover/6 <br/> &nbsp;&nbsp;&nbsp;&nbsp;pid: &lt;0.30764.89&gt; <br/> &nbsp;&nbsp;&nbsp;&nbsp;registered_name: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;exception exit: {exited, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&#39;EXIT&#39;,&lt;0.25817.89&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{exited, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&#39;EXIT&#39;,&lt;0.30696.89&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{replicator_died,{&#39;EXIT&#39;,&lt;16944.21895.27&gt;,normal}}}}}} <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in function ns_single_vbucket_mover:mover_inner/6 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in call from ns_replicas_builder:try_with_maybe_ignorant_after/2 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in call from ns_single_vbucket_mover:mover/6 <br/> &nbsp;&nbsp;&nbsp;&nbsp;ancestors: [&lt;0.25817.89&gt;,&lt;0.25765.89&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;messages: [{&#39;EXIT&#39;,&lt;0.25817.89&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{exited, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&#39;EXIT&#39;,&lt;0.30696.89&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{replicator_died, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&#39;EXIT&#39;,&lt;16944.21895.27&gt;,normal}}}}}] <br/> &nbsp;&nbsp;&nbsp;&nbsp;links: [&lt;0.25817.89&gt;,&lt;0.30766.89&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;dictionary: [{cleanup_list,[&lt;0.30765.89&gt;]}] <br/> &nbsp;&nbsp;&nbsp;&nbsp;trap_exit: true <br/> &nbsp;&nbsp;&nbsp;&nbsp;status: running <br/> &nbsp;&nbsp;&nbsp;&nbsp;heap_size: 2584 <br/> &nbsp;&nbsp;&nbsp;&nbsp;stack_size: 24 <br/> &nbsp;&nbsp;&nbsp;&nbsp;reductions: 1767 <br/> &nbsp;&nbsp;neighbours: <br/> Hitting this occasionally on 181-907-rel Can you upload the latest diag logs? We have logs from June 7th. Hitting this on 181-910-rel (latest) <br/> <br/> I will upload the logs. <br/> @Chiyoung, does it look to be a issue similar to memcached dropping connection? <br/> <br/> [ns_server:info] [2012-06-14 2:09:36] [<a href='mailto:ns_1@10.1.3.109'>ns_1@10.1.3.109</a>:ns_port_memcached:ns_port_server:log:166] memcached&lt;0.252.0&gt;: Schedule cleanup of &quot;eq_tapq:anon_23189&quot; <br/> memcached&lt;0.252.0&gt;: TAP (Producer) eq_tapq:replication_building_94_&#39;<a href='mailto:ns_1@10.1.3.112'>ns_1@10.1.3.112</a>&#39; - Clear the tap queues by force <br/> memcached&lt;0.252.0&gt;: TAP (Consumer) eq_tapq:anon_23186 - disconnected <br/> memcached&lt;0.252.0&gt;: Schedule cleanup of &quot;eq_tapq:anon_23186&quot; <br/> memcached&lt;0.252.0&gt;: TAP (Producer) eq_tapq:<a href='mailto:replication_ns_1@10.1.3.112'>replication_ns_1@10.1.3.112</a> - disconnected, keep alive for 300 seconds <br/> memcached&lt;0.252.0&gt;: TAP (Consumer) eq_tapq:anon_23187 - disconnected <br/> memcached&lt;0.252.0&gt;: Schedule cleanup of &quot;eq_tapq:anon_23187&quot; <br/> <br/> [ns_server:info] [2012-06-14 2:09:36] [<a href='mailto:ns_1@10.1.3.109'>ns_1@10.1.3.109</a>:&lt;0.19238.14&gt;:ns_replicas_builder:kill_a_bunch_of_tap_names:219] Killed the following tap names on &#39;<a href='mailto:ns_1@10.1.3.112'>ns_1@10.1.3.112</a>&#39;: [&lt;&lt;&quot;replication_building_606_&#39;<a href='mailto:ns_1@10.1.3.109'>ns_1@10.1.3.109</a>&#39;&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;replication_building_606_&#39;<a href='mailto:ns_1@10.1.3.111'>ns_1@10.1.3.111</a>&#39;&quot;&gt;&gt;] <br/> [ns_server:info] [2012-06-14 2:09:36] [<a href='mailto:ns_1@10.1.3.109'>ns_1@10.1.3.109</a>:&lt;0.19238.14&gt;:ns_replicas_builder:build_replicas_main:143] Got exit: {&#39;EXIT&#39;,&lt;0.19237.14&gt;,shutdown} <br/> [ns_server:error] [2012-06-14 2:09:36] [<a href='mailto:ns_1@10.1.3.109'>ns_1@10.1.3.109</a>:&lt;0.19272.14&gt;:ns_replicas_builder:build_replicas_main:109] Got premature exit from one of ebucketmigrators: {&#39;EXIT&#39;,&lt;16817.4853.4&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;normal} Seeing same issue on Swap-Rebalance on 181-916 build. <br/> Rebalance exited with reason {exited, <br/> {&#39;EXIT&#39;,&lt;0.12083.14&gt;, <br/> {replicator_died, <br/> {&#39;EXIT&#39;,&lt;15753.26224.12&gt;,normal}}}} <br/> <br/> Steps: <br/> 1.A large cluster (18 nodes), with 3 buckets ( 50M , 22M, 0.5 M items each). <br/> 2. Enable auto-failover <br/> 3. Remove master-node -105, add node 126 and Rebalance <br/> <br/> Output <br/> Rebalance fails with &quot;replicator_died&quot; error. <br/> <br/> Logs attached at - <br/> <a href="https://s3.amazonaws.com/bugdb/jira/bug3-rebalance-181/bug3.tar">https://s3.amazonaws.com/bugdb/jira/bug3-rebalance-181/bug3.tar</a> jira-error : Not able to delete above comment- Please ignore. Let&#39;s see if we catch in 2.0 regression runs. got the same error on build-1554 for test: <br/> ./testrunner -i andrei.ini -t view.createdeleteview.CreateDeleteViewTests.rebalance_in_and_out_with_ddoc_ops,ddoc_ops=delete,test_with_view=True,num_ddocs=2,num_views_per_ddoc=3,items=2000 <br/> <br/> [error_logger:error] [2012-08-08 12:54:03] [<a href='mailto:ns_1@10.5.2.11'>ns_1@10.5.2.11</a>:error_logger:ale_error_logger_handler:log_report:72] <br/> =========================CRASH REPORT========================= <br/> &nbsp;&nbsp;crasher: <br/> &nbsp;&nbsp;&nbsp;&nbsp;initial call: ns_single_vbucket_mover:mover/6 <br/> &nbsp;&nbsp;&nbsp;&nbsp;pid: &lt;0.17517.2&gt; <br/> &nbsp;&nbsp;&nbsp;&nbsp;registered_name: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;exception error: {bulk_set_vbucket_state_failed, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{&#39;<a href='mailto:ns_1@10.5.2.19'>ns_1@10.5.2.19</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&#39;EXIT&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{{{normal, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server,call, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&lt;20615.11732.0&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{set_state,[],&quot;k&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;18,19,20,21,22,23,24,25,26,27,28,29,30,31,32, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;33,34,35,36,37,38,39,40,41,42,43,44,45,46,47, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;48,49,50,51,52,53,54,55,56,57,58,59,60,61,62, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;63,64,65,66,67,68,69,70,71,72,73,74,75,76,77, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;78,79,80,81,82,83,84,85,86,87,88,89,90,91,92, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;93,94,95,96,97,98,99,100,101,102,103,104,105, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;106,108,109,110,111,112,113,114,115,116,117, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;118,119,120,121,122,123,124,125,126,127]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;infinity]}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server,call, <br/> <br/> <br/> attached logs It seems like it&#39;s another issue, fill a separate bug <a href="http://www.couchbase.com/issues/browse/MB-6160">http://www.couchbase.com/issues/browse/MB-6160</a> . please, skip previous comment Date of First Response Mon, 21 May 2012 14:15:41 -0500 Rank 5106 Time In Status