http://www.couchbase.com/issues This file is an XML representation of an issue en-us 5.2.4 845 26-12-2012 http://www.couchbase.com/issues/browse/MB-6992 Couchbase Server 1 node went down while loading data on 22 node cluster. (possibly related to xen-hypervisor as it could not ping gateway and network interface needed to be restarted) <br/> While node was down I tried to fail it over and rebalance. <br/> However, rebalance never completes and looks like there is no rebalance activity occuring on tap. <br/> <br/> <br/> Some activity seen in logs at time of node down: <br/> <br/> 10.3.3.59 sees .60 nodedown : <br/> <br/> [user:warn,2012-10-22T11:06:38.896,<a href='mailto:ns_1@10.3.3.59'>ns_1@10.3.3.59</a>:ns_node_disco:ns_node_disco:handle_info:168]Node &#39;<a href='mailto:ns_1@10.3.3.59'>ns_1@10.3.3.59</a>&#39; saw that node &#39;<a href='mailto:ns_1@10.3.3.60'>ns_1@10.3.3.60</a>&#39; went down. <br/> <br/> at the same time stamp node .60 shows: <br/> <br/> [ns_server:error,2012-10-22T11:06:00.350,<a href='mailto:ns_1@10.3.3.60'>ns_1@10.3.3.60</a>:&lt;0.12281.36&gt;:ns_janitor:cleanup_with_states:84]Bucket &quot;default&quot; not yet ready on [&#39;<a href='mailto:ns_1@10.3.2.84'>ns_1@10.3.2.84</a>&#39;,&#39;<a href='mailto:ns_1@10.3.2'>ns_1@10.3.2</a>. <br/> 85&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.3.2.110'>ns_1@10.3.2.110</a>&#39;,&#39;<a href='mailto:ns_1@10.3.2.111'>ns_1@10.3.2.111</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.3.2.112'>ns_1@10.3.2.112</a>&#39;,&#39;<a href='mailto:ns_1@10.3.2.113'>ns_1@10.3.2.113</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.3.2.114'>ns_1@10.3.2.114</a>&#39;,&#39;<a href='mailto:ns_1@10.3.2.115'>ns_1@10.3.2.115</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.3.3.59'>ns_1@10.3.3.59</a>&#39;,&#39;<a href='mailto:ns_1@10.3.3.62'>ns_1@10.3.3.62</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;,&#39;<a href='mailto:ns_1@10.3.3.66'>ns_1@10.3.3.66</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.3.3.69'>ns_1@10.3.3.69</a>&#39;,&#39;<a href='mailto:ns_1@10.3.3.70'>ns_1@10.3.3.70</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.3.121.90'>ns_1@10.3.121.90</a>&#39;,&#39;<a href='mailto:ns_1@10.3.121.91'>ns_1@10.3.121.91</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.3.2.107'>ns_1@10.3.2.107</a>&#39;,&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.3.2.109'>ns_1@10.3.2.109</a>&#39;] <br/> [ns_server:debug,2012-10-22T11:06:07.388,<a href='mailto:ns_1@10.3.3.60'>ns_1@10.3.3.60</a>:&lt;0.12508.36&gt;:janitor_agent:new_style_query_vbucket_states_loop:116]Exception from query_vbucket_states of &quot;defau <br/> lt&quot;:&#39;<a href='mailto:ns_1@10.3.2.85'>ns_1@10.3.2.85</a>&#39; <br/> {&#39;EXIT&#39;,{{nodedown,&#39;<a href='mailto:ns_1@10.3.2.85'>ns_1@10.3.2.85</a>&#39;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server,call, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{&#39;janitor_agent-default&#39;,&#39;<a href='mailto:ns_1@10.3.2.85'>ns_1@10.3.2.85</a>&#39;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;query_vbucket_states,infinity]}}} <br/> build 10.3.3.59 MB-6992

rebalance hangs after failing over disconnected node

Bug Blocker Closed Fixed Public Chiyoung Seo Tommie McAfee system-test Tue, 23 Oct 2012 15:39:58 -0500 Thu, 10 Jan 2013 01:55:22 -0600 Fri, 26 Oct 2012 19:18:34 -0500 2.0 2.0 couchbase-bucket ns_server 0 1 Also, when Jin took a look he noticed modes nodes are at 100% rebalance status, however 10.3.2.112 is still at 63% and found the following CRASH report was seen on that host. <br/> <br/> [couchdb:error,2012-10-21T23:07:11.144,<a href='mailto:ns_1@127.0.0.1'>ns_1@127.0.0.1</a>:couch_view:couch_log:error:42]Exit on non-updater process: config_change <br/> [couchdb:error,2012-10-21T23:07:11.144,<a href='mailto:ns_1@127.0.0.1'>ns_1@127.0.0.1</a>:couch_set_view:couch_log:error:42]Exit on non-updater process: config_change <br/> [error_logger:error,2012-10-21T23:07:11.144,<a href='mailto:ns_1@127.0.0.1'>ns_1@127.0.0.1</a>:error_logger:ale_error_logger_handler:log_report:72] <br/> =========================CRASH REPORT========================= <br/> &nbsp;&nbsp;crasher: <br/> &nbsp;&nbsp;&nbsp;&nbsp;initial call: couch_server:init/1 <br/> &nbsp;&nbsp;&nbsp;&nbsp;pid: &lt;0.216.0&gt; <br/> &nbsp;&nbsp;&nbsp;&nbsp;registered_name: couch_server <br/> &nbsp;&nbsp;&nbsp;&nbsp;exception exit: {function_clause, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{couch_server,&#39;-terminate/2-fun-0-&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{&lt;&lt;&quot;_replicator&quot;&gt;&gt;,&lt;0.444.0&gt;}]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{lists,foreach,2}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server,terminate,6}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{proc_lib,init_p_do_apply,3}]} <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in function gen_server:terminate/6 <br/> &nbsp;&nbsp;&nbsp;&nbsp;ancestors: [couch_primary_services,couch_server_sup,cb_couch_sup, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ns_server_cluster_sup,&lt;0.60.0&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;messages: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;links: [&lt;0.230.0&gt;,&lt;0.444.0&gt;,&lt;0.211.0&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;dictionary: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;trap_exit: true <br/> &nbsp;&nbsp;&nbsp;&nbsp;status: running <br/> &nbsp;&nbsp;&nbsp;&nbsp;heap_size: 2584 <br/> &nbsp;&nbsp;&nbsp;&nbsp;stack_size: 24 <br/> &nbsp;&nbsp;&nbsp;&nbsp;reductions: 4530 <br/> &nbsp;&nbsp;neighbours: Alk,<br/> <br/> Replication building is completed on 10.3.2.112 but ns_server has not killed the tap stream. Please take a look at this and let me know if there is anything else I can look at.<br/> <br/> Mike-Wiederholds-MacBook-Pro:ep-engine mikewied$ management/cbstats 10.3.2.112:11210 tap | egrep replication_building<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:ack_log_size: 0<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:ack_seqno: 359937<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:ack_window_full: false<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:backfill_completed: true<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:backfill_start_timestamp: 1350926553<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:bg_jobs_completed: 166602<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:bg_jobs_issued: 166602<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:bg_result_size: 0<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:connected: true<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:created: 39985<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:flags: 85 (ack,backfill,vblist,checkpoints)<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:has_queued_item: false<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:idle: true<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:paused: 1<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:pending_backfill: false<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:pending_disconnect: false<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:pending_disk_backfill: false<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:qlen: 0<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:qlen_high_pri: 0<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:qlen_low_pri: 0<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:queue_backfillremaining: 0<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:queue_backoff: 0<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:queue_drain: 359877<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:queue_fill: 0<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:queue_itemondisk: 0<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:queue_memory: 0<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:rec_fetched: 193330<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:recv_ack_seqno: 359936<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:reserved: 1<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:seqno_ack_requested: 359936<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:supports_ack: true<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:suspended: false<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:total_backlog_size: 0<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:total_noops: 3963<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:type: producer<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:vb_filter: { 494 }<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.2.108'>ns_1@10.3.2.108</a>&#39;:vb_filters: 1<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:ack_log_size: 0<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:ack_seqno: 97912<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:ack_window_full: false<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:backfill_completed: true<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:backfill_start_timestamp: 0<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:bg_jobs_completed: 0<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:bg_jobs_issued: 0<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:bg_result_size: 0<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:connected: true<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:created: 39985<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:flags: 85 (ack,backfill,vblist,checkpoints)<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:has_queued_item: false<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:idle: true<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:paused: 1<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:pending_backfill: false<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:pending_disconnect: false<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:pending_disk_backfill: false<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:qlen: 0<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:qlen_high_pri: 0<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:qlen_low_pri: 0<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:queue_backfillremaining: 0<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:queue_backoff: 0<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:queue_drain: 97854<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:queue_fill: 0<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:queue_itemondisk: 0<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:queue_memory: 0<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:rec_fetched: 97909<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:recv_ack_seqno: 97911<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:reserved: 1<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:seqno_ack_requested: 97911<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:supports_ack: true<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:suspended: false<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:total_backlog_size: 0<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:total_noops: 3964<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:type: producer<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:vb_filter: { 494 }<br/> &nbsp;eq_tapq:replication_building_494_&#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;:vb_filters: 1 All logs here: <br/> <a href="https://docs.google.com/folder/d/0B48zH0LlqjKrNm9UYlhuNzZjSnM/edit">https://docs.google.com/folder/d/0B48zH0LlqjKrNm9UYlhuNzZjSnM/edit</a> <br/> <br/> This behavior was noticed during the 1.8.1 use case testing in 2.0 build 1870 <br/> <a href="http://hub.internal.couchbase.com/confluence/display/QA/2.0+System+Test+Specs#2.0SystemTestSpecs-SystemTest%3A1.8.1usecase%28PlumWorkloadv1.0%29">http://hub.internal.couchbase.com/confluence/display/QA/2.0+System+Test+Specs#2.0SystemTestSpecs-SystemTest%3A1.8.1usecase%28PlumWorkloadv1.0%29</a> BTW, I don&#39;t have a convenient way to download all logs from google drive. We&#39;re stuck waiting for checkpoint 67 to be persisted on vbucket 494 on node .65.<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&lt;17848.6247.34&gt;,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{registered_name,[]},<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{status,waiting},<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{initial_call,{proc_lib,init_p,3}},<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{backtrace,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&lt;&lt;&quot;Program counter: 0x00007f5e10ee9e48 (gen:do_call/4 + 576)&quot;&gt;&gt;,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;CP: 0x0000000000000000 (invalid)&quot;&gt;&gt;,&lt;&lt;&quot;arity = 0&quot;&gt;&gt;,&lt;&lt;&gt;&gt;,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;0x00007f5e0b871858 Return addr 0x00007f5e10f68870 (gen_server:call/3 + 128)&quot;&gt;&gt;,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(0) #Ref&lt;0.0.50.98207&gt;&quot;&gt;&gt;,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(1) &#39;<a href='mailto:ns_1@10.3.3.65'>ns_1@10.3.3.65</a>&#39;&quot;&gt;&gt;,&lt;&lt;&quot;y(2) []&quot;&gt;&gt;,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(3) infinity&quot;&gt;&gt;,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(4) {wait_for_checkpoint_persistence,494,67}&quot;&gt;&gt;,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(5) &#39;$gen_call&#39;&quot;&gt;&gt;,&lt;&lt;&quot;y(6) &lt;0.2607.0&gt;&quot;&gt;&gt;,&lt;&lt;&gt;&gt;,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;x00007f5e0b871898 Return addr 0x00007f5e06f9e228 (janitor_agent:do_wait_checkpoint_persist&quot;&gt;&gt;,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(0) infinity&quot;&gt;&gt;,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(1) {wait_for_checkpoint_persistence,494,67}&quot;&gt;&gt;,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(2) &#39;ns_memcached-default&#39;&quot;&gt;&gt;,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(3) Catch 0x00007f5e10f68870 (gen_server:call/3 + 128)&quot;&gt;&gt;,<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&gt;&gt;,<br/> <br/> (thats janitor_agent child on .65 doing that call)<br/> <br/> And stats indicate that it&#39;s actually persisted:<br/> <br/> &nbsp;{&lt;&lt;&quot;vb_494:persisted_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;92&quot;&gt;&gt;},<br/> &nbsp;{&lt;&lt;&quot;vb_494:checkpoint_extension&quot;&gt;&gt;,&lt;&lt;&quot;false&quot;&gt;&gt;},<br/> &nbsp;{&lt;&lt;&quot;vb_494:num_items_for_persistence&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;},<br/> &nbsp;{&lt;&lt;&quot;vb_494:num_checkpoints&quot;&gt;&gt;,&lt;&lt;&quot;1&quot;&gt;&gt;},<br/> &nbsp;{&lt;&lt;&quot;vb_494:num_open_checkpoint_items&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;},<br/> &nbsp;{&lt;&lt;&quot;vb_494:num_checkpoint_items&quot;&gt;&gt;,&lt;&lt;&quot;1&quot;&gt;&gt;},<br/> &nbsp;{&lt;&lt;&quot;vb_494:num_tap_cursors&quot;&gt;&gt;,&lt;&lt;&quot;0&quot;&gt;&gt;},<br/> &nbsp;{&lt;&lt;&quot;vb_494:last_closed_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;92&quot;&gt;&gt;},<br/> &nbsp;{&lt;&lt;&quot;vb_494:open_checkpoint_id&quot;&gt;&gt;,&lt;&lt;&quot;93&quot;&gt;&gt;},<br/> Chiyoung, <br/> <br/> This is an issue with the new checkpoint persistence command you added. I think we should add a stat for checkpoint persistence commands in progress. For example a stat that is incremented when we receive a checkpoint persistence request and is decremented when the request completes. I dumped connections stats from memcached layer in 10.3.3.65 and found one connection that is still in ewouldblock state and corresponds to vb_checkpoint_persistence command <br/> <br/> STAT conn 0x58cf080 <br/> STAT socket 83 <br/> STAT protocol binary <br/> STAT transport TCP <br/> STAT nevents 20 <br/> STAT sasl_conn 0xcbaf5390 <br/> STAT state conn_nread <br/> STAT substate bin_reading_packet <br/> STAT registered_in_libevent 0 <br/> STAT ev_flags 12 <br/> STAT which 2 <br/> STAT rbuf 0x58d0000 <br/> STAT rcurr 0x58d0020 <br/> STAT rsize 2048 <br/> STAT rbytes 0 <br/> STAT wbuf 0x58e7800 <br/> STAT wcurr 0x58fb000 <br/> STAT wsize 2048 <br/> STAT wbytes 24 <br/> STAT write_and_go 0x4104f0 <br/> STAT write_and_free (nil) <br/> STAT ritem 0x58d0020 <br/> STAT rlbytes 0 <br/> STAT item (nil) <br/> STAT store_op 0 <br/> STAT sbytes 0 <br/> STAT iov 0x58e0800 <br/> STAT iovsize 400 <br/> STAT iovused 0 <br/> STAT msglist 0x58d5440 <br/> STAT msgsize 10 <br/> STAT msgused 1 <br/> STAT msgcurr 0 <br/> STAT msgbytes 0 <br/> STAT ilist 0x58d9100 <br/> STAT isize 200 <br/> STAT icurr 0x58d9100 <br/> STAT ileft 0 <br/> STAT suffixlist 0x58539a0 <br/> STAT suffixsize 20 <br/> STAT suffixcurr 0x58539a0 <br/> STAT suffixleft 0 <br/> STAT noreply 0 <br/> STAT refcount 1 <br/> STAT dynamic_buffer.buffer (nil) <br/> STAT dynamic_buffer.size 2048 <br/> STAT dynamic_buffer.offset 24 <br/> STAT engine_storage 0xcd6fb0a0 <br/> STAT cas 0 <br/> STAT cmd 177 <br/> STAT opaque 0 <br/> STAT keylen 0 <br/> STAT list_state 0 <br/> STAT next (nil) <br/> STAT thread 0x10c55f0 <br/> STAT aiostat 0 <br/> STAT ewouldblock 1 <br/> STAT tap_iterator (nil) <br/> <br/> I&#39;m further debugging it now. <br/> Tommie, <br/> <br/> As we saw, the rebalance out 10.3.3.65 was successful. While debugging this issue, I had some issues in .65 while doing GETs for non-resident items, which puts my connection in ewouldblock state in memcached layer. I will continue to debug this issue. <br/> <br/> Please update the bug if you see the same rebalance hung issue again. Integrated in github-ep-engine-2-0 #451 (See [<a href="http://qa.hq.northscale.net/job/github-ep-engine-2-0/451/">http://qa.hq.northscale.net/job/github-ep-engine-2-0/451/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.couchbase.com/issues/browse/MB-6992" title="rebalance hangs after failing over disconnected node"><strike>MB-6992</strike></a> Add more informative logs to checkpoint prioritization (Revision 3c719d47ca41285bbcbc61817f719180448f1042)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> Chiyoung Seo : <br/> Files : <br/> * src/ep.cc<br/> <a href="http://review.couchbase.org/#/c/22022/">http://review.couchbase.org/#/c/22022/</a> Integrated in github-ep-engine-2-0 #452 (See [<a href="http://qa.hq.northscale.net/job/github-ep-engine-2-0/452/">http://qa.hq.northscale.net/job/github-ep-engine-2-0/452/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.couchbase.com/issues/browse/MB-6992" title="rebalance hangs after failing over disconnected node"><strike>MB-6992</strike></a> Control the flusher execution by the transaction size (Revision b327be09a1f971145fda5c249b4fa7a8304b1920)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> Chiyoung Seo : <br/> Files : <br/> * tests/ep_testsuite.cc<br/> * src/ep.cc<br/> * src/ep.hh<br/> * src/flusher.hh<br/> * src/flusher.cc<br/> Date of First Response Tue, 23 Oct 2012 16:22:45 -0500 Rank 3685 Time In Status