http://www.couchbase.com/issues This file is an XML representation of an issue en-us 5.2.4 845 26-12-2012 http://www.couchbase.com/issues/browse/MB-4864 Couchbase Server Install couchbase server 1.8.0 release with hotfix mb-4738 on 4 nodes (12 GB RAM each) cluster in ec2. <br/> Load 38 million items to cluster. <br/> Resident ratio: 41% <br/> Data size on disk: 78GB <br/> Remove a node. <br/> Rebalance OK. <br/> Add another node in (not the removed node). <br/> Rebalance hang at around 80+% <br/> windows 2008 R2 64bit MB-4864

rebalancing can get stuck due to a bug detecting the backfill completion during vbucket takeover

Bug Major Closed Fixed Public Chiyoung Seo Thuan Nguyen 1.8.0-release-notes 1.8.1-release-notes Fri, 2 Mar 2012 16:55:38 -0600 Wed, 9 Jan 2013 22:53:00 -0600 Wed, 14 Mar 2012 20:38:38 -0500 1.8.0 1.8.1 1.8.2 couchbase-bucket 0 0 Do rebalance out 2 nodes and rebalance hang<br/> <br/> eq_tapq:rebalance_169:ack_log_size: 0<br/> &nbsp;eq_tapq:rebalance_169:ack_playback_size: 0<br/> &nbsp;eq_tapq:rebalance_169:ack_seqno: 41291<br/> &nbsp;eq_tapq:rebalance_169:ack_window_full: false<br/> &nbsp;eq_tapq:rebalance_169:backfill_completed: false<br/> &nbsp;eq_tapq:rebalance_169:bg_backlog_size: 0<br/> &nbsp;eq_tapq:rebalance_169:bg_jobs_completed: 37724<br/> &nbsp;eq_tapq:rebalance_169:bg_jobs_issued: 37724<br/> &nbsp;eq_tapq:rebalance_169:bg_queued: 37724<br/> &nbsp;eq_tapq:rebalance_169:bg_result_size: 0<br/> &nbsp;eq_tapq:rebalance_169:bg_results: 0<br/> &nbsp;eq_tapq:rebalance_169:bg_wait_for_results: false<br/> &nbsp;eq_tapq:rebalance_169:complete: false<br/> &nbsp;eq_tapq:rebalance_169:connected: true<br/> &nbsp;eq_tapq:rebalance_169:created: 1019807<br/> &nbsp;eq_tapq:rebalance_169:empty: false<br/> &nbsp;eq_tapq:rebalance_169:flags: 93 (ack,backfill,vblist,takeover,checkpoints)<br/> &nbsp;eq_tapq:rebalance_169:has_item: false<br/> &nbsp;eq_tapq:rebalance_169:has_queued_item: true<br/> &nbsp;eq_tapq:rebalance_169:idle: false<br/> &nbsp;eq_tapq:rebalance_169:num_tap_nack: 0<br/> &nbsp;eq_tapq:rebalance_169:num_tap_tmpfail_survivors: 0<br/> &nbsp;eq_tapq:rebalance_169:paused: 1<br/> &nbsp;eq_tapq:rebalance_169:pending_backfill: false<br/> &nbsp;eq_tapq:rebalance_169:pending_disconnect: false<br/> &nbsp;eq_tapq:rebalance_169:pending_disk_backfill: false<br/> &nbsp;eq_tapq:rebalance_169:qlen: 0<br/> &nbsp;eq_tapq:rebalance_169:qlen_high_pri: 0<br/> &nbsp;eq_tapq:rebalance_169:qlen_low_pri: 1<br/> &nbsp;eq_tapq:rebalance_169:queue_backfillremaining: 0<br/> &nbsp;eq_tapq:rebalance_169:queue_backoff: 0<br/> &nbsp;eq_tapq:rebalance_169:queue_drain: 41329<br/> &nbsp;eq_tapq:rebalance_169:queue_fill: 0<br/> &nbsp;eq_tapq:rebalance_169:queue_itemondisk: 0<br/> &nbsp;eq_tapq:rebalance_169:queue_memory: 0<br/> &nbsp;eq_tapq:rebalance_169:rec_fetched: 3609<br/> &nbsp;eq_tapq:rebalance_169:recv_ack_seqno: 41290<br/> &nbsp;eq_tapq:rebalance_169:reserved: 1<br/> &nbsp;eq_tapq:rebalance_169:seqno_ack_requested: 41290<br/> &nbsp;eq_tapq:rebalance_169:supports_ack: true<br/> &nbsp;eq_tapq:rebalance_169:suspended: false<br/> &nbsp;eq_tapq:rebalance_169:total_backlog_size: 1<br/> &nbsp;eq_tapq:rebalance_169:total_noops: 7935<br/> &nbsp;eq_tapq:rebalance_169:type: producer<br/> &nbsp;eq_tapq:rebalance_169:vb_filter: { 169 }<br/> &nbsp;eq_tapq:rebalance_169:vb_filters: 1<br/> Fixed in 1.8.1 branch This reason was that there is a bug in detecting the backfill completion for a vbucket takeover during rebalance. <br/> <br/> For example, the following are the TAP stats for vbucket 685 takeover: <br/> <br/> &nbsp;eq_tapq:rebalance_685:ack_window_full: false <br/> &nbsp;eq_tapq:rebalance_685:backfill_completed: false <br/> &nbsp;... <br/> &nbsp;eq_tapq:rebalance_685:pending_backfill: false <br/> &nbsp;eq_tapq:rebalance_685:pending_disconnect: false <br/> &nbsp;eq_tapq:rebalance_685:pending_disk_backfill: false <br/> &nbsp;eq_tapq:rebalance_685:queue_backfillremaining: 0 <br/> <br/> You can see that there are no items remaining for backfill, but &quot;backfill_completed&quot; flag is still false, which caused the takeover operation to be stuck. <br/> Date of First Response Wed, 14 Mar 2012 20:38:38 -0500 Rank 5608 Time In Status