[MB-4828] rebalancing multiple nodes can hang if a bucket has less than 100k items due to a race condition in tap take-over

Thu, 10 Jan 2013 01:09:45 -0600

this was observed by one of our users which had 10 buckets . some buckets had less than 10k items and tap takeover got stuck.

tap stats :

   6179: vb_1014:cursor_checkpoint_id:eq_tapq:rebalance_1014: 1
  97312: eq_tapq:rebalance_1014:ack_log_size: 0
  97313: eq_tapq:rebalance_1014:ack_playback_size: 0
  97314: eq_tapq:rebalance_1014:ack_seqno: 10
  97315: eq_tapq:rebalance_1014:ack_window_full: false
  97316: eq_tapq:rebalance_1014:backfill_completed: false
  97317: eq_tapq:rebalance_1014:bg_backlog_size: 0
  97318: eq_tapq:rebalance_1014:bg_jobs_completed: 0
  97319: eq_tapq:rebalance_1014:bg_jobs_issued: 0
  97320: eq_tapq:rebalance_1014:bg_queued: 0
  97321: eq_tapq:rebalance_1014:bg_result_size: 0
  97322: eq_tapq:rebalance_1014:bg_results: 0
  97323: eq_tapq:rebalance_1014:bg_wait_for_results: false
  97324: eq_tapq:rebalance_1014:complete: false
  97325: eq_tapq:rebalance_1014:connected: true
  97326: eq_tapq:rebalance_1014:created: 1272317
  97327: eq_tapq:rebalance_1014:empty: false
  97328: eq_tapq:rebalance_1014:flags: 93 (ack,backfill,vblist,takeover,checkpoints)
  97329: eq_tapq:rebalance_1014:has_item: false
  97330: eq_tapq:rebalance_1014:has_queued_item: true
  97331: eq_tapq:rebalance_1014:idle: false
  97332: eq_tapq:rebalance_1014:num_tap_nack: 0
  97333: eq_tapq:rebalance_1014:num_tap_tmpfail_survivors: 0
  97334: eq_tapq:rebalance_1014:paused: 1
  97335: eq_tapq:rebalance_1014:pending_backfill: false
  97336: eq_tapq:rebalance_1014:pending_disconnect: false
  97337: eq_tapq:rebalance_1014:pending_disk_backfill: false
  97338: eq_tapq:rebalance_1014:qlen: 0
  97339: eq_tapq:rebalance_1014:qlen_high_pri: 0
  97340: eq_tapq:rebalance_1014:qlen_low_pri: 1
  97341: eq_tapq:rebalance_1014:queue_backfillremaining: 0
  97342: eq_tapq:rebalance_1014:queue_backoff: 0
  97343: eq_tapq:rebalance_1014:queue_drain: 0
  97344: eq_tapq:rebalance_1014:queue_fill: 0
  97345: eq_tapq:rebalance_1014:queue_itemondisk: 0
  97346: eq_tapq:rebalance_1014:queue_memory: 0
  97347: eq_tapq:rebalance_1014:rec_fetched: 5
  97348: eq_tapq:rebalance_1014:recv_ack_seqno: 8
  97349: eq_tapq:rebalance_1014:reserved: 1
  97350: eq_tapq:rebalance_1014:seqno_ack_requested: 9
  97351: eq_tapq:rebalance_1014:supports_ack: true
  97352: eq_tapq:rebalance_1014:suspended: false
  97353: eq_tapq:rebalance_1014:total_backlog_size: 10
  97354: eq_tapq:rebalance_1014:total_noops: 20036
  97355: eq_tapq:rebalance_1014:type: producer
  97356: eq_tapq:rebalance_1014:vb_filter: { 1014 }
  97357: eq_tapq:rebalance_1014:vb_filters: 1

Couchbase

[MB-4828] rebalancing multiple nodes can hang if a bucket has less than 100k items due to a race condition in tap take-over