Couchbase Server

dispatcher gives more priority to vbucket deletion even though disk write queue has > 1M items

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: 1.7 GA, 1.7.1, 1.7.1.1
  • Fix Version/s: 1.7.2
  • Component/s: None
  • Security Level: Public
  • Labels:
    None

Description

known issue with dispatcher scheduler.

during rebalance disk write queue size grows to more than 1M items and due to this bug the dispatcher on some nodes keeps doing the vbucket deletion on different vbuckets over and over again andthis causes issues and make rebalancing very slow

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Farshid Ghods added a comment -
i have monitored this node for the last 40 minutes and disk write queue size remained 1013k and vbucket-deletion task has been running and deleted more than 20 vbucket.s
Show
Farshid Ghods added a comment - i have monitored this node for the last 40 minutes and disk write queue size remained 1013k and vbucket-deletion task has been running and deleted more than 20 vbucket.s
Hide
Permalink
Farshid Ghods added a comment -
stats :

 accepting_conns: 1
 auth_cmds: 1088
 auth_errors: 0
 bucket_active_conns: 1
 bucket_conns: 20
 bytes_read: 63442439125
 bytes_written: 6603510477
 cas_badval: 0
 cas_hits: 0
 cas_misses: 0
 cmd_flush: 0
 cmd_get: 7531306
 cmd_set: 26812344
 conn_yields: 429139
 connection_structures: 92
 curr_connections: 28
 curr_items: 308258
 curr_items_tot: 1285805
 daemon_connections: 10
 decr_hits: 0
 decr_misses: 0
 delete_hits: 0
 delete_misses: 0
 ep_bg_fetched: 165
 ep_bg_load: 879409
 ep_bg_load_avg: 5329
 ep_bg_max_load: 140610
 ep_bg_max_wait: 33345
 ep_bg_min_load: 40
 ep_bg_min_wait: 39
 ep_bg_num_samples: 165
 ep_bg_wait: 81150
 ep_bg_wait_avg: 491
 ep_commit_num: 61737
 ep_commit_time: 0
 ep_commit_time_total: 24086
 ep_data_age: 1
 ep_data_age_highwat: 2144
 ep_db_cleaner_status: complete
 ep_db_strategy: multiMTVBDB
 ep_dbinit: 0
 ep_dbname: /mnt/default-data/default
 ep_dbshards: 4
 ep_diskqueue_drain: 24648
 ep_diskqueue_fill: 1058203
 ep_diskqueue_items: 1033566
 ep_diskqueue_memory: 90953808
 ep_diskqueue_pending: 1043187366
 ep_expired: 0
 ep_flush_all: false
 ep_flush_duration: 0
 ep_flush_duration_highwat: 775
 ep_flush_duration_total: 27480
 ep_flush_preempts: 0
 ep_flusher_state: running
 ep_flusher_todo: 0
 ep_io_num_read: 215273
 ep_io_num_write: 54748601
 ep_io_read_bytes: 222955770
 ep_io_write_bytes: 50648016228
 ep_item_begin_failed: 0
 ep_item_commit_failed: 0
 ep_item_flush_expired: 0
 ep_item_flush_failed: 0
 ep_items_rm_from_checkpoints: 49088066
 ep_kv_size: 1640026847
 ep_latency_arith_cmd: 0
 ep_latency_get_cmd: 7532472
 ep_latency_store_cmd: 26812344
 ep_max_data_size: 34902900736
 ep_max_txn_size: 1000
 ep_mem_high_wat: 26177175552
 ep_mem_low_wat: 20941740441
 ep_min_data_age: 0
 ep_num_active_non_resident: 0
 ep_num_checkpoint_remover_runs: 10016
 ep_num_eject_failures: 22978
 ep_num_eject_replicas: 6131491
 ep_num_expiry_pager_runs: 13
 ep_num_non_resident: 18008
 ep_num_not_my_vbuckets: 6263
 ep_num_pager_runs: 0
 ep_num_value_ejects: 6134897
 ep_onlineupdate: false
 ep_onlineupdate_revert_add: 0
 ep_onlineupdate_revert_delete: 0
 ep_onlineupdate_revert_update: 0
 ep_oom_errors: 0
 ep_overhead: 127924533
 ep_pending_ops: 0
 ep_pending_ops_max: 2
 ep_pending_ops_max_duration: 163767
 ep_pending_ops_total: 4
 ep_queue_age_cap: 900
 ep_queue_size: 1033566
 ep_storage_age: 1
 ep_storage_age_highwat: 2144
 ep_storage_type: featured
 ep_store_max_concurrency: 10
 ep_store_max_readers: 9
 ep_store_max_readwrite: 1
 ep_tap_bg_fetch_requeued: 0
 ep_tap_bg_fetched: 214608
 ep_tap_keepalive: 300
 ep_tmp_oom_errors: 0
 ep_too_old: 627317
 ep_too_young: 0
 ep_total_cache_size: 1393281051
 ep_total_del_items: 0
 ep_total_enqueued: 54832410
 ep_total_new_items: 26434634
 ep_total_persisted: 54748601
 ep_uncommitted_items: 0
 ep_value_size: 1458047980
 ep_vb_total: 22
 ep_vbucket_del: 158
 ep_vbucket_del_avg_walltime: 69102409
 ep_vbucket_del_fail: 0
 ep_vbucket_del_max_walltime: 134101378
 ep_vbucket_del_total_walltime: 10918180750
 ep_version: 1.7.0_56_g8d17d20
 ep_warmed_up: 0
 ep_warmup: true
 ep_warmup_dups: 0
 ep_warmup_oom: 0
 ep_warmup_thread: complete
 ep_warmup_time: 13369
 get_hits: 5784337
 get_misses: 1746969
 incr_hits: 0
 incr_misses: 0
 libevent: 2.0.11-stable
 limit_maxbytes: 67108864
 listen_disabled_num: 0
 mem_used: 1767951380
 pid: 6866
 pointer_size: 64
 rejected_conns: 0
 rusage_system: 3084.488768
 rusage_user: 2890.868668
 tap_checkpoint_end_received: 5657
 tap_checkpoint_end_sent: 5766
 tap_checkpoint_start_received: 5785
 tap_checkpoint_start_sent: 5996
 tap_connect_received: 248
 tap_mutation_received: 38494383
 tap_mutation_sent: 41704562
 tap_opaque_received: 408
 tap_opaque_sent: 653
 tap_vbucket_set_received: 96
 tap_vbucket_set_sent: 224
 threads: 4
 time: 1318884291
 total_connections: 1982
 uptime: 51910
 vb_active_curr_items: 308258
 vb_active_eject: 0
 vb_active_ht_memory: 3146688
 vb_active_itm_memory: 322166374
 vb_active_num: 2
 vb_active_num_non_resident: 0
 vb_active_ops_create: 0
 vb_active_ops_delete: 0
 vb_active_ops_reject: 0
 vb_active_ops_update: 0
 vb_active_perc_mem_resident: 100
 vb_active_queue_age: 1648144677000
 vb_active_queue_drain: 0
 vb_active_queue_fill: 342206
 vb_active_queue_memory: 30114128
 vb_active_queue_pending: 349263252
 vb_active_queue_size: 342206
 vb_dead_num: 2
 vb_pending_curr_items: 545398
 vb_pending_eject: 0
 vb_pending_ht_memory: 4327856
 vb_pending_itm_memory: 545353810
 vb_pending_num: 6
 vb_pending_num_non_resident: 0
 vb_pending_ops_create: 0
 vb_pending_ops_delete: 0
 vb_pending_ops_reject: 0
 vb_pending_ops_update: 0
 vb_pending_perc_mem_resident: 100
 vb_pending_queue_age: 2763075692000
 vb_pending_queue_drain: 0
 vb_pending_queue_fill: 545398
 vb_pending_queue_memory: 47995024
 vb_pending_queue_pending: 549576622
 vb_pending_queue_size: 545398
 vb_replica_curr_items: 432149
 vb_replica_eject: 102850
 vb_replica_ht_memory: 2586144
 vb_replica_itm_memory: 430624163
 vb_replica_num: 12
 vb_replica_num_non_resident: 18008
 vb_replica_ops_create: 1128
 vb_replica_ops_delete: 0
 vb_replica_ops_reject: 0
 vb_replica_ops_update: 23520
 vb_replica_perc_mem_resident: 95
 vb_replica_queue_age: 769715487000
 vb_replica_queue_drain: 24648
 vb_replica_queue_fill: 170599
 vb_replica_queue_memory: 12844656
 vb_replica_queue_pending: 144347492
 vb_replica_queue_size: 145962
 version: 1.4.4_461_gf99c147
Show
Farshid Ghods added a comment - stats :  accepting_conns: 1  auth_cmds: 1088  auth_errors: 0  bucket_active_conns: 1  bucket_conns: 20  bytes_read: 63442439125  bytes_written: 6603510477  cas_badval: 0  cas_hits: 0  cas_misses: 0  cmd_flush: 0  cmd_get: 7531306  cmd_set: 26812344  conn_yields: 429139  connection_structures: 92  curr_connections: 28  curr_items: 308258  curr_items_tot: 1285805  daemon_connections: 10  decr_hits: 0  decr_misses: 0  delete_hits: 0  delete_misses: 0  ep_bg_fetched: 165  ep_bg_load: 879409  ep_bg_load_avg: 5329  ep_bg_max_load: 140610  ep_bg_max_wait: 33345  ep_bg_min_load: 40  ep_bg_min_wait: 39  ep_bg_num_samples: 165  ep_bg_wait: 81150  ep_bg_wait_avg: 491  ep_commit_num: 61737  ep_commit_time: 0  ep_commit_time_total: 24086  ep_data_age: 1  ep_data_age_highwat: 2144  ep_db_cleaner_status: complete  ep_db_strategy: multiMTVBDB  ep_dbinit: 0  ep_dbname: /mnt/default-data/default  ep_dbshards: 4  ep_diskqueue_drain: 24648  ep_diskqueue_fill: 1058203  ep_diskqueue_items: 1033566  ep_diskqueue_memory: 90953808  ep_diskqueue_pending: 1043187366  ep_expired: 0  ep_flush_all: false  ep_flush_duration: 0  ep_flush_duration_highwat: 775  ep_flush_duration_total: 27480  ep_flush_preempts: 0  ep_flusher_state: running  ep_flusher_todo: 0  ep_io_num_read: 215273  ep_io_num_write: 54748601  ep_io_read_bytes: 222955770  ep_io_write_bytes: 50648016228  ep_item_begin_failed: 0  ep_item_commit_failed: 0  ep_item_flush_expired: 0  ep_item_flush_failed: 0  ep_items_rm_from_checkpoints: 49088066  ep_kv_size: 1640026847  ep_latency_arith_cmd: 0  ep_latency_get_cmd: 7532472  ep_latency_store_cmd: 26812344  ep_max_data_size: 34902900736  ep_max_txn_size: 1000  ep_mem_high_wat: 26177175552  ep_mem_low_wat: 20941740441  ep_min_data_age: 0  ep_num_active_non_resident: 0  ep_num_checkpoint_remover_runs: 10016  ep_num_eject_failures: 22978  ep_num_eject_replicas: 6131491  ep_num_expiry_pager_runs: 13  ep_num_non_resident: 18008  ep_num_not_my_vbuckets: 6263  ep_num_pager_runs: 0  ep_num_value_ejects: 6134897  ep_onlineupdate: false  ep_onlineupdate_revert_add: 0  ep_onlineupdate_revert_delete: 0  ep_onlineupdate_revert_update: 0  ep_oom_errors: 0  ep_overhead: 127924533  ep_pending_ops: 0  ep_pending_ops_max: 2  ep_pending_ops_max_duration: 163767  ep_pending_ops_total: 4  ep_queue_age_cap: 900  ep_queue_size: 1033566  ep_storage_age: 1  ep_storage_age_highwat: 2144  ep_storage_type: featured  ep_store_max_concurrency: 10  ep_store_max_readers: 9  ep_store_max_readwrite: 1  ep_tap_bg_fetch_requeued: 0  ep_tap_bg_fetched: 214608  ep_tap_keepalive: 300  ep_tmp_oom_errors: 0  ep_too_old: 627317  ep_too_young: 0  ep_total_cache_size: 1393281051  ep_total_del_items: 0  ep_total_enqueued: 54832410  ep_total_new_items: 26434634  ep_total_persisted: 54748601  ep_uncommitted_items: 0  ep_value_size: 1458047980  ep_vb_total: 22  ep_vbucket_del: 158  ep_vbucket_del_avg_walltime: 69102409  ep_vbucket_del_fail: 0  ep_vbucket_del_max_walltime: 134101378  ep_vbucket_del_total_walltime: 10918180750  ep_version: 1.7.0_56_g8d17d20  ep_warmed_up: 0  ep_warmup: true  ep_warmup_dups: 0  ep_warmup_oom: 0  ep_warmup_thread: complete  ep_warmup_time: 13369  get_hits: 5784337  get_misses: 1746969  incr_hits: 0  incr_misses: 0  libevent: 2.0.11-stable  limit_maxbytes: 67108864  listen_disabled_num: 0  mem_used: 1767951380  pid: 6866  pointer_size: 64  rejected_conns: 0  rusage_system: 3084.488768  rusage_user: 2890.868668  tap_checkpoint_end_received: 5657  tap_checkpoint_end_sent: 5766  tap_checkpoint_start_received: 5785  tap_checkpoint_start_sent: 5996  tap_connect_received: 248  tap_mutation_received: 38494383  tap_mutation_sent: 41704562  tap_opaque_received: 408  tap_opaque_sent: 653  tap_vbucket_set_received: 96  tap_vbucket_set_sent: 224  threads: 4  time: 1318884291  total_connections: 1982  uptime: 51910  vb_active_curr_items: 308258  vb_active_eject: 0  vb_active_ht_memory: 3146688  vb_active_itm_memory: 322166374  vb_active_num: 2  vb_active_num_non_resident: 0  vb_active_ops_create: 0  vb_active_ops_delete: 0  vb_active_ops_reject: 0  vb_active_ops_update: 0  vb_active_perc_mem_resident: 100  vb_active_queue_age: 1648144677000  vb_active_queue_drain: 0  vb_active_queue_fill: 342206  vb_active_queue_memory: 30114128  vb_active_queue_pending: 349263252  vb_active_queue_size: 342206  vb_dead_num: 2  vb_pending_curr_items: 545398  vb_pending_eject: 0  vb_pending_ht_memory: 4327856  vb_pending_itm_memory: 545353810  vb_pending_num: 6  vb_pending_num_non_resident: 0  vb_pending_ops_create: 0  vb_pending_ops_delete: 0  vb_pending_ops_reject: 0  vb_pending_ops_update: 0  vb_pending_perc_mem_resident: 100  vb_pending_queue_age: 2763075692000  vb_pending_queue_drain: 0  vb_pending_queue_fill: 545398  vb_pending_queue_memory: 47995024  vb_pending_queue_pending: 549576622  vb_pending_queue_size: 545398  vb_replica_curr_items: 432149  vb_replica_eject: 102850  vb_replica_ht_memory: 2586144  vb_replica_itm_memory: 430624163  vb_replica_num: 12  vb_replica_num_non_resident: 18008  vb_replica_ops_create: 1128  vb_replica_ops_delete: 0  vb_replica_ops_reject: 0  vb_replica_ops_update: 23520  vb_replica_perc_mem_resident: 95  vb_replica_queue_age: 769715487000  vb_replica_queue_drain: 24648  vb_replica_queue_fill: 170599  vb_replica_queue_memory: 12844656  vb_replica_queue_pending: 144347492  vb_replica_queue_size: 145962  version: 1.4.4_461_gf99c147
Hide
Permalink
Farshid Ghods added a comment -
its been an hour and half now and dispatcher is still removing items from vbucket
Show
Farshid Ghods added a comment - its been an hour and half now and dispatcher is still removing items from vbucket
Farshid Ghods made changes -
Field Original Value New Value
Assignee Chiyoung Seo [ chiyoung ]
Farshid Ghods made changes -
Fix Version/s 1.7.2 [ 10203 ]
Affects Version/s 1.7.1.1 [ 10210 ]
Affects Version/s 1.7.1 [ 10183 ]
Affects Version/s 1.7 GA [ 10111 ]
Hide
Permalink
Farshid Ghods added a comment -
fixed in 1.7.2
Show
Farshid Ghods added a comment - fixed in 1.7.2
Farshid Ghods made changes -
Status Open [ 1 ] Resolved [ 5 ]
Resolution Fixed [ 1 ]
Farshid Ghods made changes -
Status Resolved [ 5 ] Closed [ 6 ]

People

  • Assignee:
    Chiyoung Seo
    Reporter:
    Farshid Ghods
Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: