[MB-4363] dispatcher gives more priority to vbucket deletion even though disk write queue has > 1M items Created: 17/Oct/11  Updated: 09/Jan/13  Resolved: 24/Oct/11

Status: Closed
Project: Couchbase Server
Component/s: None
Affects Version/s: 1.7 GA, 1.7.1, 1.7.1.1
Fix Version/s: 1.7.2
Security Level: Public

Type: Bug Priority: Major
Reporter: Farshid Ghods (Inactive) Assignee: Chiyoung Seo
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   
known issue with dispatcher scheduler.

during rebalance disk write queue size grows to more than 1M items and due to this bug the dispatcher on some nodes keeps doing the vbucket deletion on different vbuckets over and over again andthis causes issues and make rebalancing very slow



 Comments   
Comment by Farshid Ghods (Inactive) [ 17/Oct/11 ]
i have monitored this node for the last 40 minutes and disk write queue size remained 1013k and vbucket-deletion task has been running and deleted more than 20 vbucket.s
Comment by Farshid Ghods (Inactive) [ 17/Oct/11 ]
stats :

 accepting_conns: 1
 auth_cmds: 1088
 auth_errors: 0
 bucket_active_conns: 1
 bucket_conns: 20
 bytes_read: 63442439125
 bytes_written: 6603510477
 cas_badval: 0
 cas_hits: 0
 cas_misses: 0
 cmd_flush: 0
 cmd_get: 7531306
 cmd_set: 26812344
 conn_yields: 429139
 connection_structures: 92
 curr_connections: 28
 curr_items: 308258
 curr_items_tot: 1285805
 daemon_connections: 10
 decr_hits: 0
 decr_misses: 0
 delete_hits: 0
 delete_misses: 0
 ep_bg_fetched: 165
 ep_bg_load: 879409
 ep_bg_load_avg: 5329
 ep_bg_max_load: 140610
 ep_bg_max_wait: 33345
 ep_bg_min_load: 40
 ep_bg_min_wait: 39
 ep_bg_num_samples: 165
 ep_bg_wait: 81150
 ep_bg_wait_avg: 491
 ep_commit_num: 61737
 ep_commit_time: 0
 ep_commit_time_total: 24086
 ep_data_age: 1
 ep_data_age_highwat: 2144
 ep_db_cleaner_status: complete
 ep_db_strategy: multiMTVBDB
 ep_dbinit: 0
 ep_dbname: /mnt/default-data/default
 ep_dbshards: 4
 ep_diskqueue_drain: 24648
 ep_diskqueue_fill: 1058203
 ep_diskqueue_items: 1033566
 ep_diskqueue_memory: 90953808
 ep_diskqueue_pending: 1043187366
 ep_expired: 0
 ep_flush_all: false
 ep_flush_duration: 0
 ep_flush_duration_highwat: 775
 ep_flush_duration_total: 27480
 ep_flush_preempts: 0
 ep_flusher_state: running
 ep_flusher_todo: 0
 ep_io_num_read: 215273
 ep_io_num_write: 54748601
 ep_io_read_bytes: 222955770
 ep_io_write_bytes: 50648016228
 ep_item_begin_failed: 0
 ep_item_commit_failed: 0
 ep_item_flush_expired: 0
 ep_item_flush_failed: 0
 ep_items_rm_from_checkpoints: 49088066
 ep_kv_size: 1640026847
 ep_latency_arith_cmd: 0
 ep_latency_get_cmd: 7532472
 ep_latency_store_cmd: 26812344
 ep_max_data_size: 34902900736
 ep_max_txn_size: 1000
 ep_mem_high_wat: 26177175552
 ep_mem_low_wat: 20941740441
 ep_min_data_age: 0
 ep_num_active_non_resident: 0
 ep_num_checkpoint_remover_runs: 10016
 ep_num_eject_failures: 22978
 ep_num_eject_replicas: 6131491
 ep_num_expiry_pager_runs: 13
 ep_num_non_resident: 18008
 ep_num_not_my_vbuckets: 6263
 ep_num_pager_runs: 0
 ep_num_value_ejects: 6134897
 ep_onlineupdate: false
 ep_onlineupdate_revert_add: 0
 ep_onlineupdate_revert_delete: 0
 ep_onlineupdate_revert_update: 0
 ep_oom_errors: 0
 ep_overhead: 127924533
 ep_pending_ops: 0
 ep_pending_ops_max: 2
 ep_pending_ops_max_duration: 163767
 ep_pending_ops_total: 4
 ep_queue_age_cap: 900
 ep_queue_size: 1033566
 ep_storage_age: 1
 ep_storage_age_highwat: 2144
 ep_storage_type: featured
 ep_store_max_concurrency: 10
 ep_store_max_readers: 9
 ep_store_max_readwrite: 1
 ep_tap_bg_fetch_requeued: 0
 ep_tap_bg_fetched: 214608
 ep_tap_keepalive: 300
 ep_tmp_oom_errors: 0
 ep_too_old: 627317
 ep_too_young: 0
 ep_total_cache_size: 1393281051
 ep_total_del_items: 0
 ep_total_enqueued: 54832410
 ep_total_new_items: 26434634
 ep_total_persisted: 54748601
 ep_uncommitted_items: 0
 ep_value_size: 1458047980
 ep_vb_total: 22
 ep_vbucket_del: 158
 ep_vbucket_del_avg_walltime: 69102409
 ep_vbucket_del_fail: 0
 ep_vbucket_del_max_walltime: 134101378
 ep_vbucket_del_total_walltime: 10918180750
 ep_version: 1.7.0_56_g8d17d20
 ep_warmed_up: 0
 ep_warmup: true
 ep_warmup_dups: 0
 ep_warmup_oom: 0
 ep_warmup_thread: complete
 ep_warmup_time: 13369
 get_hits: 5784337
 get_misses: 1746969
 incr_hits: 0
 incr_misses: 0
 libevent: 2.0.11-stable
 limit_maxbytes: 67108864
 listen_disabled_num: 0
 mem_used: 1767951380
 pid: 6866
 pointer_size: 64
 rejected_conns: 0
 rusage_system: 3084.488768
 rusage_user: 2890.868668
 tap_checkpoint_end_received: 5657
 tap_checkpoint_end_sent: 5766
 tap_checkpoint_start_received: 5785
 tap_checkpoint_start_sent: 5996
 tap_connect_received: 248
 tap_mutation_received: 38494383
 tap_mutation_sent: 41704562
 tap_opaque_received: 408
 tap_opaque_sent: 653
 tap_vbucket_set_received: 96
 tap_vbucket_set_sent: 224
 threads: 4
 time: 1318884291
 total_connections: 1982
 uptime: 51910
 vb_active_curr_items: 308258
 vb_active_eject: 0
 vb_active_ht_memory: 3146688
 vb_active_itm_memory: 322166374
 vb_active_num: 2
 vb_active_num_non_resident: 0
 vb_active_ops_create: 0
 vb_active_ops_delete: 0
 vb_active_ops_reject: 0
 vb_active_ops_update: 0
 vb_active_perc_mem_resident: 100
 vb_active_queue_age: 1648144677000
 vb_active_queue_drain: 0
 vb_active_queue_fill: 342206
 vb_active_queue_memory: 30114128
 vb_active_queue_pending: 349263252
 vb_active_queue_size: 342206
 vb_dead_num: 2
 vb_pending_curr_items: 545398
 vb_pending_eject: 0
 vb_pending_ht_memory: 4327856
 vb_pending_itm_memory: 545353810
 vb_pending_num: 6
 vb_pending_num_non_resident: 0
 vb_pending_ops_create: 0
 vb_pending_ops_delete: 0
 vb_pending_ops_reject: 0
 vb_pending_ops_update: 0
 vb_pending_perc_mem_resident: 100
 vb_pending_queue_age: 2763075692000
 vb_pending_queue_drain: 0
 vb_pending_queue_fill: 545398
 vb_pending_queue_memory: 47995024
 vb_pending_queue_pending: 549576622
 vb_pending_queue_size: 545398
 vb_replica_curr_items: 432149
 vb_replica_eject: 102850
 vb_replica_ht_memory: 2586144
 vb_replica_itm_memory: 430624163
 vb_replica_num: 12
 vb_replica_num_non_resident: 18008
 vb_replica_ops_create: 1128
 vb_replica_ops_delete: 0
 vb_replica_ops_reject: 0
 vb_replica_ops_update: 23520
 vb_replica_perc_mem_resident: 95
 vb_replica_queue_age: 769715487000
 vb_replica_queue_drain: 24648
 vb_replica_queue_fill: 170599
 vb_replica_queue_memory: 12844656
 vb_replica_queue_pending: 144347492
 vb_replica_queue_size: 145962
 version: 1.4.4_461_gf99c147
Comment by Farshid Ghods (Inactive) [ 17/Oct/11 ]
its been an hour and half now and dispatcher is still removing items from vbucket
Comment by Farshid Ghods (Inactive) [ 24/Oct/11 ]
fixed in 1.7.2
Generated at Tue Sep 16 17:37:53 CDT 2014 using JIRA 5.2.4#845-sha1:c9f4cc41abe72fb236945343a1f485c2c844dac9.