[MB-4363] dispatcher gives more priority to vbucket deletion even though disk write queue has > 1M items Created: 17/Oct/11 Updated: 09/Jan/13 Resolved: 24/Oct/11 |
|
| Status: | Closed |
| Project: | Couchbase Server |
| Component/s: | None |
| Affects Version/s: | 1.7 GA, 1.7.1, 1.7.1.1 |
| Fix Version/s: | 1.7.2 |
| Security Level: | Public |
| Type: | Bug | Priority: | Major |
| Reporter: | Farshid Ghods | Assignee: | Chiyoung Seo |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | None | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Description |
|
known issue with dispatcher scheduler.
during rebalance disk write queue size grows to more than 1M items and due to this bug the dispatcher on some nodes keeps doing the vbucket deletion on different vbuckets over and over again andthis causes issues and make rebalancing very slow |
| Comments |
| Comment by Farshid Ghods [ 17/Oct/11 ] |
| i have monitored this node for the last 40 minutes and disk write queue size remained 1013k and vbucket-deletion task has been running and deleted more than 20 vbucket.s |
| Comment by Farshid Ghods [ 17/Oct/11 ] |
|
stats :
accepting_conns: 1 auth_cmds: 1088 auth_errors: 0 bucket_active_conns: 1 bucket_conns: 20 bytes_read: 63442439125 bytes_written: 6603510477 cas_badval: 0 cas_hits: 0 cas_misses: 0 cmd_flush: 0 cmd_get: 7531306 cmd_set: 26812344 conn_yields: 429139 connection_structures: 92 curr_connections: 28 curr_items: 308258 curr_items_tot: 1285805 daemon_connections: 10 decr_hits: 0 decr_misses: 0 delete_hits: 0 delete_misses: 0 ep_bg_fetched: 165 ep_bg_load: 879409 ep_bg_load_avg: 5329 ep_bg_max_load: 140610 ep_bg_max_wait: 33345 ep_bg_min_load: 40 ep_bg_min_wait: 39 ep_bg_num_samples: 165 ep_bg_wait: 81150 ep_bg_wait_avg: 491 ep_commit_num: 61737 ep_commit_time: 0 ep_commit_time_total: 24086 ep_data_age: 1 ep_data_age_highwat: 2144 ep_db_cleaner_status: complete ep_db_strategy: multiMTVBDB ep_dbinit: 0 ep_dbname: /mnt/default-data/default ep_dbshards: 4 ep_diskqueue_drain: 24648 ep_diskqueue_fill: 1058203 ep_diskqueue_items: 1033566 ep_diskqueue_memory: 90953808 ep_diskqueue_pending: 1043187366 ep_expired: 0 ep_flush_all: false ep_flush_duration: 0 ep_flush_duration_highwat: 775 ep_flush_duration_total: 27480 ep_flush_preempts: 0 ep_flusher_state: running ep_flusher_todo: 0 ep_io_num_read: 215273 ep_io_num_write: 54748601 ep_io_read_bytes: 222955770 ep_io_write_bytes: 50648016228 ep_item_begin_failed: 0 ep_item_commit_failed: 0 ep_item_flush_expired: 0 ep_item_flush_failed: 0 ep_items_rm_from_checkpoints: 49088066 ep_kv_size: 1640026847 ep_latency_arith_cmd: 0 ep_latency_get_cmd: 7532472 ep_latency_store_cmd: 26812344 ep_max_data_size: 34902900736 ep_max_txn_size: 1000 ep_mem_high_wat: 26177175552 ep_mem_low_wat: 20941740441 ep_min_data_age: 0 ep_num_active_non_resident: 0 ep_num_checkpoint_remover_runs: 10016 ep_num_eject_failures: 22978 ep_num_eject_replicas: 6131491 ep_num_expiry_pager_runs: 13 ep_num_non_resident: 18008 ep_num_not_my_vbuckets: 6263 ep_num_pager_runs: 0 ep_num_value_ejects: 6134897 ep_onlineupdate: false ep_onlineupdate_revert_add: 0 ep_onlineupdate_revert_delete: 0 ep_onlineupdate_revert_update: 0 ep_oom_errors: 0 ep_overhead: 127924533 ep_pending_ops: 0 ep_pending_ops_max: 2 ep_pending_ops_max_duration: 163767 ep_pending_ops_total: 4 ep_queue_age_cap: 900 ep_queue_size: 1033566 ep_storage_age: 1 ep_storage_age_highwat: 2144 ep_storage_type: featured ep_store_max_concurrency: 10 ep_store_max_readers: 9 ep_store_max_readwrite: 1 ep_tap_bg_fetch_requeued: 0 ep_tap_bg_fetched: 214608 ep_tap_keepalive: 300 ep_tmp_oom_errors: 0 ep_too_old: 627317 ep_too_young: 0 ep_total_cache_size: 1393281051 ep_total_del_items: 0 ep_total_enqueued: 54832410 ep_total_new_items: 26434634 ep_total_persisted: 54748601 ep_uncommitted_items: 0 ep_value_size: 1458047980 ep_vb_total: 22 ep_vbucket_del: 158 ep_vbucket_del_avg_walltime: 69102409 ep_vbucket_del_fail: 0 ep_vbucket_del_max_walltime: 134101378 ep_vbucket_del_total_walltime: 10918180750 ep_version: 1.7.0_56_g8d17d20 ep_warmed_up: 0 ep_warmup: true ep_warmup_dups: 0 ep_warmup_oom: 0 ep_warmup_thread: complete ep_warmup_time: 13369 get_hits: 5784337 get_misses: 1746969 incr_hits: 0 incr_misses: 0 libevent: 2.0.11-stable limit_maxbytes: 67108864 listen_disabled_num: 0 mem_used: 1767951380 pid: 6866 pointer_size: 64 rejected_conns: 0 rusage_system: 3084.488768 rusage_user: 2890.868668 tap_checkpoint_end_received: 5657 tap_checkpoint_end_sent: 5766 tap_checkpoint_start_received: 5785 tap_checkpoint_start_sent: 5996 tap_connect_received: 248 tap_mutation_received: 38494383 tap_mutation_sent: 41704562 tap_opaque_received: 408 tap_opaque_sent: 653 tap_vbucket_set_received: 96 tap_vbucket_set_sent: 224 threads: 4 time: 1318884291 total_connections: 1982 uptime: 51910 vb_active_curr_items: 308258 vb_active_eject: 0 vb_active_ht_memory: 3146688 vb_active_itm_memory: 322166374 vb_active_num: 2 vb_active_num_non_resident: 0 vb_active_ops_create: 0 vb_active_ops_delete: 0 vb_active_ops_reject: 0 vb_active_ops_update: 0 vb_active_perc_mem_resident: 100 vb_active_queue_age: 1648144677000 vb_active_queue_drain: 0 vb_active_queue_fill: 342206 vb_active_queue_memory: 30114128 vb_active_queue_pending: 349263252 vb_active_queue_size: 342206 vb_dead_num: 2 vb_pending_curr_items: 545398 vb_pending_eject: 0 vb_pending_ht_memory: 4327856 vb_pending_itm_memory: 545353810 vb_pending_num: 6 vb_pending_num_non_resident: 0 vb_pending_ops_create: 0 vb_pending_ops_delete: 0 vb_pending_ops_reject: 0 vb_pending_ops_update: 0 vb_pending_perc_mem_resident: 100 vb_pending_queue_age: 2763075692000 vb_pending_queue_drain: 0 vb_pending_queue_fill: 545398 vb_pending_queue_memory: 47995024 vb_pending_queue_pending: 549576622 vb_pending_queue_size: 545398 vb_replica_curr_items: 432149 vb_replica_eject: 102850 vb_replica_ht_memory: 2586144 vb_replica_itm_memory: 430624163 vb_replica_num: 12 vb_replica_num_non_resident: 18008 vb_replica_ops_create: 1128 vb_replica_ops_delete: 0 vb_replica_ops_reject: 0 vb_replica_ops_update: 23520 vb_replica_perc_mem_resident: 95 vb_replica_queue_age: 769715487000 vb_replica_queue_drain: 24648 vb_replica_queue_fill: 170599 vb_replica_queue_memory: 12844656 vb_replica_queue_pending: 144347492 vb_replica_queue_size: 145962 version: 1.4.4_461_gf99c147 |
| Comment by Farshid Ghods [ 17/Oct/11 ] |
| its been an hour and half now and dispatcher is still removing items from vbucket |
| Comment by Farshid Ghods [ 24/Oct/11 ] |
| fixed in 1.7.2 |