vb_active_num_non_resident overflow : active resident ratio shows negative number

Aleksey Kondratenko added a comment - 31/May/12 8:22 PM I need raw memcached stats from this node. cbstats node:11210 raw '' (two single quotes without any space between)

Farshid Ghods added a comment - 31/May/12 8:24 PM safari or chrome crash after a while when you switch to http://ec2-184-73-111-82.compute-1.amazonaws.com:8091/index.html#sec=analytics&statsBucket=%2Fpools%2Fdefault%2Fbuckets%2Fdefault&statsStatName=vb_active_resident_items_ratio view

Farshid Ghods added a comment - 31/May/12 8:27 PM posting stats from the correct node: [ root@ip-10-32-61-83 ~]# /opt/couchbase/bin/cbstats 10.80.105.246:11210 raw ""  accepting_conns: 1  auth_cmds: 673  auth_errors: 0  bucket_active_conns: 1  bucket_conns: 58  bytes_read: 12354428524  bytes_written: 6872270860  cas_badval: 0  cas_hits: 0  cas_misses: 0  cmd_flush: 0  cmd_get: 10797681  cmd_set: 3048133  conn_yields: 547080  connection_structures: 54  curr_connections: 52  curr_conns_on_port_11209: 22  curr_conns_on_port_11210: 28  curr_items: 3311247  curr_items_tot: 6621695  daemon_connections: 4  decr_hits: 0  decr_misses: 0  delete_hits: 282117  delete_misses: 28910  ep_bg_fetched: 9  ep_bg_load: 60395  ep_bg_load_avg: 6710  ep_bg_max_load: 12509  ep_bg_max_wait: 132  ep_bg_min_load: 203  ep_bg_min_wait: 66  ep_bg_num_samples: 9  ep_bg_wait: 759  ep_bg_wait_avg: 84  ep_commit_num: 15546  ep_commit_time: 0  ep_commit_time_total: 3066  ep_data_age: 2  ep_data_age_highwat: 346  ep_db_cleaner_status: complete  ep_db_strategy: multiMTVBDB  ep_dbinit: 0  ep_dbname: /data/default-data/default  ep_dbshards: 4  ep_diskqueue_drain: 10125692  ep_diskqueue_fill: 10054951  ep_diskqueue_items: 54  ep_diskqueue_memory: 2160  ep_diskqueue_pending: 3510  ep_exp_pager_stime: 3600  ep_expired: 0  ep_flush_all: false  ep_flush_duration: 0  ep_flush_duration_highwat: 176  ep_flush_duration_total: 5319  ep_flush_preempts: 0  ep_flusher_deduplication: 5031048  ep_flusher_state: running  ep_flusher_todo: 0  ep_inconsistent_slave_chk: 0  ep_io_num_read: 39058  ep_io_num_write: 10572550  ep_io_read_bytes: 22486421  ep_io_write_bytes: 5943671543  ep_item_begin_failed: 0  ep_item_commit_failed: 0  ep_item_flush_expired: 0  ep_item_flush_failed: 0  ep_items_rm_from_checkpoints: 1029428  ep_keep_closed_checkpoints: 0  ep_kv_size: 3770087716  ep_latency_arith_cmd: 0  ep_latency_get_cmd: 10797709  ep_latency_store_cmd: 3048133  ep_max_data_size: 6448742400  ep_max_txn_size: 1000  ep_mem_high_wat: 4836556800  ep_mem_low_wat: 3869245440  ep_min_data_age: 0  ep_num_active_non_resident: 18446744073709551614  ep_num_checkpoint_remover_runs: 2657  ep_num_eject_failures: 0  ep_num_eject_replicas: 1221218  ep_num_expiry_pager_runs: 3  ep_num_non_resident: 1090405  ep_num_not_my_vbuckets: 23  ep_num_pager_runs: 0  ep_num_value_ejects: 1221218  ep_onlineupdate: false  ep_onlineupdate_revert_add: 0  ep_onlineupdate_revert_delete: 0  ep_onlineupdate_revert_update: 0  ep_oom_errors: 0  ep_overhead: 54621937  ep_pending_ops: 0  ep_pending_ops_max: 0  ep_pending_ops_max_duration: 0  ep_pending_ops_total: 0  ep_queue_age_cap: 900  ep_queue_size: 37  ep_storage_age: 47  ep_storage_age_highwat: 1127  ep_storage_type: featured  ep_store_max_concurrency: 10  ep_store_max_readers: 9  ep_store_max_readwrite: 1  ep_tap_bg_fetch_requeued: 0  ep_tap_bg_fetched: 39049  ep_tap_keepalive: 300  ep_tmp_oom_errors: 0  ep_too_old: 96  ep_too_young: 0  ep_total_cache_size: 4520935689  ep_total_del_items: 261729  ep_total_enqueued: 10934305  ep_total_new_items: 7583655  ep_total_persisted: 10834279  ep_uncommitted_items: 0  ep_value_size: 3116517595  ep_vb_total: 170  ep_vbucket_del: 20  ep_vbucket_del_avg_walltime: 25604  ep_vbucket_del_fail: 0  ep_vbucket_del_max_walltime: 298670  ep_vbucket_del_total_walltime: 512094  ep_version: 1.8.0_116_g9c329ef  ep_warmed_up: 0  ep_warmup: true  ep_warmup_dups: 0  ep_warmup_oom: 0  ep_warmup_thread: complete  ep_warmup_time: 16622  get_hits: 9786409  get_misses: 1011272  incr_hits: 0  incr_misses: 0  libevent: 2.0.11-stable  limit_maxbytes: 67108864  listen_disabled_num: 0  max_conns_on_port_11209: 1000  max_conns_on_port_11210: 9000  mem_used: 3948608208  pid: 3504  pointer_size: 64  rejected_conns: 0  rusage_system: 376.370782  rusage_user: 528.956586  tap_checkpoint_end_received: 1852  tap_checkpoint_end_sent: 1806  tap_checkpoint_start_received: 2286  tap_checkpoint_start_sent: 2002  tap_connect_received: 197  tap_delete_received: 136659  tap_delete_sent: 134993  tap_mutation_received: 16877089  tap_mutation_sent: 2757129  tap_opaque_received: 1618  tap_opaque_sent: 452  tap_vbucket_set_received: 204  tap_vbucket_set_sent: 34  threads: 4  time: 1338514002  total_connections: 709  uptime: 13306  vb_active_curr_items: 3311247  vb_active_eject: 153  vb_active_ht_memory: 25203264  vb_active_itm_memory: 2143553545  vb_active_num: 85  vb_active_num_non_resident: 18446744073709551614  vb_active_ops_create: 3440606  vb_active_ops_delete: 129360  vb_active_ops_reject: 3007  vb_active_ops_update: 1408966  vb_active_perc_mem_resident: 100  vb_active_queue_age: 135342000  vb_active_queue_drain: 4998166  vb_active_queue_fill: 4989550  vb_active_queue_memory: 1280  vb_active_queue_pending: 2080  vb_active_queue_size: 32  vb_dead_num: 0  vb_pending_curr_items: 0  vb_pending_eject: 0  vb_pending_ht_memory: 0  vb_pending_itm_memory: 0  vb_pending_num: 0  vb_pending_num_non_resident: 0  vb_pending_ops_create: 0  vb_pending_ops_delete: 0  vb_pending_ops_reject: 0  vb_pending_ops_update: 0  vb_pending_perc_mem_resident: 0  vb_pending_queue_age: 0  vb_pending_queue_drain: 0  vb_pending_queue_fill: 0  vb_pending_queue_memory: 0  vb_pending_queue_pending: 0  vb_pending_queue_size: 0  vb_replica_curr_items: 3310448  vb_replica_eject: 1182102  vb_replica_ht_memory: 24809888  vb_replica_itm_memory: 1547070096  vb_replica_num: 85  vb_replica_num_non_resident: 1090407  vb_replica_ops_create: 3399662  vb_replica_ops_delete: 128486  vb_replica_ops_reject: 5594  vb_replica_ops_update: 1537036  vb_replica_perc_mem_resident: 67  vb_replica_queue_age: 92131000  vb_replica_queue_drain: 5127526  vb_replica_queue_fill: 5065401  vb_replica_queue_memory: 880  vb_replica_queue_pending: 1430  vb_replica_queue_size: 22  version: 1.4.4_491_g6acfa96

Farshid Ghods added a comment - 31/May/12 8:27 PM coming from ep-engine: [ root@ip-10-32-61-83 ~]# /opt/couchbase/bin/cbstats 10.80.105.246:11210 raw "" | grep resident  ep_num_active_non_resident: 18446744073709551614  ep_num_non_resident: 1090296  vb_active_num_non_resident: 18446744073709551614

Aleksey Kondratenko added a comment - 31/May/12 8:31 PM right. Those are way off.

Aleksey Kondratenko added a comment - 31/May/12 8:32 PM passing down to ep-engine

Farshid Ghods added a comment - 01/Jun/12 7:19 PM if this is risky change we should defer this to future releases

Chiyoung Seo added a comment - 13/Jun/12 2:04 PM Mike, if you can reproduce this issue, this should be fixed for 1.8.1 release.

Chiyoung Seo added a comment - 14/Jun/12 5:19 PM http://review.couchbase.org/#change,17284

Thuan Nguyen added a comment - 14/Jun/12 7:25 PM Integrated in github-ep-engine-2-0 #311 (See [ http://qa.hq.northscale.net/job/github-ep-engine-2-0/311/ ])      MB-5418 Fix to the race in updating the non-resident item count. (Revision 261230595b3a9a4448bc7199edd6f017dfe86e27)      Result = SUCCESS Chiyoung Seo : Files : * stored-value.hh