Details
-
Type:
Bug
-
Status:
Resolved
-
Resolution: Fixed
-
Affects Version/s: 1.6.0 beta4
-
Fix Version/s: 1.6.0 GA
-
Component/s: bucket-engine
-
Labels:None
-
Environment:Operating System: All
Platform: All
Description
This has been reproduced a few times in a few different ways:
-Load memory until disk > RAM
-Rebalance
-See data missing
I don't believe there is anything useful in the logs, stats show:
STAT delete_misses 0
STAT ep_io_num_write 724494
STAT rejected_conns 0
STAT connection_structures 53
STAT limit_maxbytes 67108864
STAT decr_hits 0
STAT ep_bg_load_avg 163 usec
STAT ep_pending_ops_max_duration 493 ms
STAT ep_flush_duration_total 102
STAT ep_item_flush_expired 0
STAT ep_bg_wait_avg 72 usec
STAT ep_too_young 0
STAT curr_connections 44
STAT rusage_system 245.781631
STAT ep_io_write_bytes 194967165
STAT ep_total_cache_size 235059971
STAT ep_storage_age 0
STAT ep_flush_duration_highwat 19
STAT ep_flush_duration 0
STAT cas_misses 0
STAT ep_flusher_todo 0
STAT ep_pending_ops 0
STAT tap_mutation_received 1122439
STAT mem_used 94243203
STAT tap_mutation_sent 1228533
STAT ep_warmup_oom 0
STAT ep_vbucket_del 15
STAT get_misses 323342
STAT ep_num_value_ejects 2225263
STAT ep_queue_size 0
STAT bytes_read 561062450
STAT get_hits 5187912
STAT tap_vbucket_set_received 4096
STAT decr_misses 0
STAT ep_bg_min_wait 11 usec
STAT ep_commit_num 111
STAT rusage_user 512.176147
STAT bucket_conns 12
STAT ep_num_non_resident 415406
STAT ep_tap_keepalive 0
STAT ep_oom_errors 470821
STAT ep_too_old 0
STAT cmd_flush 0
STAT ep_bg_max_load 1936 ms
STAT ep_max_txn_size 100000
STAT ep_version 1.6.0beta4rc1_8_g4f2372b
STAT uptime 3492
STAT ep_data_age_highwat 23
STAT ep_queue_age_cap 1800
STAT incr_hits 0
STAT time 1285201669
STAT ep_warmup_dups 0
STAT ep_total_persisted 720187
STAT daemon_connections 20
STAT ep_flusher_state running
STAT pointer_size 64
STAT version 1.4.4_292_gc61961b
STAT ep_max_data_size 104857600
STAT ep_commit_time_total 86
STAT ep_warmup_time 2
STAT ep_item_commit_failed 0
STAT total_connections 4152
STAT curr_items 259726
STAT ep_data_age 5
STAT delete_hits 0
STAT ep_bg_max_wait 3932 ms
STAT ep_storage_type featured
STAT curr_items_tot 414213
STAT ep_total_enqueued 720187
STAT ep_mem_low_wat 62914560
STAT ep_kv_size 42663939
STAT ep_vbucket_del_fail 0
STAT ep_min_data_age 0
STAT ep_io_num_read 2427318
STAT ep_warmed_up 203677
STAT ep_item_flush_failed 0
STAT cas_hits 0
STAT ep_warmup true
STAT ep_dbname /opt/NorthScale/1.6.0beta4rc1/data/ns_1/test
STAT ep_commit_time 0
STAT auth_errors 0
STAT ep_bg_fetched 1473716
STAT ep_storage_age_highwat 21
STAT threads 8
STAT pid 20336
STAT auth_cmds 2064
STAT cas_badval 0
STAT cmd_set 565702
STAT ep_io_read_bytes 655719954
STAT cmd_get 5511254
STAT ep_expired 0
STAT tap_vbucket_set_sent 4096
STAT conn_yields 17602
STAT ep_warmup_thread complete
STAT ep_flush_preempts 0
STAT tap_connect_received 2054
STAT ep_num_eject_failures 30263677
STAT bytes_written 1605242771
STAT libevent 1.4.13-stable
STAT ep_bg_num_samples 1473585
STAT ep_num_pager_runs 302
STAT ep_mem_high_wat 78643200
STAT ep_dbinit 0
STAT incr_misses 0
STAT ep_bg_min_load 22 usec
STAT ep_pending_ops_total 5
STAT ep_pending_ops_max 1
STAT ep_overhead 51579264
END
-Load memory until disk > RAM
-Rebalance
-See data missing
I don't believe there is anything useful in the logs, stats show:
STAT delete_misses 0
STAT ep_io_num_write 724494
STAT rejected_conns 0
STAT connection_structures 53
STAT limit_maxbytes 67108864
STAT decr_hits 0
STAT ep_bg_load_avg 163 usec
STAT ep_pending_ops_max_duration 493 ms
STAT ep_flush_duration_total 102
STAT ep_item_flush_expired 0
STAT ep_bg_wait_avg 72 usec
STAT ep_too_young 0
STAT curr_connections 44
STAT rusage_system 245.781631
STAT ep_io_write_bytes 194967165
STAT ep_total_cache_size 235059971
STAT ep_storage_age 0
STAT ep_flush_duration_highwat 19
STAT ep_flush_duration 0
STAT cas_misses 0
STAT ep_flusher_todo 0
STAT ep_pending_ops 0
STAT tap_mutation_received 1122439
STAT mem_used 94243203
STAT tap_mutation_sent 1228533
STAT ep_warmup_oom 0
STAT ep_vbucket_del 15
STAT get_misses 323342
STAT ep_num_value_ejects 2225263
STAT ep_queue_size 0
STAT bytes_read 561062450
STAT get_hits 5187912
STAT tap_vbucket_set_received 4096
STAT decr_misses 0
STAT ep_bg_min_wait 11 usec
STAT ep_commit_num 111
STAT rusage_user 512.176147
STAT bucket_conns 12
STAT ep_num_non_resident 415406
STAT ep_tap_keepalive 0
STAT ep_oom_errors 470821
STAT ep_too_old 0
STAT cmd_flush 0
STAT ep_bg_max_load 1936 ms
STAT ep_max_txn_size 100000
STAT ep_version 1.6.0beta4rc1_8_g4f2372b
STAT uptime 3492
STAT ep_data_age_highwat 23
STAT ep_queue_age_cap 1800
STAT incr_hits 0
STAT time 1285201669
STAT ep_warmup_dups 0
STAT ep_total_persisted 720187
STAT daemon_connections 20
STAT ep_flusher_state running
STAT pointer_size 64
STAT version 1.4.4_292_gc61961b
STAT ep_max_data_size 104857600
STAT ep_commit_time_total 86
STAT ep_warmup_time 2
STAT ep_item_commit_failed 0
STAT total_connections 4152
STAT curr_items 259726
STAT ep_data_age 5
STAT delete_hits 0
STAT ep_bg_max_wait 3932 ms
STAT ep_storage_type featured
STAT curr_items_tot 414213
STAT ep_total_enqueued 720187
STAT ep_mem_low_wat 62914560
STAT ep_kv_size 42663939
STAT ep_vbucket_del_fail 0
STAT ep_min_data_age 0
STAT ep_io_num_read 2427318
STAT ep_warmed_up 203677
STAT ep_item_flush_failed 0
STAT cas_hits 0
STAT ep_warmup true
STAT ep_dbname /opt/NorthScale/1.6.0beta4rc1/data/ns_1/test
STAT ep_commit_time 0
STAT auth_errors 0
STAT ep_bg_fetched 1473716
STAT ep_storage_age_highwat 21
STAT threads 8
STAT pid 20336
STAT auth_cmds 2064
STAT cas_badval 0
STAT cmd_set 565702
STAT ep_io_read_bytes 655719954
STAT cmd_get 5511254
STAT ep_expired 0
STAT tap_vbucket_set_sent 4096
STAT conn_yields 17602
STAT ep_warmup_thread complete
STAT ep_flush_preempts 0
STAT tap_connect_received 2054
STAT ep_num_eject_failures 30263677
STAT bytes_written 1605242771
STAT libevent 1.4.13-stable
STAT ep_bg_num_samples 1473585
STAT ep_num_pager_runs 302
STAT ep_mem_high_wat 78643200
STAT ep_dbinit 0
STAT incr_misses 0
STAT ep_bg_min_load 22 usec
STAT ep_pending_ops_total 5
STAT ep_pending_ops_max 1
STAT ep_overhead 51579264
END
Activating acks may make this far more reliable and it can pay attention to these at the same time.