Couchbase bucket memory leak
For some reason in all of my couchbase buckets, the stats in Memory Used are far larger than the actual data that's in the bucket.
Example:
Bucket Quota: 600 MB
Memory Used : 550 MB
High Water : 471 MB
Low Water : 337 MB
Items : 11.2k
Item Expire : 2 hours
Items Added : 2.5 / sec
Avg Image Size: 2k
User data in RAM: 1.51M
Meta data in RAM: 25.6M
In the web interface we see items being added and removed during the days and nights (and during the hourly cleanup) as expected, but ram continues to balloon until we start to get OOM errors. When we moved this to an Memcache bucket we never spike above 10 MB of RAM, so why is the Couchbase bucket using 550+ MB ?
https://groups.google.com/forum/?fromgroups=#!topic/couchbase/Ip1zUnPoPiw
We moved SessionCache (couchbase bucket) to SessionCacheMem bucket (Memcache bucket), but all others are still on couchbase buckets. One example is CacheBase. It has about 11M of data (1.42K items), but yet the bucket uses 1.2GB of RAM!
Did you get my files? Any ideas?
I took a look at them yesterday and wasn't able to find anything suspicious. I have passed them on to another developer though and he will take a look at them as well. I will get back to you once he has had a look.
We continue to have memory leak style problems on all of our nodes. All user data should easily fit within 1% of the bucket's ram, yet the bucket's ram usage continues to skyrocket (and items remain constant). If this continues, I'm afraid we'll have to ditch Couchbase, which would be a shame because we REALLY want to love it.
I'm still trying to figure out what is going on here and this issue is of interest to me. Can you post the results of cbstats host:11210 tap? These stats might be able to help me identify the issue.
Stats 'tap' are not available from the requested engine.
Did you run this on one of your memcached buckets on accident? Also, by default this goes to the default bucket. If your Couchbase bucket is named something else there should be a -b option for bucket name and a -p option for password if needed.
/opt/couchbase/bin# ./cbstats 192.168.1.41:11211 -b CacheBase tap
ep_tap_ack_grace_period: 900
ep_tap_ack_interval: 3000
ep_tap_ack_window_size: 30
ep_tap_backoff_period: 15
ep_tap_bg_fetch_requeued: 0
ep_tap_bg_fetched: 0
ep_tap_bg_max_pending: 1500
ep_tap_count: 12
ep_tap_deletes: 1238
ep_tap_fg_fetched: 7230695
ep_tap_keepalive: 900
ep_tap_noop_interval: 60
ep_tap_queue_backfillremaining: 0
ep_tap_queue_backoff: 0
ep_tap_queue_drain: 7102686
ep_tap_queue_fill: 0
ep_tap_queue_itemondisk: 0
ep_tap_throttle_queue_cap: 3000000
ep_tap_throttle_threshold: 270
ep_tap_throttled: 0
ep_tap_total_backlog_size: 1
ep_tap_total_fetched: 99664338
ep_tap_total_queue: 0
eq_tapq:anon_11621:connected: true
eq_tapq:anon_11621:created: 9617
eq_tapq:anon_11621:num_checkpoint_end: 498671
eq_tapq:anon_11621:num_checkpoint_end_failed: 0
eq_tapq:anon_11621:num_checkpoint_start: 498841
eq_tapq:anon_11621:num_checkpoint_start_failed: 0
eq_tapq:anon_11621:num_delete: 220
eq_tapq:anon_11621:num_delete_failed: 0
eq_tapq:anon_11621:num_flush: 0
eq_tapq:anon_11621:num_flush_failed: 0
eq_tapq:anon_11621:num_mutation: 395163
eq_tapq:anon_11621:num_mutation_failed: 0
eq_tapq:anon_11621:num_opaque: 2
eq_tapq:anon_11621:num_opaque_failed: 0
eq_tapq:anon_11621:num_unknown: 0
eq_tapq:anon_11621:num_vbucket_set: 0
eq_tapq:anon_11621:num_vbucket_set_failed: 0
eq_tapq:anon_11621:pending_disconnect: false
eq_tapq:anon_11621:reserved: 0
eq_tapq:anon_11621:supports_ack: true
eq_tapq:anon_11621:type: consumer
eq_tapq:anon_11964:connected: true
eq_tapq:anon_11964:created: 9906
eq_tapq:anon_11964:num_checkpoint_end: 1488782
eq_tapq:anon_11964:num_checkpoint_end_failed: 0
eq_tapq:anon_11964:num_checkpoint_start: 1488783
eq_tapq:anon_11964:num_checkpoint_start_failed: 0
eq_tapq:anon_11964:num_delete: 216
eq_tapq:anon_11964:num_delete_failed: 0
eq_tapq:anon_11964:num_flush: 0
eq_tapq:anon_11964:num_flush_failed: 0
eq_tapq:anon_11964:num_mutation: 411693
eq_tapq:anon_11964:num_mutation_failed: 0
eq_tapq:anon_11964:num_opaque: 2
eq_tapq:anon_11964:num_opaque_failed: 0
eq_tapq:anon_11964:num_unknown: 0
eq_tapq:anon_11964:num_vbucket_set: 0
eq_tapq:anon_11964:num_vbucket_set_failed: 0
eq_tapq:anon_11964:pending_disconnect: false
eq_tapq:anon_11964:reserved: 0
eq_tapq:anon_11964:supports_ack: true
eq_tapq:anon_11964:type: consumer
eq_tapq:anon_1683:connected: true
eq_tapq:anon_1683:created: 86
eq_tapq:anon_1683:num_checkpoint_end: 509326
eq_tapq:anon_1683:num_checkpoint_end_failed: 0
eq_tapq:anon_1683:num_checkpoint_start: 509327
eq_tapq:anon_1683:num_checkpoint_start_failed: 0
eq_tapq:anon_1683:num_delete: 185
eq_tapq:anon_1683:num_delete_failed: 0
eq_tapq:anon_1683:num_flush: 0
eq_tapq:anon_1683:num_flush_failed: 0
eq_tapq:anon_1683:num_mutation: 395530
eq_tapq:anon_1683:num_mutation_failed: 0
eq_tapq:anon_1683:num_opaque: 2
eq_tapq:anon_1683:num_opaque_failed: 0
eq_tapq:anon_1683:num_unknown: 0
eq_tapq:anon_1683:num_vbucket_set: 0
eq_tapq:anon_1683:num_vbucket_set_failed: 0
eq_tapq:anon_1683:pending_disconnect: false
eq_tapq:anon_1683:reserved: 0
eq_tapq:anon_1683:supports_ack: true
eq_tapq:anon_1683:type: consumer
eq_tapq:anon_1708:connected: true
eq_tapq:anon_1708:created: 88
eq_tapq:anon_1708:num_checkpoint_end: 23003351
eq_tapq:anon_1708:num_checkpoint_end_failed: 0
eq_tapq:anon_1708:num_checkpoint_start: 23003352
eq_tapq:anon_1708:num_checkpoint_start_failed: 0
eq_tapq:anon_1708:num_delete: 223
eq_tapq:anon_1708:num_delete_failed: 0
eq_tapq:anon_1708:num_flush: 0
eq_tapq:anon_1708:num_flush_failed: 0
eq_tapq:anon_1708:num_mutation: 367018
eq_tapq:anon_1708:num_mutation_failed: 0
eq_tapq:anon_1708:num_opaque: 2
eq_tapq:anon_1708:num_opaque_failed: 0
eq_tapq:anon_1708:num_unknown: 0
eq_tapq:anon_1708:num_vbucket_set: 0
eq_tapq:anon_1708:num_vbucket_set_failed: 0
eq_tapq:anon_1708:pending_disconnect: false
eq_tapq:anon_1708:reserved: 0
eq_tapq:anon_1708:supports_ack: true
eq_tapq:anon_1708:type: consumer
eq_tapq:anon_21521:connected: true
eq_tapq:anon_21521:created: 319988
eq_tapq:anon_21521:num_checkpoint_end: 18803098
eq_tapq:anon_21521:num_checkpoint_end_failed: 0
eq_tapq:anon_21521:num_checkpoint_start: 18803268
eq_tapq:anon_21521:num_checkpoint_start_failed: 0
eq_tapq:anon_21521:num_delete: 211
eq_tapq:anon_21521:num_delete_failed: 0
eq_tapq:anon_21521:num_flush: 0
eq_tapq:anon_21521:num_flush_failed: 0
eq_tapq:anon_21521:num_mutation: 389663
eq_tapq:anon_21521:num_mutation_failed: 0
eq_tapq:anon_21521:num_opaque: 2
eq_tapq:anon_21521:num_opaque_failed: 0
eq_tapq:anon_21521:num_unknown: 0
eq_tapq:anon_21521:num_vbucket_set: 0
eq_tapq:anon_21521:num_vbucket_set_failed: 0
eq_tapq:anon_21521:pending_disconnect: false
eq_tapq:anon_21521:reserved: 0
eq_tapq:anon_21521:supports_ack: true
eq_tapq:anon_21521:type: consumer
eq_tapq:anon_21863:connected: true
eq_tapq:anon_21863:created: 320277
eq_tapq:anon_21863:num_checkpoint_end: 1820448
eq_tapq:anon_21863:num_checkpoint_end_failed: 0
eq_tapq:anon_21863:num_checkpoint_start: 1820449
eq_tapq:anon_21863:num_checkpoint_start_failed: 0
eq_tapq:anon_21863:num_delete: 183
eq_tapq:anon_21863:num_delete_failed: 0
eq_tapq:anon_21863:num_flush: 0
eq_tapq:anon_21863:num_flush_failed: 0
eq_tapq:anon_21863:num_mutation: 5142381
eq_tapq:anon_21863:num_mutation_failed: 0
eq_tapq:anon_21863:num_opaque: 2
eq_tapq:anon_21863:num_opaque_failed: 0
eq_tapq:anon_21863:num_unknown: 0
eq_tapq:anon_21863:num_vbucket_set: 0
eq_tapq:anon_21863:num_vbucket_set_failed: 0
eq_tapq:anon_21863:pending_disconnect: false
eq_tapq:anon_21863:reserved: 0
eq_tapq:anon_21863:supports_ack: true
eq_tapq:anon_21863:type: consumer
eq_tapq:replication_ns_1@192.168.1.41:ack_log_size: 0
eq_tapq:replication_ns_1@192.168.1.41:ack_playback_size: 0
eq_tapq:replication_ns_1@192.168.1.41:ack_seqno: 47790022
eq_tapq:replication_ns_1@192.168.1.41:ack_window_full: false
eq_tapq:replication_ns_1@192.168.1.41:backfill_completed: true
eq_tapq:replication_ns_1@192.168.1.41:bg_backlog_size: 0
eq_tapq:replication_ns_1@192.168.1.41:bg_jobs_completed: 0
eq_tapq:replication_ns_1@192.168.1.41:bg_jobs_issued: 0
eq_tapq:replication_ns_1@192.168.1.41:bg_queued: 0
eq_tapq:replication_ns_1@192.168.1.41:bg_result_size: 0
eq_tapq:replication_ns_1@192.168.1.41:bg_results: 0
eq_tapq:replication_ns_1@192.168.1.41:bg_wait_for_results: false
eq_tapq:replication_ns_1@192.168.1.41:complete: false
eq_tapq:replication_ns_1@192.168.1.41:connected: true
eq_tapq:replication_ns_1@192.168.1.41:created: 330184
eq_tapq:replication_ns_1@192.168.1.41:disconnects: 340
eq_tapq:replication_ns_1@192.168.1.41:empty: true
eq_tapq:replication_ns_1@192.168.1.41:has_item: false
eq_tapq:replication_ns_1@192.168.1.41:has_queued_item: false
eq_tapq:replication_ns_1@192.168.1.41:idle: true
eq_tapq:replication_ns_1@192.168.1.41:num_tap_nack: 0
eq_tapq:replication_ns_1@192.168.1.41:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@192.168.1.41:paused: 2
eq_tapq:replication_ns_1@192.168.1.41:pending_backfill: false
eq_tapq:replication_ns_1@192.168.1.41:pending_disconnect: false
eq_tapq:replication_ns_1@192.168.1.41:pending_disk_backfill: false
eq_tapq:replication_ns_1@192.168.1.41:qlen: 0
eq_tapq:replication_ns_1@192.168.1.41:qlen_high_pri: 0
eq_tapq:replication_ns_1@192.168.1.41:qlen_low_pri: 0
eq_tapq:replication_ns_1@192.168.1.41:queue_backfillremaining: 0
eq_tapq:replication_ns_1@192.168.1.41:queue_backoff: 0
eq_tapq:replication_ns_1@192.168.1.41:queue_drain: 762956
eq_tapq:replication_ns_1@192.168.1.41:queue_fill: 0
eq_tapq:replication_ns_1@192.168.1.41:queue_itemondisk: 0
eq_tapq:replication_ns_1@192.168.1.41:queue_memory: 0
eq_tapq:replication_ns_1@192.168.1.41:rec_fetched: 47789336
eq_tapq:replication_ns_1@192.168.1.41:reconnects: 340
eq_tapq:replication_ns_1@192.168.1.41:recv_ack_seqno: 47790020
eq_tapq:replication_ns_1@192.168.1.41:reserved: 2
eq_tapq:replication_ns_1@192.168.1.41:seqno_ack_requested: 47790020
eq_tapq:replication_ns_1@192.168.1.41:supports_ack: true
eq_tapq:replication_ns_1@192.168.1.41:suspended: false
eq_tapq:replication_ns_1@192.168.1.41:total_backlog_size: 0
eq_tapq:replication_ns_1@192.168.1.41:total_noops: 518708
eq_tapq:replication_ns_1@192.168.1.41:type: producer
eq_tapq:replication_ns_1@192.168.1.41:vb_filters: 342
eq_tapq:replication_ns_1@192.168.1.42:ack_log_size: 0
eq_tapq:replication_ns_1@192.168.1.42:ack_playback_size: 0
eq_tapq:replication_ns_1@192.168.1.42:ack_seqno: 46780561
eq_tapq:replication_ns_1@192.168.1.42:ack_window_full: false
eq_tapq:replication_ns_1@192.168.1.42:backfill_completed: true
eq_tapq:replication_ns_1@192.168.1.42:bg_backlog_size: 0
eq_tapq:replication_ns_1@192.168.1.42:bg_jobs_completed: 0
eq_tapq:replication_ns_1@192.168.1.42:bg_jobs_issued: 0
eq_tapq:replication_ns_1@192.168.1.42:bg_queued: 0
eq_tapq:replication_ns_1@192.168.1.42:bg_result_size: 0
eq_tapq:replication_ns_1@192.168.1.42:bg_results: 0
eq_tapq:replication_ns_1@192.168.1.42:bg_wait_for_results: false
eq_tapq:replication_ns_1@192.168.1.42:complete: false
eq_tapq:replication_ns_1@192.168.1.42:connected: true
eq_tapq:replication_ns_1@192.168.1.42:created: 9631
eq_tapq:replication_ns_1@192.168.1.42:disconnects: 170
eq_tapq:replication_ns_1@192.168.1.42:empty: false
eq_tapq:replication_ns_1@192.168.1.42:has_item: false
eq_tapq:replication_ns_1@192.168.1.42:has_queued_item: true
eq_tapq:replication_ns_1@192.168.1.42:idle: false
eq_tapq:replication_ns_1@192.168.1.42:num_tap_nack: 0
eq_tapq:replication_ns_1@192.168.1.42:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@192.168.1.42:paused: 2
eq_tapq:replication_ns_1@192.168.1.42:pending_backfill: false
eq_tapq:replication_ns_1@192.168.1.42:pending_disconnect: false
eq_tapq:replication_ns_1@192.168.1.42:pending_disk_backfill: false
eq_tapq:replication_ns_1@192.168.1.42:qlen: 0
eq_tapq:replication_ns_1@192.168.1.42:qlen_high_pri: 0
eq_tapq:replication_ns_1@192.168.1.42:qlen_low_pri: 0
eq_tapq:replication_ns_1@192.168.1.42:queue_backfillremaining: 0
eq_tapq:replication_ns_1@192.168.1.42:queue_backoff: 0
eq_tapq:replication_ns_1@192.168.1.42:queue_drain: 5532438
eq_tapq:replication_ns_1@192.168.1.42:queue_fill: 0
eq_tapq:replication_ns_1@192.168.1.42:queue_itemondisk: 0
eq_tapq:replication_ns_1@192.168.1.42:queue_memory: 0
eq_tapq:replication_ns_1@192.168.1.42:rec_fetched: 46780215
eq_tapq:replication_ns_1@192.168.1.42:reconnects: 170
eq_tapq:replication_ns_1@192.168.1.42:recv_ack_seqno: 46780559
eq_tapq:replication_ns_1@192.168.1.42:reserved: 2
eq_tapq:replication_ns_1@192.168.1.42:seqno_ack_requested: 46780559
eq_tapq:replication_ns_1@192.168.1.42:supports_ack: true
eq_tapq:replication_ns_1@192.168.1.42:suspended: false
eq_tapq:replication_ns_1@192.168.1.42:total_backlog_size: 1
eq_tapq:replication_ns_1@192.168.1.42:total_noops: 518736
eq_tapq:replication_ns_1@192.168.1.42:type: producer
eq_tapq:replication_ns_1@192.168.1.42:vb_filters: 341
eq_tapq:replication_ns_1@192.168.1.43:ack_log_size: 0
eq_tapq:replication_ns_1@192.168.1.43:ack_playback_size: 0
eq_tapq:replication_ns_1@192.168.1.43:ack_seqno: 4783229
eq_tapq:replication_ns_1@192.168.1.43:ack_window_full: false
eq_tapq:replication_ns_1@192.168.1.43:backfill_completed: true
eq_tapq:replication_ns_1@192.168.1.43:bg_backlog_size: 0
eq_tapq:replication_ns_1@192.168.1.43:bg_jobs_completed: 0
eq_tapq:replication_ns_1@192.168.1.43:bg_jobs_issued: 0
eq_tapq:replication_ns_1@192.168.1.43:bg_queued: 0
eq_tapq:replication_ns_1@192.168.1.43:bg_result_size: 0
eq_tapq:replication_ns_1@192.168.1.43:bg_results: 0
eq_tapq:replication_ns_1@192.168.1.43:bg_wait_for_results: false
eq_tapq:replication_ns_1@192.168.1.43:complete: false
eq_tapq:replication_ns_1@192.168.1.43:connected: true
eq_tapq:replication_ns_1@192.168.1.43:created: 320002
eq_tapq:replication_ns_1@192.168.1.43:disconnects: 170
eq_tapq:replication_ns_1@192.168.1.43:empty: true
eq_tapq:replication_ns_1@192.168.1.43:has_item: false
eq_tapq:replication_ns_1@192.168.1.43:has_queued_item: false
eq_tapq:replication_ns_1@192.168.1.43:idle: true
eq_tapq:replication_ns_1@192.168.1.43:num_tap_nack: 0
eq_tapq:replication_ns_1@192.168.1.43:num_tap_tmpfail_survivors: 0
eq_tapq:replication_ns_1@192.168.1.43:paused: 2
eq_tapq:replication_ns_1@192.168.1.43:pending_backfill: false
eq_tapq:replication_ns_1@192.168.1.43:pending_disconnect: false
eq_tapq:replication_ns_1@192.168.1.43:pending_disk_backfill: false
eq_tapq:replication_ns_1@192.168.1.43:qlen: 0
eq_tapq:replication_ns_1@192.168.1.43:qlen_high_pri: 0
eq_tapq:replication_ns_1@192.168.1.43:qlen_low_pri: 0
eq_tapq:replication_ns_1@192.168.1.43:queue_backfillremaining: 0
eq_tapq:replication_ns_1@192.168.1.43:queue_backoff: 0
eq_tapq:replication_ns_1@192.168.1.43:queue_drain: 807292
eq_tapq:replication_ns_1@192.168.1.43:queue_fill: 0
eq_tapq:replication_ns_1@192.168.1.43:queue_itemondisk: 0
eq_tapq:replication_ns_1@192.168.1.43:queue_memory: 0
eq_tapq:replication_ns_1@192.168.1.43:rec_fetched: 4782883
eq_tapq:replication_ns_1@192.168.1.43:reconnects: 170
eq_tapq:replication_ns_1@192.168.1.43:recv_ack_seqno: 4783227
eq_tapq:replication_ns_1@192.168.1.43:reserved: 2
eq_tapq:replication_ns_1@192.168.1.43:seqno_ack_requested: 4783227
eq_tapq:replication_ns_1@192.168.1.43:supports_ack: true
eq_tapq:replication_ns_1@192.168.1.43:suspended: false
eq_tapq:replication_ns_1@192.168.1.43:total_backlog_size: 0
eq_tapq:replication_ns_1@192.168.1.43:total_noops: 518723
eq_tapq:replication_ns_1@192.168.1.43:type: producer
eq_tapq:replication_ns_1@192.168.1.43:vb_filters: 341
Please let me know if I'm being dumb, but from what the web interface says this is what I understand. We have 1.42k items in this bucket. They are using 14M of total ram (which sounds about right), yet the bucket is taking up 1.2GB of RAM.
Items: 1.42K
Memory Used: 1.2G
High Water: 1.17G
Includes 1 Replica
Total User Data In RAM: 14.3M
Total Metadata in RAM: 51.3M
Your not being dumb. There is definitely something going on here. The strange thing for me is that I have never heard of anyone else running into an issue like this. Can you comment on what your workload is like? Are you doing a lot of appends?
We refresh all items every hour. We only do sets, no appends, deletes, adds etc.. only ->set().
We are using Debian Linux, PHP with the 1.0.5 API and Couchbase 1.8.1
One more question. Can you also post the results of cbstats host:11210 memory?
Thanks,
- Mike
memory is not an option
/opt/couchbase/bin/cbstats 127.0.0.1:11210 -b CacheBase memory
Usage: cbstats host:port all
or cbstats host:port allocator
or cbstats host:port checkpoint [vbid]
or cbstats host:port dispatcher [logs]
or cbstats host:port hash [detail]
or cbstats host:port items
or cbstats host:port kvstore
or cbstats host:port kvtimings
or cbstats host:port raw argument
or cbstats host:port reset
or cbstats host:port slabs
or cbstats host:port tap [username password]
or cbstats host:port tapagg
or cbstats host:port timings
or cbstats host:port vkey keyname vbid
Flags:
-a : iterate over all buckets (requires admin u/p)
-p : the password for the bucket if one exists
-b : the bucket to get stats from (Default: default)
Sorry, we changed this for 2.0. Try this:
/opt/couchbase/bin/cbstats 127.0.0.1:11210 -b CacheBase raw memory
SERVER 1
ep_kv_size: 7003279
ep_max_data_size: 524288000
ep_mem_high_wat: 393216000
ep_mem_low_wat: 314572800
ep_mem_tracker_enabled: true
ep_oom_errors: 0
ep_overhead: 17950984
ep_tmp_oom_errors: 0
ep_value_size: 6921399
mem_used: 393032232
tcmalloc_current_thread_cache_bytes: 4888904
tcmalloc_max_thread_cache_bytes: 4194304
tcmalloc_unmapped_bytes: 19750912
total_allocated_bytes: 1498417064
total_fragmentation_bytes: 108632152
total_free_bytes: 23486464
total_heap_bytes: 1630535680
SERVER 2
ep_kv_size: 6323176
ep_max_data_size: 524288000
ep_mem_high_wat: 393216000
ep_mem_low_wat: 314572800
ep_mem_tracker_enabled: true
ep_oom_errors: 0
ep_overhead: 18018744
ep_tmp_oom_errors: 0
ep_value_size: 6241724
mem_used: 392985432
tcmalloc_current_thread_cache_bytes: 4777640
tcmalloc_max_thread_cache_bytes: 4194304
tcmalloc_unmapped_bytes: 23404544
total_allocated_bytes: 1497253960
total_fragmentation_bytes: 110794680
total_free_bytes: 23535616
total_heap_bytes: 1631584256
SERVER 3
ep_kv_size: 1128955
ep_max_data_size: 524288000
ep_mem_high_wat: 393216000
ep_mem_low_wat: 314572800
ep_mem_tracker_enabled: true
ep_oom_errors: 0
ep_overhead: 17978320
ep_tmp_oom_errors: 0
ep_value_size: 1044726
mem_used: 412762312
tcmalloc_current_thread_cache_bytes: 5320584
tcmalloc_max_thread_cache_bytes: 4194304
tcmalloc_unmapped_bytes: 4833280
total_allocated_bytes: 1510066488
total_fragmentation_bytes: 92632776
total_free_bytes: 30982144
total_heap_bytes: 1633681408
Josh,
Sorry to ask for so many stats, but if you could can you also give me the results of cbstats allocator. This is a very strange issue that I have never seen before and has never been reported by any other users. Let me describe what I think might be happening. In the 1.8.1 release of Couchbase we track the exact size of all memory allocations and use the total size per bucket to make decisions on what to do to keep the cache at an appropriate size. One thing for example is to remove items from the cache when it is getting too big. From the memory stats I can see that the amount of space being take up is 1.1MB (ep_kv_size) which is the size all of the meta data and values in the cache. The bucket overhead is about 18MB (ep_overhead) and this is the size of all of the structures that the engine uses internally. The strange part, which you have pointed out is that the bucket uses about 400MB on each node to hold data, but from looking at the stats none of them have accounted for this memory.
One other thing I want to point out is that we only use allocator tracking for the mem_used stat and not for any of the other stats. This means that for mem_used we actually ask the allocator how big of an allocation it made (since if we as for 16B we might actually get a 32B chuck), but for the other stats I pointed out above we just estimate the size by using the sizeof operator. cbstats allocator will dump a snapshot of what the underlying allocator is doing an will tell me if the allocator for some reason has decided to give you very large chunks for small objects. At the moment this seems like the only plausible explanation for the behavior that your seeing.
I have sent them to you directly via Private Message on this forum. For some reason, when I pasted the results of that command from my 3 servers, it wouldn't let me post them. It said they were spam!
How many buckets do you have and what is the quota for each bucket?
Couch Bucket Quotas
CacheBase = 1.46 GB
EndecaCache = 900 MB
ProductLinePageCache = 2.92 GB
MemCache Bucket Quotas
SessionCacheMem 600 MB
No problem! I'll do whatever you want if we can get this figured out!
SERVER 1
/opt/couchbase/bin/cbstats 127.0.0.1:11210 -b CacheBase allocator
NOTE: SMALL MEMORY MODEL IS IN USE, PERFORMANCE MAY SUFFER.
------------------------------------------------
MALLOC: 1505244440 ( 1435.5 MiB) Bytes in use by application
MALLOC: + 22216704 ( 21.2 MiB) Bytes in page heap freelist
MALLOC: + 78837776 ( 75.2 MiB) Bytes in central cache freelist
MALLOC: + 0 ( 0.0 MiB) Bytes in transfer cache freelist
MALLOC: + 4944600 ( 4.7 MiB) Bytes in thread cache freelists
MALLOC: + 7631000 ( 7.3 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 1618874520 ( 1543.9 MiB) Actual memory used (physical + swap)
MALLOC: + 20340736 ( 19.4 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 1639215256 ( 1563.3 MiB) Virtual address space used
MALLOC:
MALLOC: 103117 Spans in use
MALLOC: 17 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
------------------------------------------------
Size class breakdown
------------------------------------------------
class 1 [ 8 bytes ] : 937 objs; 0.0 MiB; 0.0 cum MiB
class 2 [ 16 bytes ] : 372 objs; 0.0 MiB; 0.0 cum MiB
class 3 [ 32 bytes ] : 181662 objs; 5.5 MiB; 5.6 cum MiB
class 4 [ 48 bytes ] : 73739 objs; 3.4 MiB; 8.9 cum MiB
class 5 [ 64 bytes ] : 37478 objs; 2.3 MiB; 11.2 cum MiB
class 6 [ 80 bytes ] : 1057 objs; 0.1 MiB; 11.3 cum MiB
class 7 [ 96 bytes ] : 1700 objs; 0.2 MiB; 11.5 cum MiB
class 8 [ 112 bytes ] : 305 objs; 0.0 MiB; 11.5 cum MiB
class 9 [ 128 bytes ] : 791 objs; 0.1 MiB; 11.6 cum MiB
class 10 [ 144 bytes ] : 219 objs; 0.0 MiB; 11.6 cum MiB
class 11 [ 160 bytes ] : 668 objs; 0.1 MiB; 11.7 cum MiB
class 12 [ 176 bytes ] : 121 objs; 0.0 MiB; 11.7 cum MiB
class 13 [ 192 bytes ] : 562 objs; 0.1 MiB; 11.8 cum MiB
class 14 [ 208 bytes ] : 68 objs; 0.0 MiB; 11.9 cum MiB
class 15 [ 224 bytes ] : 97 objs; 0.0 MiB; 11.9 cum MiB
class 16 [ 240 bytes ] : 110 objs; 0.0 MiB; 11.9 cum MiB
class 17 [ 256 bytes ] : 108 objs; 0.0 MiB; 11.9 cum MiB
class 18 [ 288 bytes ] : 166 objs; 0.0 MiB; 12.0 cum MiB
class 19 [ 320 bytes ] : 138 objs; 0.0 MiB; 12.0 cum MiB
class 20 [ 352 bytes ] : 90 objs; 0.0 MiB; 12.0 cum MiB
class 21 [ 384 bytes ] : 420 objs; 0.2 MiB; 12.2 cum MiB
class 22 [ 416 bytes ] : 84 objs; 0.0 MiB; 12.2 cum MiB
class 23 [ 448 bytes ] : 96 objs; 0.0 MiB; 12.3 cum MiB
class 24 [ 480 bytes ] : 54 objs; 0.0 MiB; 12.3 cum MiB
class 25 [ 512 bytes ] : 225 objs; 0.1 MiB; 12.4 cum MiB
class 26 [ 576 bytes ] : 245 objs; 0.1 MiB; 12.5 cum MiB
class 27 [ 640 bytes ] : 28 objs; 0.0 MiB; 12.6 cum MiB
class 28 [ 704 bytes ] : 44 objs; 0.0 MiB; 12.6 cum MiB
class 29 [ 768 bytes ] : 81 objs; 0.1 MiB; 12.6 cum MiB
class 30 [ 896 bytes ] : 464 objs; 0.4 MiB; 13.0 cum MiB
class 31 [ 1024 bytes ] : 112 objs; 0.1 MiB; 13.2 cum MiB
class 32 [ 1152 bytes ] : 141 objs; 0.2 MiB; 13.3 cum MiB
class 33 [ 1280 bytes ] : 309 objs; 0.4 MiB; 13.7 cum MiB
class 34 [ 1408 bytes ] : 148 objs; 0.2 MiB; 13.9 cum MiB
class 35 [ 1536 bytes ] : 109 objs; 0.2 MiB; 14.0 cum MiB
class 36 [ 1792 bytes ] : 533 objs; 0.9 MiB; 15.0 cum MiB
class 37 [ 2048 bytes ] : 211 objs; 0.4 MiB; 15.4 cum MiB
class 38 [ 2304 bytes ] : 103 objs; 0.2 MiB; 15.6 cum MiB
class 39 [ 2560 bytes ] : 198 objs; 0.5 MiB; 16.1 cum MiB
class 40 [ 2816 bytes ] : 247 objs; 0.7 MiB; 16.7 cum MiB
class 41 [ 3072 bytes ] : 160 objs; 0.5 MiB; 17.2 cum MiB
class 42 [ 3328 bytes ] : 182 objs; 0.6 MiB; 17.8 cum MiB
class 43 [ 4096 bytes ] : 251 objs; 1.0 MiB; 18.8 cum MiB
class 44 [ 4608 bytes ] : 239 objs; 1.1 MiB; 19.8 cum MiB
class 45 [ 5120 bytes ] : 112 objs; 0.5 MiB; 20.4 cum MiB
class 46 [ 6144 bytes ] : 119 objs; 0.7 MiB; 21.1 cum MiB
class 47 [ 6656 bytes ] : 77 objs; 0.5 MiB; 21.5 cum MiB
class 48 [ 8192 bytes ] : 29 objs; 0.2 MiB; 21.8 cum MiB
class 49 [ 9216 bytes ] : 30 objs; 0.3 MiB; 22.0 cum MiB
class 50 [ 10240 bytes ] : 27 objs; 0.3 MiB; 22.3 cum MiB
class 51 [ 12288 bytes ] : 27 objs; 0.3 MiB; 22.6 cum MiB
class 52 [ 13312 bytes ] : 17 objs; 0.2 MiB; 22.8 cum MiB
class 53 [ 16384 bytes ] : 7 objs; 0.1 MiB; 22.9 cum MiB
class 54 [ 20480 bytes ] : 3 objs; 0.1 MiB; 23.0 cum MiB
class 55 [ 24576 bytes ] : 2 objs; 0.0 MiB; 23.1 cum MiB
class 56 [ 26624 bytes ] : 20 objs; 0.5 MiB; 23.6 cum MiB
class 57 [ 32768 bytes ] : 5 objs; 0.2 MiB; 23.7 cum MiB
class 67 [ 114688 bytes ] : 1 objs; 0.1 MiB; 23.8 cum MiB
class 73 [ 163840 bytes ] : 3 objs; 0.5 MiB; 24.3 cum MiB
------------------------------------------------
PageHeap: 31 sizes; 21.2 MiB free; 19.4 MiB unmapped
------------------------------------------------
1 pages * 575 spans ~ 4.5 MiB; 4.5 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
2 pages * 54 spans ~ 0.8 MiB; 5.3 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
4 pages * 9 spans ~ 0.3 MiB; 5.6 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
5 pages * 28 spans ~ 1.1 MiB; 6.7 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
6 pages * 8 spans ~ 0.4 MiB; 7.1 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
7 pages * 6 spans ~ 0.3 MiB; 7.4 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
8 pages * 11 spans ~ 0.7 MiB; 8.1 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
9 pages * 4 spans ~ 0.3 MiB; 8.4 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
10 pages * 2 spans ~ 0.2 MiB; 8.5 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
11 pages * 8 spans ~ 0.7 MiB; 9.2 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
12 pages * 1 spans ~ 0.1 MiB; 9.3 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
13 pages * 2 spans ~ 0.2 MiB; 9.5 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
14 pages * 9 spans ~ 1.0 MiB; 10.5 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
15 pages * 3 spans ~ 0.4 MiB; 10.9 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
16 pages * 3 spans ~ 0.4 MiB; 11.2 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
20 pages * 1 spans ~ 0.2 MiB; 11.4 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
21 pages * 1 spans ~ 0.2 MiB; 11.6 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
23 pages * 1 spans ~ 0.2 MiB; 11.7 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
24 pages * 1 spans ~ 0.2 MiB; 11.9 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
25 pages * 2 spans ~ 0.4 MiB; 12.3 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
29 pages * 2 spans ~ 0.5 MiB; 12.8 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
32 pages * 1 spans ~ 0.2 MiB; 13.0 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
33 pages * 1 spans ~ 0.3 MiB; 13.3 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
35 pages * 1 spans ~ 0.3 MiB; 13.5 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
38 pages * 1 spans ~ 0.3 MiB; 13.8 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
39 pages * 1 spans ~ 0.3 MiB; 14.1 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
43 pages * 1 spans ~ 0.3 MiB; 14.5 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
57 pages * 2 spans ~ 0.9 MiB; 15.4 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
64 pages * 1 spans ~ 0.5 MiB; 15.9 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
70 pages * 1 spans ~ 0.5 MiB; 16.4 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
86 pages * 1 spans ~ 0.7 MiB; 17.1 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
>255 large * 4 spans ~ 23.5 MiB; 40.6 MiB cum; unmapped: 19.4 MiB; 19.4 MiB cum
SERVER 2
/opt/couchbase/bin/cbstats 127.0.0.1:11210 -b CacheBase allocator
NOTE: SMALL MEMORY MODEL IS IN USE, PERFORMANCE MAY SUFFER.
------------------------------------------------
MALLOC: 1506611864 ( 1436.8 MiB) Bytes in use by application
MALLOC: + 16457728 ( 15.7 MiB) Bytes in page heap freelist
MALLOC: + 80686528 ( 76.9 MiB) Bytes in central cache freelist
MALLOC: + 0 ( 0.0 MiB) Bytes in transfer cache freelist
MALLOC: + 4849576 ( 4.6 MiB) Bytes in thread cache freelists
MALLOC: + 7631000 ( 7.3 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 1616236696 ( 1541.4 MiB) Actual memory used (physical + swap)
MALLOC: + 21929984 ( 20.9 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 1638166680 ( 1562.3 MiB) Virtual address space used
MALLOC:
MALLOC: 102984 Spans in use
MALLOC: 17 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
------------------------------------------------
Size class breakdown
------------------------------------------------
class 1 [ 8 bytes ] : 935 objs; 0.0 MiB; 0.0 cum MiB
class 2 [ 16 bytes ] : 706 objs; 0.0 MiB; 0.0 cum MiB
class 3 [ 32 bytes ] : 187428 objs; 5.7 MiB; 5.7 cum MiB
class 4 [ 48 bytes ] : 87653 objs; 4.0 MiB; 9.8 cum MiB
class 5 [ 64 bytes ] : 56225 objs; 3.4 MiB; 13.2 cum MiB
class 6 [ 80 bytes ] : 1161 objs; 0.1 MiB; 13.3 cum MiB
class 7 [ 96 bytes ] : 2268 objs; 0.2 MiB; 13.5 cum MiB
class 8 [ 112 bytes ] : 184 objs; 0.0 MiB; 13.5 cum MiB
class 9 [ 128 bytes ] : 599 objs; 0.1 MiB; 13.6 cum MiB
class 10 [ 144 bytes ] : 224 objs; 0.0 MiB; 13.6 cum MiB
class 11 [ 160 bytes ] : 616 objs; 0.1 MiB; 13.7 cum MiB
class 12 [ 176 bytes ] : 79 objs; 0.0 MiB; 13.7 cum MiB
class 13 [ 192 bytes ] : 519 objs; 0.1 MiB; 13.8 cum MiB
class 14 [ 208 bytes ] : 66 objs; 0.0 MiB; 13.8 cum MiB
class 15 [ 224 bytes ] : 96 objs; 0.0 MiB; 13.8 cum MiB
class 16 [ 240 bytes ] : 75 objs; 0.0 MiB; 13.9 cum MiB
class 17 [ 256 bytes ] : 107 objs; 0.0 MiB; 13.9 cum MiB
class 18 [ 288 bytes ] : 138 objs; 0.0 MiB; 13.9 cum MiB
class 19 [ 320 bytes ] : 136 objs; 0.0 MiB; 14.0 cum MiB
class 20 [ 352 bytes ] : 74 objs; 0.0 MiB; 14.0 cum MiB
class 21 [ 384 bytes ] : 357 objs; 0.1 MiB; 14.1 cum MiB
class 22 [ 416 bytes ] : 61 objs; 0.0 MiB; 14.1 cum MiB
class 23 [ 448 bytes ] : 75 objs; 0.0 MiB; 14.2 cum MiB
class 24 [ 480 bytes ] : 37 objs; 0.0 MiB; 14.2 cum MiB
class 25 [ 512 bytes ] : 465 objs; 0.2 MiB; 14.4 cum MiB
class 26 [ 576 bytes ] : 398 objs; 0.2 MiB; 14.6 cum MiB
class 27 [ 640 bytes ] : 38 objs; 0.0 MiB; 14.7 cum MiB
class 28 [ 704 bytes ] : 84 objs; 0.1 MiB; 14.7 cum MiB
class 29 [ 768 bytes ] : 50 objs; 0.0 MiB; 14.8 cum MiB
class 30 [ 896 bytes ] : 455 objs; 0.4 MiB; 15.1 cum MiB
class 31 [ 1024 bytes ] : 127 objs; 0.1 MiB; 15.3 cum MiB
class 32 [ 1152 bytes ] : 119 objs; 0.1 MiB; 15.4 cum MiB
class 33 [ 1280 bytes ] : 341 objs; 0.4 MiB; 15.8 cum MiB
class 34 [ 1408 bytes ] : 126 objs; 0.2 MiB; 16.0 cum MiB
class 35 [ 1536 bytes ] : 91 objs; 0.1 MiB; 16.1 cum MiB
class 36 [ 1792 bytes ] : 468 objs; 0.8 MiB; 16.9 cum MiB
class 37 [ 2048 bytes ] : 203 objs; 0.4 MiB; 17.3 cum MiB
class 38 [ 2304 bytes ] : 118 objs; 0.3 MiB; 17.6 cum MiB
class 39 [ 2560 bytes ] : 470 objs; 1.1 MiB; 18.7 cum MiB
class 40 [ 2816 bytes ] : 255 objs; 0.7 MiB; 19.4 cum MiB
class 41 [ 3072 bytes ] : 186 objs; 0.5 MiB; 19.9 cum MiB
class 42 [ 3328 bytes ] : 235 objs; 0.7 MiB; 20.7 cum MiB
class 43 [ 4096 bytes ] : 192 objs; 0.8 MiB; 21.4 cum MiB
class 44 [ 4608 bytes ] : 148 objs; 0.7 MiB; 22.1 cum MiB
class 45 [ 5120 bytes ] : 122 objs; 0.6 MiB; 22.7 cum MiB
class 46 [ 6144 bytes ] : 97 objs; 0.6 MiB; 23.3 cum MiB
class 47 [ 6656 bytes ] : 56 objs; 0.4 MiB; 23.6 cum MiB
class 48 [ 8192 bytes ] : 25 objs; 0.2 MiB; 23.8 cum MiB
class 49 [ 9216 bytes ] : 35 objs; 0.3 MiB; 24.1 cum MiB
class 50 [ 10240 bytes ] : 21 objs; 0.2 MiB; 24.3 cum MiB
class 51 [ 12288 bytes ] : 15 objs; 0.2 MiB; 24.5 cum MiB
class 52 [ 13312 bytes ] : 11 objs; 0.1 MiB; 24.6 cum MiB
class 53 [ 16384 bytes ] : 6 objs; 0.1 MiB; 24.7 cum MiB
class 54 [ 20480 bytes ] : 5 objs; 0.1 MiB; 24.8 cum MiB
class 55 [ 24576 bytes ] : 3 objs; 0.1 MiB; 24.9 cum MiB
class 56 [ 26624 bytes ] : 7 objs; 0.2 MiB; 25.1 cum MiB
class 57 [ 32768 bytes ] : 6 objs; 0.2 MiB; 25.3 cum MiB
class 58 [ 40960 bytes ] : 1 objs; 0.0 MiB; 25.3 cum MiB
class 60 [ 57344 bytes ] : 1 objs; 0.1 MiB; 25.4 cum MiB
class 64 [ 90112 bytes ] : 1 objs; 0.1 MiB; 25.4 cum MiB
class 67 [ 114688 bytes ] : 1 objs; 0.1 MiB; 25.6 cum MiB
class 73 [ 163840 bytes ] : 3 objs; 0.5 MiB; 26.0 cum MiB
------------------------------------------------
PageHeap: 31 sizes; 15.7 MiB free; 20.9 MiB unmapped
------------------------------------------------
1 pages * 373 spans ~ 2.9 MiB; 2.9 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
2 pages * 50 spans ~ 0.8 MiB; 3.7 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
3 pages * 1 spans ~ 0.0 MiB; 3.7 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
4 pages * 44 spans ~ 1.4 MiB; 5.1 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
5 pages * 16 spans ~ 0.6 MiB; 5.7 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
6 pages * 11 spans ~ 0.5 MiB; 6.2 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
7 pages * 12 spans ~ 0.7 MiB; 6.9 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
8 pages * 14 spans ~ 0.9 MiB; 7.8 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
9 pages * 6 spans ~ 0.4 MiB; 8.2 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
10 pages * 6 spans ~ 0.5 MiB; 8.7 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
11 pages * 2 spans ~ 0.2 MiB; 8.8 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
13 pages * 8 spans ~ 0.8 MiB; 9.6 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
14 pages * 7 spans ~ 0.8 MiB; 10.4 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
15 pages * 2 spans ~ 0.2 MiB; 10.6 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
16 pages * 2 spans ~ 0.2 MiB; 10.9 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
17 pages * 2 spans ~ 0.3 MiB; 11.2 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
18 pages * 1 spans ~ 0.1 MiB; 11.3 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
20 pages * 1 spans ~ 0.2 MiB; 11.5 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
21 pages * 1 spans ~ 0.2 MiB; 11.6 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
23 pages * 1 spans ~ 0.2 MiB; 11.8 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
24 pages * 1 spans ~ 0.2 MiB; 12.0 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
27 pages * 1 spans ~ 0.2 MiB; 12.2 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
31 pages * 1 spans ~ 0.2 MiB; 12.4 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
32 pages * 1 spans ~ 0.2 MiB; 12.7 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
35 pages * 1 spans ~ 0.3 MiB; 13.0 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
36 pages * 1 spans ~ 0.3 MiB; 13.2 MiB cum; unmapped: 0.3 MiB; 0.3 MiB cum
38 pages * 1 spans ~ 0.3 MiB; 13.5 MiB cum; unmapped: 0.3 MiB; 0.6 MiB cum
39 pages * 1 spans ~ 0.3 MiB; 13.8 MiB cum; unmapped: 0.0 MiB; 0.6 MiB cum
45 pages * 1 spans ~ 0.4 MiB; 14.2 MiB cum; unmapped: 0.0 MiB; 0.6 MiB cum
46 pages * 1 spans ~ 0.4 MiB; 14.6 MiB cum; unmapped: 0.0 MiB; 0.6 MiB cum
73 pages * 1 spans ~ 0.6 MiB; 15.1 MiB cum; unmapped: 0.0 MiB; 0.6 MiB cum
>255 large * 4 spans ~ 21.5 MiB; 36.6 MiB cum; unmapped: 20.3 MiB; 20.9 MiB cum
SERVER 3
/opt/couchbase/bin/cbstats 127.0.0.1:11210 -b CacheBase allocator
NOTE: SMALL MEMORY MODEL IS IN USE, PERFORMANCE MAY SUFFER.
------------------------------------------------
MALLOC: 1503370312 ( 1433.7 MiB) Bytes in use by application
MALLOC: + 32284672 ( 30.8 MiB) Bytes in page heap freelist
MALLOC: + 84729888 ( 80.8 MiB) Bytes in central cache freelist
MALLOC: + 0 ( 0.0 MiB) Bytes in transfer cache freelist
MALLOC: + 5186456 ( 4.9 MiB) Bytes in thread cache freelists
MALLOC: + 7499928 ( 7.2 MiB) Bytes in malloc metadata
MALLOC: ------------
MALLOC: = 1633071256 ( 1557.4 MiB) Actual memory used (physical + swap)
MALLOC: + 8110080 ( 7.7 MiB) Bytes released to OS (aka unmapped)
MALLOC: ------------
MALLOC: = 1641181336 ( 1565.2 MiB) Virtual address space used
MALLOC:
MALLOC: 103195 Spans in use
MALLOC: 17 Thread heaps in use
MALLOC: 8192 Tcmalloc page size
------------------------------------------------
Call ReleaseFreeMemory() to release freelist memory to the OS (via madvise()).
Bytes released to the OS take up virtual address space but no physical memory.
------------------------------------------------
Size class breakdown
------------------------------------------------
class 1 [ 8 bytes ] : 941 objs; 0.0 MiB; 0.0 cum MiB
class 2 [ 16 bytes ] : 1044 objs; 0.0 MiB; 0.0 cum MiB
class 3 [ 32 bytes ] : 177879 objs; 5.4 MiB; 5.5 cum MiB
class 4 [ 48 bytes ] : 77712 objs; 3.6 MiB; 9.0 cum MiB
class 5 [ 64 bytes ] : 57205 objs; 3.5 MiB; 12.5 cum MiB
class 6 [ 80 bytes ] : 1255 objs; 0.1 MiB; 12.6 cum MiB
class 7 [ 96 bytes ] : 1602 objs; 0.1 MiB; 12.7 cum MiB
class 8 [ 112 bytes ] : 245 objs; 0.0 MiB; 12.8 cum MiB
class 9 [ 128 bytes ] : 535 objs; 0.1 MiB; 12.8 cum MiB
class 10 [ 144 bytes ] : 140 objs; 0.0 MiB; 12.9 cum MiB
class 11 [ 160 bytes ] : 610 objs; 0.1 MiB; 12.9 cum MiB
class 12 [ 176 bytes ] : 83 objs; 0.0 MiB; 13.0 cum MiB
class 13 [ 192 bytes ] : 395 objs; 0.1 MiB; 13.0 cum MiB
class 14 [ 208 bytes ] : 109 objs; 0.0 MiB; 13.1 cum MiB
class 15 [ 224 bytes ] : 58 objs; 0.0 MiB; 13.1 cum MiB
class 16 [ 240 bytes ] : 116 objs; 0.0 MiB; 13.1 cum MiB
class 17 [ 256 bytes ] : 102 objs; 0.0 MiB; 13.1 cum MiB
class 18 [ 288 bytes ] : 144 objs; 0.0 MiB; 13.2 cum MiB
class 19 [ 320 bytes ] : 112 objs; 0.0 MiB; 13.2 cum MiB
class 20 [ 352 bytes ] : 75 objs; 0.0 MiB; 13.2 cum MiB
class 21 [ 384 bytes ] : 258 objs; 0.1 MiB; 13.3 cum MiB
class 22 [ 416 bytes ] : 62 objs; 0.0 MiB; 13.3 cum MiB
class 23 [ 448 bytes ] : 79 objs; 0.0 MiB; 13.4 cum MiB
class 24 [ 480 bytes ] : 53 objs; 0.0 MiB; 13.4 cum MiB
class 25 [ 512 bytes ] : 455 objs; 0.2 MiB; 13.6 cum MiB
class 26 [ 576 bytes ] : 365 objs; 0.2 MiB; 13.8 cum MiB
class 27 [ 640 bytes ] : 54 objs; 0.0 MiB; 13.8 cum MiB
class 28 [ 704 bytes ] : 55 objs; 0.0 MiB; 13.9 cum MiB
class 29 [ 768 bytes ] : 40 objs; 0.0 MiB; 13.9 cum MiB
class 30 [ 896 bytes ] : 417 objs; 0.4 MiB; 14.3 cum MiB
class 31 [ 1024 bytes ] : 113 objs; 0.1 MiB; 14.4 cum MiB
class 32 [ 1152 bytes ] : 137 objs; 0.2 MiB; 14.5 cum MiB
class 33 [ 1280 bytes ] : 498 objs; 0.6 MiB; 15.1 cum MiB
class 34 [ 1408 bytes ] : 125 objs; 0.2 MiB; 15.3 cum MiB
class 35 [ 1536 bytes ] : 125 objs; 0.2 MiB; 15.5 cum MiB
class 36 [ 1792 bytes ] : 498 objs; 0.9 MiB; 16.3 cum MiB
class 37 [ 2048 bytes ] : 194 objs; 0.4 MiB; 16.7 cum MiB
class 38 [ 2304 bytes ] : 134 objs; 0.3 MiB; 17.0 cum MiB
class 39 [ 2560 bytes ] : 688 objs; 1.7 MiB; 18.7 cum MiB
class 40 [ 2816 bytes ] : 371 objs; 1.0 MiB; 19.7 cum MiB
class 41 [ 3072 bytes ] : 201 objs; 0.6 MiB; 20.3 cum MiB
class 42 [ 3328 bytes ] : 299 objs; 0.9 MiB; 21.2 cum MiB
class 43 [ 4096 bytes ] : 255 objs; 1.0 MiB; 22.2 cum MiB
class 44 [ 4608 bytes ] : 277 objs; 1.2 MiB; 23.4 cum MiB
class 45 [ 5120 bytes ] : 127 objs; 0.6 MiB; 24.1 cum MiB
class 46 [ 6144 bytes ] : 165 objs; 1.0 MiB; 25.0 cum MiB
class 47 [ 6656 bytes ] : 53 objs; 0.3 MiB; 25.4 cum MiB
class 48 [ 8192 bytes ] : 91 objs; 0.7 MiB; 26.1 cum MiB
class 49 [ 9216 bytes ] : 54 objs; 0.5 MiB; 26.6 cum MiB
class 50 [ 10240 bytes ] : 40 objs; 0.4 MiB; 26.9 cum MiB
class 51 [ 12288 bytes ] : 36 objs; 0.4 MiB; 27.4 cum MiB
class 52 [ 13312 bytes ] : 27 objs; 0.3 MiB; 27.7 cum MiB
class 53 [ 16384 bytes ] : 7 objs; 0.1 MiB; 27.8 cum MiB
class 54 [ 20480 bytes ] : 43 objs; 0.8 MiB; 28.7 cum MiB
class 55 [ 24576 bytes ] : 6 objs; 0.1 MiB; 28.8 cum MiB
class 56 [ 26624 bytes ] : 13 objs; 0.3 MiB; 29.1 cum MiB
class 57 [ 32768 bytes ] : 5 objs; 0.2 MiB; 29.3 cum MiB
class 58 [ 40960 bytes ] : 3 objs; 0.1 MiB; 29.4 cum MiB
class 62 [ 73728 bytes ] : 1 objs; 0.1 MiB; 29.5 cum MiB
class 64 [ 90112 bytes ] : 1 objs; 0.1 MiB; 29.6 cum MiB
class 67 [ 114688 bytes ] : 2 objs; 0.2 MiB; 29.8 cum MiB
class 73 [ 163840 bytes ] : 3 objs; 0.5 MiB; 30.2 cum MiB
------------------------------------------------
PageHeap: 44 sizes; 30.8 MiB free; 7.7 MiB unmapped
------------------------------------------------
1 pages * 358 spans ~ 2.8 MiB; 2.8 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
2 pages * 109 spans ~ 1.7 MiB; 4.5 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
3 pages * 29 spans ~ 0.7 MiB; 5.2 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
4 pages * 108 spans ~ 3.4 MiB; 8.6 MiB cum; unmapped: 0.0 MiB; 0.0 MiB cum
5 pages * 28 spans ~ 1.1 MiB; 9.6 MiB cum; unmapped: 0.0 MiB; 0.1 MiB cum
6 pages * 19 spans ~ 0.9 MiB; 10.5 MiB cum; unmapped: 0.0 MiB; 0.1 MiB cum
7 pages * 10 spans ~ 0.5 MiB; 11.1 MiB cum; unmapped: 0.0 MiB; 0.1 MiB cum
8 pages * 19 spans ~ 1.2 MiB; 12.3 MiB cum; unmapped: 0.0 MiB; 0.1 MiB cum
9 pages * 10 spans ~ 0.7 MiB; 13.0 MiB cum; unmapped: 0.0 MiB; 0.1 MiB cum
10 pages * 7 spans ~ 0.5 MiB; 13.5 MiB cum; unmapped: 0.0 MiB; 0.1 MiB cum
11 pages * 16 spans ~ 1.4 MiB; 14.9 MiB cum; unmapped: 0.0 MiB; 0.1 MiB cum
12 pages * 5 spans ~ 0.5 MiB; 15.4 MiB cum; unmapped: 0.1 MiB; 0.2 MiB cum
13 pages * 6 spans ~ 0.6 MiB; 16.0 MiB cum; unmapped: 0.0 MiB; 0.2 MiB cum
14 pages * 10 spans ~ 1.1 MiB; 17.1 MiB cum; unmapped: 0.0 MiB; 0.2 MiB cum
15 pages * 1 spans ~ 0.1 MiB; 17.2 MiB cum; unmapped: 0.0 MiB; 0.2 MiB cum
16 pages * 3 spans ~ 0.4 MiB; 17.6 MiB cum; unmapped: 0.0 MiB; 0.2 MiB cum
17 pages * 4 spans ~ 0.5 MiB; 18.1 MiB cum; unmapped: 0.0 MiB; 0.2 MiB cum
18 pages * 5 spans ~ 0.7 MiB; 18.8 MiB cum; unmapped: 0.0 MiB; 0.2 MiB cum
19 pages * 2 spans ~ 0.3 MiB; 19.1 MiB cum; unmapped: 0.0 MiB; 0.2 MiB cum
20 pages * 1 spans ~ 0.2 MiB; 19.2 MiB cum; unmapped: 0.2 MiB; 0.3 MiB cum
21 pages * 4 spans ~ 0.7 MiB; 19.9 MiB cum; unmapped: 0.3 MiB; 0.6 MiB cum
22 pages * 6 spans ~ 1.0 MiB; 20.9 MiB cum; unmapped: 0.3 MiB; 1.0 MiB cum
23 pages * 7 spans ~ 1.3 MiB; 22.2 MiB cum; unmapped: 0.5 MiB; 1.5 MiB cum
24 pages * 6 spans ~ 1.1 MiB; 23.3 MiB cum; unmapped: 0.6 MiB; 2.1 MiB cum
25 pages * 6 spans ~ 1.2 MiB; 24.5 MiB cum; unmapped: 0.6 MiB; 2.7 MiB cum
26 pages * 4 spans ~ 0.8 MiB; 25.3 MiB cum; unmapped: 0.4 MiB; 3.1 MiB cum
27 pages * 1 spans ~ 0.2 MiB; 25.5 MiB cum; unmapped: 0.0 MiB; 3.1 MiB cum
28 pages * 2 spans ~ 0.4 MiB; 26.0 MiB cum; unmapped: 0.0 MiB; 3.1 MiB cum
29 pages * 2 spans ~ 0.5 MiB; 26.4 MiB cum; unmapped: 0.2 MiB; 3.3 MiB cum
30 pages * 2 spans ~ 0.5 MiB; 26.9 MiB cum; unmapped: 0.2 MiB; 3.5 MiB cum
31 pages * 1 spans ~ 0.2 MiB; 27.1 MiB cum; unmapped: 0.0 MiB; 3.5 MiB cum
32 pages * 2 spans ~ 0.5 MiB; 27.6 MiB cum; unmapped: 0.0 MiB; 3.5 MiB cum
34 pages * 1 spans ~ 0.3 MiB; 27.9 MiB cum; unmapped: 0.3 MiB; 3.8 MiB cum
35 pages * 1 spans ~ 0.3 MiB; 28.2 MiB cum; unmapped: 0.3 MiB; 4.1 MiB cum
36 pages * 2 spans ~ 0.6 MiB; 28.7 MiB cum; unmapped: 0.3 MiB; 4.4 MiB cum
38 pages * 2 spans ~ 0.6 MiB; 29.3 MiB cum; unmapped: 0.0 MiB; 4.4 MiB cum
39 pages * 1 spans ~ 0.3 MiB; 29.6 MiB cum; unmapped: 0.0 MiB; 4.4 MiB cum
41 pages * 1 spans ~ 0.3 MiB; 29.9 MiB cum; unmapped: 0.0 MiB; 4.4 MiB cum
42 pages * 1 spans ~ 0.3 MiB; 30.3 MiB cum; unmapped: 0.0 MiB; 4.4 MiB cum
44 pages * 1 spans ~ 0.3 MiB; 30.6 MiB cum; unmapped: 0.3 MiB; 4.7 MiB cum
45 pages * 1 spans ~ 0.4 MiB; 31.0 MiB cum; unmapped: 0.0 MiB; 4.7 MiB cum
63 pages * 1 spans ~ 0.5 MiB; 31.5 MiB cum; unmapped: 0.0 MiB; 4.7 MiB cum
65 pages * 1 spans ~ 0.5 MiB; 32.0 MiB cum; unmapped: 0.0 MiB; 4.7 MiB cum
76 pages * 1 spans ~ 0.6 MiB; 32.6 MiB cum; unmapped: 0.0 MiB; 4.7 MiB cum
>255 large * 3 spans ~ 6.0 MiB; 38.5 MiB cum; unmapped: 3.0 MiB; 7.7 MiB cum
Any progress big shooter?
Did you need any more info from me?
I don't need anything else from you. At this time I have asked that our QE team attempt to see if they can reproduce this issue, but like I mentioned before I have never heard of any other user experience problems like you are experiencing. I will report back if they find anything.
I looked through your post and it appears that your bucket is not writing items to disk. Have you disabled persistence? If not can you upload you logs for me to look at? Instructions for how to do this can be found here:
http://www.couchbase.com/wiki/display/couchbase/Working+with+the+Couchba...