ep_total_cache_size continues to grow and never shrinks
There is an interesting situation we've run into.
It appears that the total_cache_size never shrinks
:~> /opt/membase/bin/ep_engine/management/stats localhost:11212 all | egrep "kv_size|overhead|cache|data_size|ep_mem|flush"
cmd_flush: 0
ep_flush_duration: 1
ep_flush_duration_highwat: 7379
ep_flush_duration_total: 1914739
ep_flush_preempts: 0
ep_flusher_state: running
ep_flusher_todo: 0
ep_item_flush_expired: 935995444
ep_item_flush_failed: 0
ep_kv_size: 58298937055
ep_max_data_size: 252287385600
ep_mem_high_wat: 189215539200
ep_mem_low_wat: 151372431360
ep_overhead: 403806016
ep_total_cache_size: 813081556151
tap_flush_received: 1
tap_flush_sent: 13
The bucket has a memory quota of 235GB and the web client reports that we are only using about 59GB. Every key in the bucket has an expire of 36 hours. The resident rate is 100% but the replica rate bounces between 0%-30% throughout the day which makes no sense because we never go above 40% space used. The buckets goes between 20mm-30mm items throughout the day.
Looking at: http://techzone.couchbase.com/wiki/display/membase/Growing+Data+Sets+Bey...
Is the ep_total_cache_size used in determining when to kick out replicas? Adding ep_overhead + ep_kv_size is still way below both high_wat and low_wat settings.
This is with 1.6.5. Is there way to manually reset the ep_total_cache_size or will I need to flush the bucket and reload the data? I can for this particular bucket since we have it archived, it's just a pain.
Ah, that mem_used stat isn't on the github page, which is probably why I missed it. Doing some math it looks like mem_used = ep_kv_Size + ep_overhead as I suspected.
On my particular bucket the mem_used is well below both water marks and definitely well below ep_max_data_size, so I'll keep looking into why the resident replica rate keeps shrinking to 0% throughout the day even though there are still 20 million keys in the bucket. The key metadata size can't be the culprit since there just isn't that many keys.
Side note, after some checking it looks like the ep_total_cache_size continuously growing issue is already known: http://www.couchbase.org/issues/browse/MB-3397
Cheers,
Dan
Just an update for anyone who stumbles upon this thread. I woke up this morning to find the replica resident rate in the web ui at -10%. Obviously I can't trust that value. I've since loaded a few million keys and it's grown back to 8%. 21.8 million keys, 43.7 million total. Since unique items show up as half of total items, so I believe the replicas exist, there is probably just a bug with the stats themselves.
I sent this question around to some of our engineers and they have confirmed that these stats issues are know issues for Membase 1.6.x. They should be resolved for 1.7 and we currently have a developer preview out for 1.7 that you can download here:
Thank you Mike, I'll grab the developer release to try on our dev cluster.
Dan
The stat called "mem_used" is what determines when to eject replicas (at the low water mark) and eventually active data (at the high water mark). You might also want to take a look at this page for a more detailed description of all the statistics: https://github.com/membase/ep-engine/blob/master/docs/stats.org
Forum support is great for free but sometimes you need a guaranteed response time and dedicated resources for your questions or issues.
Consider purchasing enterprise-level support from Couchbase: http://www.couchbase.com/products-and-services/overview
Call or email "sales -at- couchbase-dot- com" today!