Temp OOM's on 1.8.1
Hi,
We are using 1.8.1 Enterprise for eval and finding that when memory is getting tight on Couchbase we start to get a lot of Temp_OOM and then things go badly in our application.
We have fairly large values at ~30KByte each, and are generating new k/v at a max of 150 per second, and doing a get (and touch - resetting TTL) on ~150/s as well.
We have sized the bucket with 180GB RAM and 1.5TB of SSD for persistence.
I am a bit baffled why we can't sustain even 150 sets/sec and 150 gat/sec as if we switch back to a simpler back end for k/v like leveldb we can get 45K/s without issues on the same SSD back end.
We are using the C libcouchbase client code as well - should it be handling these temp_oom issues better?
For testing two nodes only. Each with 96GB of RAM with 90GB dedicated to CB. 6*512GB SSD's in R10 as the persistence store. Each machine 2 8 core Xeon 5500 series, 2.66Ghz.
Machined are hovering around 1% CPU usage - not working hard. IOstat reports R10 doing ~6MB/s reads and writes.
Here is the memory info, this is after a restart and no oom's yet. It seems 1.8.1 might be helping in this regard, as prior to 1.8.1 we'd see RAM utilization go to 100% of the machine and some swap, where now it seems to be abiding by the high/low water marks.
root@mc1:~# /opt/couchbase/bin/cbstats 127.0.0.1:11210 raw memory
ep_kv_size: 52926784818
ep_max_data_size: 96636764160
ep_mem_high_wat: 72477573120
ep_mem_low_wat: 57982058496
ep_mem_tracker_enabled: true
ep_oom_errors: 0
ep_overhead: 52433558
ep_tmp_oom_errors: 0
ep_value_size: 51810972929
mem_used: 58433355304
tcmalloc_current_thread_cache_bytes: 3199672
tcmalloc_max_thread_cache_bytes: 4194304
tcmalloc_unmapped_bytes: 18669568
total_allocated_bytes: 58435581824
total_fragmentation_bytes: 3730106496
total_free_bytes: 11583807488
total_heap_bytes: 73749495808
thanks for the info. Are you able to gather those ram memory stats when hitting the OOMs and post back. Would be interesting to see total_fragmentation_bytes as a percentage of total_allocated_bytes as well as tcmalloc #s when temp_ooms are hit.
Note that currently, none of the clients handle temp OOM for you. Depending on what you're doing, you'll either not proceed with the operation or do an exponential backoff and retry in the event of a temp OOM.
There's a good example of this in the Java bulk loading example:
http://www.couchbase.com/wiki/display/couchbase/Couchbase+Java+Client+Li...
What does your app do in the case of a tmp oom?
Katana,
I will certainly try to do this, but with the 1.8.1 high watermark now appearing to work for us we're no longer seeing Temp_OOM's.
It appears that under 1.8.0 the high watermark was not enforced as the GUI graphs show us clearly bouncing against 180GB in use, where now we see a saw-tooth pattern for mem used between high and low watermarks during continuous stress tests.
Ingenthr,
Thanks, we will use that to model our code after. Right now our app crashes and burns when it encounters the temp_oom's but we'll fix that. It was using the APR_memcache client and we've ported it over to libcouchbase C SDK so the above change should help to stabilize it greatly.
That's very cool. Is it Open Source by chance? Perhaps somewhere on github? I'd like to link to it if so.
As you've already noted, the memory accounting is much better in 1.8.1 and that should make it much easier to build a more reliable service.
As you've already noted, the memory accounting is much better in 1.8.1 and that should make it much easier to build a more reliable service.
Yes, just doing the upgrade fixed our immediate issue which is always nice. It seems not just the accounting and reporting but under the hood Couchbase is doing something differently under 1.8.1 than 1.8 WRT to memory management?
We're adding a new cache-type back end to a OSS project called "Mapcache". Right now it speaks sqllite, bdb, memcache, tokyotyrant, and a few disk caches. We're working to make couchbase a first-class citizen of the project. We're using it to store small image chips to accelerate web based mapping applications.
When it's all polished and working we'll give it back and let you know where to find it.
The main parts of how memory is managed have not changed, but there's tighter integration with tcmalloc and it's hooks for getting information on memory allocation actions. There may have been some other changes to some of the memory management against quota, but it's mostly fixing of issues found in 1.8.0.
I look forward to the update on Mapcache. Thanks!
I look forward to the update on Mapcache. Thanks!
One more question: can you point me to any documents with the API response codes all listed? I see in the Java example you point out ETMPFAIL being pointed out, but what is EBUSY?
Thanks
One more question: can you point me to any documents with the API response codes all listed? I see in the Java example you point out ETMPFAIL being pointed out, but what is EBUSY?
Each client library may expose errors from the server differently. Since you're using libcouchbase, all of the possible return error codes should be documented in the header file. See include/libcouchbase/couchbase.h and include/libcouchbase/types.h. It doesn't appear the error codes are all that clear for set though. I've filed http://www.couchbase.com/issues/browse/CCBC-87 to get that handled.
30k value size seems slightly on the larger side. Assuming you are not doing any append or prepend operations, the temp_ooms seem a little bit strange. You should be able to get much better Ops/sec than what you are seeing. Can you share your cluster sizing details (h/w, # of nodes etc)
maybe worth digging a little deeper into the memory allocator to see what's going on
/opt/couchbase/bin/cbstats 127.0.0.1:11210 ‐b bucket ‐0 ‐p password raw memory