couchbase stops ejecting keys to disk, hits quota, rejects new creates
Hello,
I have a problem where a single-node couchbase 1.8.1 cluster works well for some time. For example, under a load testing scenario I ran for about an hour at 250-400 creates per second. The disk write queue kept up well. However, as the bucket began to use up it's RAM quota it became slower. At some points the RAM usage went up and down, hovering around the high water mark.
However, eventually it exceeded the high water mark and never backed down. The disk write queue went to zero. Evictions cycled from 0 to 600 over the length of the test, and then went to zero (where it stayed.) Temp OOM errors went high. Our software is configured to perform exponential backoff as per the guidance found on your site. After a certain wait it will discard the write attempts.
When load was removed, the memory stayed high. It very, very slowly started to drain. After 20 minutes ejections per second went back up, on the order of several hundred per second. This dropped off until it was on the order of a few dozen, then less than 25 per second, and finally fell to zero again.
I'm trying to understand why couchbase hits this wall. It doesn't appear to be I/O throughput, and there's very little reading of data going on during the load test. The system itself is able to cope with the load very well, and Couchbase appears to keep up with the persistence load fine. It simply fails to eject keys at a fast enough rate at some point, and then appears to stop even trying.
We are using it to capture real-time trace information, and it's very important that couchbase be able to drain it's bucket in a timely enough manner that we don't lose large sections of trace data.
Any guidance would be appreciated.
Yes. It is a single-node cluster, there are over 3 million items in the vbucket.
I did discover the root cause, after wading through the documentation and various other Internet resources. The problem is that the metadata becomes so large that I can no longer insert keys into the vbucket. Since metadata must always be kept in memory for every single key, eventually memory usage for metadata begins to dominate the vbucket and it's impossible to write to it anymore.
This is very annoying. I should be able to have as many keys in the database as disk space allows. Only metadata for the active keyset should be kept in RAM. If I access a flushed key then I have to pay the price for that. It is much better than going into temp OOM and having no idea why.
That was actually my guess to what the problem might have been and I understand your frustration with the fact that we keep meta data in memory. We plan on removing this requirement in a future release of Couchbase, but it is not a trivial thing to do. We will also have alerts in Couchbase 2.0 that will warn you when meta data is getting close to filling memory.
Great! Well, it will be nice to see those alerts when we upgrade to 2.0.
I understand that it's not a trivial change and has a substantial performance impact. Thanks for looking at the post and your work on the project!
Can you provide information about your cluster? How many nodes? How much memory? How many items you have in the cluster?