disk > memory problems
It would appear that the "disk > memory" features aren't exactly working.
Tested on community & Enterprise, 1.6.5 & 1.6.4.1 - results are all the same.
1) Create a bucket with a memory limit. (We are also setting the port)
2) Have an insert speed of more than ~ 100/sec and fill it.
3) Watch Rome burn.
(This is using the latest spymemcached with vbucket support, but I don't believe that has anything to do with it)
All joking aside, what we're finding is that at any reasonable insert speed (for my test this was around 10k/sec) it would appear that the thread that is supposed to be evicting things based on the high and low water marks has a problem and eventually stops running altogether.
At first you will see large eviction events happen as memory usage crosses the mem_high_wat mark, which of course reduces the in-use memory, but they get progressively smaller and smaller over time and come nowhere close to the mem_low_wat setting. This of course causes the bucket to fill given a constant insert rate. Once the bucket gets to 100% the eviction events are no longer happening and the thing just starts throwing the temp OOM errors to the client. Insert rate drops to say ... 5/sec (yes five) and it just stumbles along ... forever. Stopping inserting has no effect, the bucket will never work again. The only way we are able to get out of that state is to increase the bucket size to where the current amount of data ends up being below the newly calculated mem_low_wat.
Adjusting the mem_low_wat and the mem_high_wat values do not change the eventual outcome. In fact, there also seems to be a problem with the math when doing so. For example, on a two node cluster using flushctl to set one of those values to 180000000 results in the setting changing to 322286400
Performing the same steps, but with an insert rate of ~ 100/sec ... works. The number of items in each large eviction event doesn't decrease over time, and the resulting memory use decreases to the mem_low_wat mark. I have a test running against the community 1.6.5 that has been running for hours - each time the mem_high_wat mark is crossed, a major eviction occurs and life is good.
However, once I stopped inserting I could trigger the same behavior exhibited in the other test by reducing the mem_high_wat mark to below the current in-use. What occurred was that evict events started up but each time were progressively smaller, and never even coming close to reducing the mem_used to the mem_low_wat. Again, this was while doing no inserts or reads - this was while the bucket was completely idle.
I am able to consistently replicate this behavior by completely removing membase, doing a fresh install of any of the versions noted above, and performing the same actions as described.
- Brian Roach
Hi guys. I can confirm this from the logs, but at a first glance it appears that you may be using up all your available RAM with the metadata of keys. We keep all of our metadata/indexes in memory for extremely fast lookups and best performance, but that means that it's possible to use all the RAM for just metadata which will dramatically impact performance. It's also a problem since we can't eject the metadata to disk so the only solution is to add more RAM (which it looks like you've discovered).
Take a look at this wiki page: http://wiki.membase.org/display/membase/Sizing+Guidelines. It talks about the calculations for appropriately sizing a cluster based upon the various factors. For instance, 770 million keys with 1 replica (so actually 1.54 billion keys) will require a few hundred gigs of memory just for metadata and then you'll want more RAM to properly keep your active dataset in memory. I realize that it seems like a lot of memory, but if you look at other solutions, they actually require MORE memory on a per-item basis than Membase does.
I agree that the system can probably be enhanced to handle this particular situation better, but the fact remains that you are reaching a hard memory limit.
Let me know if that makes sense or if something else is going on here.
Perry
I actually suspected this would be your answer, but the math doesn't work unless your metadata is fairly large for each key.
On a 6 node 300G cluster we had a 130G bucket.
The data in question came out to be ~ 750mm keys (1.5B with replication).
The largest keys are 32 bytes (I didn't bother breaking it down to the average key size, going for a worst-case here) which gives you a memory footprint of 44G for the keys.
How much metadata are we talking about per key?
EDIT: 120 bytes? Ouch.
- Roach
Perry,
I suspected it might be the metadata. That's a pretty huge cost for any system with lots of keys. For now we are going to greatly reduce the data set. It would definitely be helpful to be able to monitor the memory usage of the metadata to know when more memory should be added to the system.
Thanks for the help,
Dan
Yeah, I agree that it's a big cost...it's what you pay for the best performance around ;-)
We've actually got some higher-level tasks on the roadmap to bring down the per-item overhead. We've actually got one customer whose metadata takes up more space than their actual dataset!
As I mentioned before though, as far as I know, Membase has one of the lowest per-item memory overheads of any solution out there...other than standard memcached. Please correct me if I'm wrong as I certainly don't want to be portraying incorrect information.
One of the huge benefits of Membase is that it is a completely distributed system and so even though you may need a large amount of RAM to store the data, you're not limited in any other way. Simply adding more nodes or more RAM to the cluster let's you continue to scale...theoretically infinitely but we're happy with a few thousand nodes right now ;-) Any system that contains its lookup table, hash table, etc in one place has inherent limitations.
As far as monitoring, it's a fairly straightforward calculation from the number of keys to the amount of metadata contained and I believe that out 'ep_overhead' statistic gives you a more exact amount of memory being used for metadata.
Thanks for your feedback guys, let me know if there's anything else I can do to help.
Perry
Also related: On our production 6 node cluster we inserted 770 million keys (at 30k a sec) into a membase bucket with a dedicated port. The UI started reporting bizarre negative values for resident % and resident replica %. After the loading finished the bucket sat there doing disk fetches and nothing else.