Membase re-reading data into memory
I'm currently setting up membase to serve our user's avatar images. There are approximately 500gb of images.
I've set the RAM quota to 2GB, and disk space unlimited (the box has 1.5T diskspace available). After this avatars project is live I plan to move the session handler to membase, change it into a cluster of at least 2 machines with commercial support.
I have 2 questions:
1) What should I set HIGH and LOW memory to?
2) What happens when an entry is read from disk, in what cases will and will it not be stored in memory? Will it be stored at all?
I must say it was a bit annoying to notice that these memory settings cannot be set in the web interface, I supposed these values would have some smart defaults. Half way finished my script to import these files from filesystem stop inserting them due to 'RAM FULL' error message.
Thanks,
Michaël
Hi Perry,
Thank you very much for your reply. I will implement the error handling as described on http://code.google.com/p/memcached/wiki/BinaryProtocolRevamped#Response_... and http://wiki.membase.org/display/membase/Moxi+Error+Responses#MoxiErrorRe... - 1 thing I do not understand, though, in the memcached protocol you talk about 2 byte error codes, in the moxi documentation these 2 bytes error code is missing. On which should I rely?
Also, looking at ngrep, I see 2 different kinds of errors on this bucket. I assume it is better to read the ascii error message:
SERVER_ERROR a2b out of memory..
SERVER_ERROR temporary failure..
No 2 bytes error code there at all I suppose, unless 'a2b' is some ngrep translation of an unknown character..
Again, thanks very much for your response about re-reading items into memory; this eye opener made the process is clear to me now. I suppose our avatars bucket is indeed full with metadata, it has 15251074 entries (average variable length 40 characters or something). The bucket ram size is 2GB. Is anything else saved in metadata? I will keep it into consideration to keep variable size as short as possible, as I have to re-insert them now anyways. Also, is there a command to show how many items are currently in memory to be sure the meta data is filling it up so much?
Thanks,
Michaël
Michael, take a look at this wiki page: http://wiki.membase.org/display/membase/Sizing+Guidelines which describes the various calculations that need to be made to properly size a cluster. For instance, that number of items (x2 actually with one replica) will require about 6gb of RAM just for metadata. We basically require about 150 bytes of RAM for each item, plus the length of the actual key. I realize it can seem like quite a lot, but it's actually less per-ite memory required than most other solutions and allows us to provide the best possible performance.
If you're running 1.6.5, the UI has various statistics to display how many active items and how many replica items are "resident" in memory and looking at the lower level statistics (/opt/membase/bin/ep_engine/stats <ip>:11210 all) can show you more details about the state of the system. I believe the 'ep_overhead' stat describes how much memory is being used for metadata.
Perry
Hi Perry,
Thank you very much for the satisfying answer again. Will take that in consideration when building the rest of the cluster!
Michaël
Hi Michael, thanks for the feedback. We're looking into a number of ways to expose these settings better to the user.
We set the high and low "watermarks" to pretty good defaults in an attempt to leave the system enough RAM to handle any spikes in traffic while still holding enough data in RAM for the best performance. You may want to take a read through this wiki page: http://wiki.membase.org/display/membase/Growing+Data+Sets+Beyond+Memory
You can certainly raise and lower those numbers manually, though I would caution against setting the high watermark too high as the system can misbehave if it doesn't have enough memory to allocate when needed.
To answer your second question, when an item is read from disk, it is held in RAM until the system needs to "eject" it back to disk to reclaim more memory. At the moment, the system handles all this automatically and there isn't really a good way to control it yourself. We do have some tools to eject specific items, but I don't know if that meets your needs here.
Any "client" (including your bulk-load script) should be able to handle our "temporary out of memory" error. The idea is that the client receives this and backs off to give the system enough time to continue writing data to disk and then reclaim memory when available. If you get an "out of memory" error (note, not temporary), that means that the system is unable to reclaim memory at all and you'll need to take some other action. This usually happens if we end up failing to write to the disk for a long time and the queue backs up, or if there are so many items within the database that the metadata alone takes up all the avialable memory.
Hope that helps, let me know if you run into any other issues.
Perry
Forum support is great for free but sometimes you need a guaranteed response time and dedicated resources for your questions or issues.
Consider purchasing enterprise-level support from Couchbase: http://www.couchbase.com/products-and-services/overview
Call or email "sales -at- couchbase-dot- com" today!