Membase performance
Versions: 1.6.5 and 1.6.5.3
2 Membase buckets: 1. 24000 MB and the 2. 1000MB
Quota for Membase set to 25000MB
Dedicated Membase Server 8 Core CPU 32GB RAM
Inserted data total 201 million key/values.
Problem is after 150 million key/values sets performance went very bad down, server started to swap strong and response times were more than bad, mysql can better with the perfect setting.
I thought Membase can handle over 10K QPS but seems it can 1K QPS in a live system where good response times are required. Until 150 million key/values between 3K and 10k without any problems reached, but above went really bad.
Tested both membase releases!
I set the qouta to 25000 MB which is round 75% from the dedicated RAM so should be fine from setting or not.
Now I am thinking to use only memcached bucket but hope I don't get the same surprise like the membase bucket.
If you have suggestions, questions about my experience or share your experience with the membase server are welcome.
Cris
Perry, thank you for your fast response.
Indeed we have very small key/values 16-24 characters, I just finished the load and with 252 million key/values and restarted the membase-server to get rid of the swap. I read with just one server 60 QPS at this moment no problems, I going to put more to read to see.
I really like the membase web control, after restarting the membase-server the web control doesn't work for while.
I will let you know about any changes in the performance.
Cris
Thanks for the response Cris. The web console not working is a side-effect of the server still "warming up" by reading it's data off of disk.
Perry
Hi Perry,
I tried also to set the limit on the virtual memory with the ulimit -v in the membase start script and caused a memcached process crash.
Finally after serveral testing with a many items ( over 120mil. items ), we got it very stable. I think the problem is just to find the right balance of bucket RAM and available dedicated RAM.
I set it to 50% of the dedicated RAM and it stable and very fast on set`s/get`s 3,5K QPS without any problems on the performance.
Hope my testing and finding are helping you and other membase users.
Thanks,
Cris
Thanks Cris. We've been finding a few cases where our memory accounting is less than accurate which leads to going into swap and other instabilities so we're focusing on that as we speak.
Thanks again for your cooperation and patience, let me know if there's anything else I can do to help.
Perry
Cris, thanks for your feedback.
We do have some known issues around our memory allocation management that can cause Membase to go into swap. At that point, "all bets are off" and performance will definitely degrade.
How large were your items? Our problems come when dealing with many small items.
It would be great to hear if you were able to achieve sustainable performance when the system was not in swap...either by decreasing the RAM quota, adding a second node or just limiting the number of items you put in for testing.
Membase can definitely handle well over 10k ops/sec...unfortunately OS swapping gives us very bad results.
Perry
Forum support is great for free but sometimes you need a guaranteed response time and dedicated resources for your questions or issues.
Consider purchasing enterprise-level support from Couchbase: http://www.couchbase.com/products-and-services/overview
Call or email "sales -at- couchbase-dot- com" today!