Membase not using allocated cache
We're experiencing an issue with Membase, where even though we have a bucket with a large cache (12000MB), it is consistently reporting 0% usage even if the memcache servers are under heavy load. Also, under Analytics, the "Total Items" graph is averaging at about 8-10 items, when it should be averaging somewhere in the thousands. It just appears that Membase is not utilizing the available cache size.
We have tried restarting the Membase service and deleting and recreated the "default" bucket to no effect. We also tried to edit the config file in "/etc/opt/Membase/1.0.3/" and added the "-m" switch to try to set the maximum cache size which did not work.
We're running CentOS 64-bit on a high-memory EC2 server (~18GB available memory).
Attached are:
memcache_stats.txt - "stats" command results from memcached
membase_diag.txt - "diag" output from Membase with IP address redacted
membase_bucket.jpg - Screenshot of bucket screen with 0% cache used
membase_analyt.jpg - Screenshot of analytics page with all graphs
Thanks for the reply. That seems to explain the cache usage. I guess the "Items" graph was throwing me off. I'll look out for the new update.
We're using the standard PHP Memcache client library to interface with the Northscale server. We actually started to look into the usage issue when we started experiencing Memcache connection issues from the PHP client library:
PHP Notice: MemcachePool::add(): Server [INTERNAL DNS REMOVED] (tcp 11211, udp 0) failed with: Connection refused (111) in ...
Would you recommend changing the configuration to up the max connections to the Northscale server (I believe it defaults to 1024) or would this be a symptom of another problem?
Thanks,
Joel
Joel, the new update is available now at our download site.
That behavior certainly is a symptom of connections, but looking at your stats it doesn't seem that you would be reaching that limit (it's currently at 88 connections, and has only seen a total of 3k over the uptime of this instance).
How often do you see these? We've seen some undesirable behavior dealing with TCP connections in EC2 so it may be related to that.
Another advantage of the new version is the ability to use a client-side Moxi process. Moxi is a Membase/memcached proxy that allows standard memcached clients to speak to a process on localhost which then handles both pooling the connections (especially useful for PHP) and will also handle dynamically updating itself when the topology of the cluster changes. It is also a more "robust" connection to the cluster with retries and reconnects.
I'll be happy to dig into that further with you if you like. There is documentation available in our Membase user guide, and I have a recorded webinar on the use of Moxi available here: [url]http://www.northscale.com/products/webinars.html[/url]
Thanks Joel, take care.
Perry
Joel, what language and client library are you using? From the stats and log output, everything appears to be working properly and the UI is reporting the correct information.
-You only have about 11mb of data in the cache. That's a very small percentage of your allocated RAM and it rounds off to 0%.
-I will double check, but I believe there is a known issue in this release where the "Items" graph is incorrectly reporting a rate (items per second) rather than a constant value. According to the stats, you have 3635 items in the cache currently. This should be fixed in our latest version (released today!!!) and I highly encourage you to download it and try it out.
Hope that helps, let me know what else I can do for you.
Perry
Forum support is great for free but sometimes you need a guaranteed response time and dedicated resources for your questions or issues.
Consider purchasing enterprise-level support from Membase: http://www.membase.com/products-and-services/overview
Call or email "sales -at- membase -dot- com" today!