Evaluating the performance of membase cluster
As a part of engineering requirement in our organization, I am evaluating the performance of various cache management systems.
Currently I have 3 nodes in the cluster running Membase server version 1.7.0. and spymemcached 2.7.3 client.
I am currently noticing some exception like these when the concurrency on the client side is high. The same operations succeeds if the concurrency is low.
2011-11-07 21:43:08.478 WARN net.spy.memcached.protocol.binary.BinaryMemcachedNodeImpl: Operation canceled because authentication or reconnection and authentication has taken more than one second to complete.
Nov 7, 2011 9:43:08 PM flipkart.platform.rabbitmq.benchmark.CachingBroker get
SEVERE: Get for key:0000000000900787 returning null
java.lang.RuntimeException: Exception waiting for value
at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1183)
at net.spy.memcached.MemcachedClient.get(MemcachedClient.java:1200)
Can someone please help me with this.
Thanks for the reply.
You are correct about the GC. After increasing the heap size for the client and also optimizing some memory usage in my client code, I am not seeing the time out.
I am currently performing the benchmark using a 3 node cluster and 4 clients. I am noticing that network bandwidth is getting saturated very early. (We have 1Gbps link)
Our current requirement is something of the sort
100KB to 200KB objects with latency of 5ms to 7ms.(Latency on the client side).The average get operations can be between 10000 per second to 30000 per second. (On the server side)
One of the options to get the above SLA is to have more nodes in the cluster. So I was wondering are there any benchmark results that are available about how much horizontal scaling can be done on Membase.
Also Can you please let me know about the performance difference b/w Membase buckets with replication factor of 2 and Memcache buckets.
I've seen results and demonstrated to myself that it should scale quite nicely as you add additional nodes. Architecturally it makes sense. The underlying design very simply distributes across the nodes in the cluster. I don't have anything specific to point to though.
There shouldn't be a significant throughput or latency difference between Membase buckets and memcache buckets as long as you have enough network IO. Obviously the replication does take IO. There is a bit more memory overhead per item in Membase buckets, but that's offset by not having to keep everything in memory. The replication is done on a background thread.
I don't see all of the output, but my suspicion is that you're seeing so many timeouts, the continuous timeout threshold is being hit and the connection is being dropped and reestablished. When you do so, it takes too long for the authentication to complete.
The client does nearly everything asynchronously, and I strongly suspect what's happening is your load is going up under the higher concurrency. Then (if you haven't tuned garbage collection) you're hitting lots of timeouts due to GC.
There are a few solutions, depending on your goals. Since you said you're evaluating cache management, I presume you do actually want to ensure response times within a specific bound. That being the case, you should probably look to doing some JVM tuning if you haven't already. You can make the JVM consume CPU time for much smaller periods of time (trading off latency for throughput and consuming more CPU time). That may get you to the level you need.
What have you done in this area already?