Performance comparison membase vs memcached

Looks like the images disappeared from the post.

So here are the links:

Memcache Get:
https://lh5.googleusercontent.com/-Ce0JMpeYwjs/TkqlZQBREbI/AAAAAAAAVBg/M...

Membase Get:
https://lh5.googleusercontent.com/-0Z526B5RY2k/TkqlZDHypUI/AAAAAAAAVBY/i...

Memcache Put:
https://lh5.googleusercontent.com/-NLi6CAXBI-I/TkqlZAD6EtI/AAAAAAAAVBc/X...

Membase Put:
https://lh6.googleusercontent.com/-A3EBCHzAuHo/TkqlZYF-AHI/AAAAAAAAVBk/q...

The series on the charts represent test cases with the different amount of concurrent threads: 1, 2, ..., 256.

Nikolay, can you share your spymemcached configuration and version?

membase-server-enterprise_x86_64_1.7.1
spymemcached-2.6

MemcachedClient client = new MemcachedClient(Arrays.asList(address));

Can you give me a bit more on how you're creating the Arrays.asList(address)? Specifically, what does the "address" look like?

InetSocketAddress address = new InetSocketAddress(server, port);
MemcachedClient client = new MemcachedClient(Arrays.asList(address));

port is 11211

Yes, that is what I thought. What's going on underneath is that you're talking to Moxi on port 11211 and it is then rehashing the key and proxying the request to (likely) another server. This introduces an extra network hop and will significantly reduce your performance.

Take a look at this page for a description on how to configure spymemcached not to go through Moxi: http://www.couchbase.org/products/sdk/membase-java

There are no other nodes. There is only one.

However I tried to connect to the Membase using the following code:

client = new MemcachedClient(Arrays.asList(new URI("http://harold:8091/pools")), "default", null, null);

Now I am getting a lot of cache misses. It seems the more concurrent threads I use to retrieve the data, the more misses I get, while there should be none.

I checked a few keys that spymemcached reported missing (returned null) using telnet and all of them were there on Membase.

Just to be clear, I do not mix set and get operations. I set 100K pre-generated keys/values using one set of threads and then after they finish, I retrieve them back using another set of threads.

That "should" be working fine. What version of spymemcached are you using? We just released 2.7.1 so it might make sense to upgrade to that.

There are a variety of things within the Java client itself that can cause these sorts of issues, mostly involving overwhelming the heap with requests before they get serviced by the server.

I'm asking one of our spymemcached engineers to take a look and help here as well.

Perry

The test you're running isn't necessarily wrong, but it's also not very close to a typical deployment. The way spymemcached works, there is an IO thread which will manage connections to the (typically multiple) servers. The IO operations against the different connections is done with asynch calls, so that IO thread can get quite a bit of concurrency from the underlying system.

In micro-benchmarks, that IO thread can become a point of contention. If you're running more threads doing the same operations in a tight loop, it'll get busier.

Now, I wouldn't expect misses, but if you're using lots of threads or using asynchronous calls, I may expect some timeouts due to just the increase in work all contending for the same connection, managed by the same thread. There is a default 2.5 second timeout for operations. I'm guessing that you're counting timeouts as misses. If you can post your code somewhere, this might help me explain the misses.

I'm guessing that each thread might be in a tight loop. With most deployments, you're not running 256 threads doing work in a tight loop with only one Membase server (which actually means one connection). While there may be a lot of threads, say in an application server, they're doing something with the data being returned and participating in a much slower HTTP conversation with some other clients.

One thing you may try is giving each thread it's own MemcachedClient object (or have them share s a simple pool of client Objects), which will simulate more connections. This is closer to what a real deployment would look like though will give it more IO threads too.

Also, this is incorrect.

client = new MemcachedClient(Arrays.asList(new URI("http://harold:8091/pools")), "default", null, null);

Should be,

client = new MemcachedClient(Arrays.asList(new URI("http://harold:8091/pools")), "default", "default", "");

for the default bucket with now password. When using these constructors there should never be a null argument.

There are no timeouts, just misses. I start seeing them with just 4 concurrent threads. At the same time, my test works perfectly fine against memcached with over 60K GET operations per second and no misses.

The code you suggested:

client = new MemcachedClient(Arrays.asList(new URI("http://harold:8091/pools")), "default", "default", "");

Fails with NPE in spymemcached 2.7:

Caused by: java.lang.NullPointerException
at net.spy.memcached.vbucket.ConfigurationProviderHTTP.readToString(ConfigurationProviderHTTP.java:270)
at net.spy.memcached.vbucket.ConfigurationProviderHTTP.readPools(ConfigurationProviderHTTP.java:104)
at net.spy.memcached.vbucket.ConfigurationProviderHTTP.getBucketConfiguration(ConfigurationProviderHTTP.java:85)
at net.spy.memcached.MemcachedClient.(MemcachedClient.java:223)
at RemoteCacheTest$MemCache.(RemoteCacheTest.java:263)
at RemoteCacheTest.run(RemoteCacheTest.java:32)

It does not fail if I pass null as username.

I followed Perry's advice and tried version 2.7.1. It works as expected now. No misses. So this must have been a bug in version 2.7