Performance issues with Enyim client 2.2 and above
We upgraded our dev branch from Enyim beta to 2.4. It looks like doing a multi-get oprtation is now about 100x slower. We ran a test through 5 versions of the Enyim client.
This requests involves retrieving 13 objects via single gets and 73 objects via one multi-get. Here's what we're seeing:
Enyim.Memcached v2.0b-0-g9f62c16, 05/30/2010
[url]http://my.devmachine.com/administrator/questions/folders/All?%5Fformat=json[/url]
Keys retrieved (seen on Membase dashboard): 37
Bytes Received: 17,463
Overall Elapsed: 00:00:00.0370000
Enyim.Memcached.2.0.beta
[url]http://my.devmachine.com/administrator/questions/folders/All?%5Fformat=json[/url]
Keys retrieved (seen on Membase dashboard): 37
Bytes Received: 17,463
Overall Elapsed: 00:00:00.0450000
Enyim.Memcached.2.0.beta2
[url]http://my.devmachine.com/administrator/questions/folders/All?%5Fformat=json[/url]
Keys retrieved (seen on Membase dashboard): 64
Bytes Received: 17,463
Overall Elapsed: 00:00:05.7390000
Enyim.Memcached.2.2
[url]http://my.devmachine.com/administrator/questions/folders/All?%5Fformat=json[/url]
Keys retrieved (seen on Membase dashboard): 64
Bytes Received: 17,463
Overall Elapsed: 00:00:05.4690000
Enyim.Memcached.2.3
[url]http://my.devmachine.com/administrator/questions/folders/All?%5Fformat=json[/url]
Keys retrieved (seen on Membase dashboard): 64
Bytes Received: 17,463
Overall Elapsed: 00:00:05.2520000
Enyim.Memcached.2.4
[url]http://my.devmachine.com/administrator/questions/folders/All?%5Fformat=json[/url]
Keys retrieved (seen on Membase dashboard): 64
Bytes Received: 17,463
Overall Elapsed: 00:00:05.0640000
As you can see, what used to be a 40ms roundtrip now takes 5 to 6 secs. Also, the Membase dashboard stats don't add up.
We're using a 2-node cluster, with no custom parameters on the client - Transcode, Nodelocator and KeyTransformer all have default values. Configuration is identical for each test. These metrics can be reproduced on every test, there's no erratic behavior. We're using the same buckets, same code base, same data, same web server for every single test. The only thing we're swapping are the Memcached and Membase dlls.
When debugging, we've identified the culprit as being the Read method of the MultiGetOperation class. Just to make sure it's not a config issue, we removed the line of code calling transcoder.Deserialize(), but that didn't improve performance.
As you can imagine, we're stuck here. This is a major performance issue. Any idea what could be causing this? Anybody else experiencing similar problems?
which clients are you using? can you paste your configuration?
do you have a warmup period? the northscale client has an initial startup time which could alter your results.
Can you share your test code?
Perry
I found the problem. The June 16 commit changed the ProxyPortType default config value from
- [ConfigurationProperty("port", IsRequired = false, DefaultValue=BucketPortType.Direct)]
To
+ [ConfigurationProperty("port", IsRequired = false, DefaultValue=BucketPortType.Proxy)].
Sure enough, if I use version 2.4 but add port="Direct" in the servers section, performance gets back to where it should be.
So now I have another question: is the Proxy port supposed to be 100x slower? Or is it a sign of a config issue on my end?
What's the benefit of using the Proxy port?
In that release of our server, we do know that the proxy port has a significant performance degradation. The purpose of the proxy port is to allow any server to access any of the data within the cluster. This negates the need to have a completely consistent and up-to-date client configuration. It's not something we generally prescribe for heavy use, but it does have it's advantages and use cases.
In the upcoming release (1.6) we have totally rewritten that area and it's much more performant.
I don't think there's any config issue on your end and I'll look into the reason behind that change.
Perry
Great, thanks for the info. The performance hit is definitely substantial enough to warrant a strong warning or at the very least not making the Proxy port the default value. I'd be fine with a 20% performance hit, but this made the application literally unusable.
Thanks again.
NOTE: the thread title is confusing - it looks like the performance problems started at beta 2.