Performance bottleneck at BaseSerializingTranscoder.deserialize
Hi,
We have some serious performance issues after migrating to Membase.
While Profiling the application, it is noticed BaseSerializingTranscoder.deserialize as the bottleneck.
Is there a workaround to deal with this bottleneck?
Client: spymemcached-2.7.1.jar
I do not have client side moxi.
9.7% - 207 ms - 153 hot spot inv. net.spy.memcached.transcoders.BaseSerializingTranscoder.deserialize
9.7% - 207 ms - 153 hot spot inv. net.spy.memcached.transcoders.SerializingTranscoder.decode
9.7% - 207 ms - 153 hot spot inv. net.spy.memcached.transcoders.TranscodeService$1.call
9.7% - 207 ms - 153 hot spot inv. java.util.concurrent.FutureTask.run
9.7% - 207 ms - 153 hot spot inv. net.spy.memcached.transcoders.TranscodeService$Task.run
8.9% - 190 ms - 142 hot spot inv. java.util.concurrent.ThreadPoolExecutor$Worker.run
0.8% - 17,351 µs - 11 hot spot inv. net.spy.memcached.transcoders.TranscodeService$Task.get
0.8% - 17,351 µs - 11 hot spot inv. java.util.concurrent.Future.get
0.8% - 17,351 µs - 11 hot spot inv. net.spy.memcached.internal.GetFuture.get
0.8% - 17,351 µs - 11 hot spot inv. java.util.concurrent.Future.get
0.8% - 17,351 µs - 11 hot spot inv. com.away.common.cache.MemcacheManager.get
0.8% - 17,351 µs - 11 hot spot inv. com.away.common.cache.MemcacheManager.get
0.8% - 17,351 µs - 11 hot spot inv. com.away.common.cache.CacheManager.get
0.7% - 14,117 µs - 1 hot spot inv. com.away.activity.business.AttractionHelper.searchAttractions
0.1% - 1,528 µs - 1 hot spot inv. com.away.qa.dao.CommonQADAO.getQADTOObjectUsingCacheOrDB
0.0% - 417 µs - 2 hot spot inv. com.away.consumer.product.dao.ExpertDestinationOverviewDAO.getDdocDetailsForOverview
0.0% - 413 µs - 1 hot spot inv. com.away.activity.business.AllArticlesPull.process
0.0% - 381 µs - 2 hot spot inv. com.away.activity.business.DestinationImageList.getAllImagesForDestination
0.0% - 199 µs - 1 hot spot inv. com.away.consumer.product.dao.CMSContentDAO.getArticleAllDetails
0.0% - 156 µs - 1 hot spot inv. com.away.consumer.product.dao.ExpertDestinationOverviewDAO.getDdocDetailsForHighlights
0.0% - 140 µs - 2 hot spot inv. com.away.consumer.common.dao.CobrandsDAO.getCobrandForDomain
Thanks
Abhilash
Thanks Matt,
Yes, ours is a content website, so in some cases the objects could be large and complicated (ex. List of Objects, where objects hold some other lists and maps).
I have tried with many clients (4, 16 and then 40), but as you said, we couldn't see any performance improvements.
Is it a good idea to increase the downstream_conn_max while staring Membase? (which is 4 now).
If so, can you please mention how to change that?
I couldn't see a config file where I can edit this, instead I found this in a binary file (moxi), but I can't edit.
Startup command: /etc/init.d/membase-server start
Thanks again!
Abhilash
http://tech.vys.in/2010/10/thrift-for-serializingdeserializing.html
This talks about this problem. But the solution explained would need major changes to the code base.
I wish, couchbase will come up with some solutions to improve de-serialization.
Is there any other workaround?
Thanks
Abhilash
I know Vinay well, but his blog doesn't exactly capture the issue here as PHP's default serialization and Java's built in serialization are rather different.
That said, a link in one of the posts you had appeared pretty interesting:
https://github.com/eishay/jvm-serializers/wiki/
It looks like there are a number of options. You can implement your own serializer and configure the client to use it.
I don't think I've seen this situation before. Are these objects particularly large or complicated? 207ms is an eternity for this kind of thing.
If you actually need to serialize/deserialize the objects and that's taking a lot of CPU time and stacking up, one thought would be to create multiple MemcachedClient objects. This wouldn't actually make anything go faster, but you'd be able to do more work in parallel if the underlying system has enough CPU to do more work.