Batch atomic operation support
Not sure if that's applicable to Membase since it would require changes to the Memcached code base, but it would be very useful for us to have a 'increments' method incrementing multiple keys at once.
Here's our use case. We've got a log processing application using Memcached as a distributed counter. One log entry generates approximately 500 increments. Because these need to be executed individually, network latency is creating a bottleneck. All other operations are run in parallel. I expect that a batch-increment feature would increase log processing speed by a factor of 2 or 3x, from 500ms to about 150ms.
I could also see the use of batch modes for all standard memcached operations - store, remove, append, prepend, increment, decrement.
Slegay, I want to dig into this a bit deeper. When you are doing these 500 increments, do you need to receive the value back for each one, or is this process only concerned with the incrementation and the value will be read separately at a later time?
I do need the incremented value immediately. I'm using the increment during log processing, and since Memcached is not persistent, I immediately send the resulting value to an eventually consistent data store.
One option would be to use membase, but a simple KV store doesn't fit our log processing / reporting model very well.
Gotcha, thanks for the info. Would Membase as a persistent "cache" remove your need to read the value back immediately? What I'm trying to get at is the ability to fire off a batch of quiet increments so that your code can move on...there would be much more work involved in firing off those requests and then waiting for all the responses.
The problem is we're creating sorted indices based on those increment values. That's where the key/value store model doesn't work out too well - deserializing and re-serializing the indices into binary data requires locking those indices during updates and therefore would create a different bottleneck and more complexity.
The current performance is not optimal, but it's acceptable for now. The log entries are queuedand processed asynchronously anyway, and the system is horizontally scalable. So as we get more entries in the queue, we can scale out our processing nodes. Over time, we'd like to optimize the actual processing of each entry to save money on CPU / hour, but again - for now, we're OK.
Thx
FYI, since we resolved the issue mentioned [URL=http://forums.northscale.com/showthread.php?446-Performance-issues-with-Enyim-client-2.2-and-above]here[/URL], our increment throughput is now 4x to 5x faster. We're seeing about 5,000 increments / sec on a 2x single-CPU nodes cluster. That matches up well with the write performance of the other distributed DB that's actually doing the persistent writes (CPU has gone from 10% avg to 40% avg on that cluster), and each log entry is now processed in about 300 ms vs 1.5 sec.
Turns out what we perceived as a network latency problem had to do with the "Proxy" port being used instead of "Direct".
<*oops*>
Thanks slegay for following up on this, we'll be moving back to the direct port as soon as possible.
Glad to hear your increment performance has gone up so much, and again thank you so much for your diligence and expertise.
Perry
Bump. I was wondering if you guys had an ETA on batch increments. I see the feature is marked 'in progress' on the enyim client project page, but I'm not sure what that means exactly.
We just deployed our new stats engine, but EC2 performance / network latency is killing us. I still think the above feature would help us quite a bit.
Thanks, and congrats on Membase! Just upgraded yesterday in our dev environment, looks like a pretty cool product so far.
I'll check on progress, but according to your previous post you had resolved this. Is that no longer the case?
Perry
Yes, the first performance issues we first experienced got fixed. In our staging environment, performance became acceptable and we could do about 4,000 increments per sec. on a 2-server cluster. Now that we've moved to production on EC2, however, we're down to only 450 increments / sec. We're trying to work around the issue by increasing connection count and using larger instances, but things are becoming much more costly than anticipated and not sustainable over the long term.
I suspect EC2 latency has something to do with it. Having the ability to process 1,000 incremented keys in one round-trip rather than 1,000 would make a big difference.
Thanks for the confirmation.
Perry
Batch operations would also help in our use-case:
- Client application calls webservice
- Webservice performs operations
- Increment a couple of counters
- Decrement another counter
- SET a few more values (e.g. a date/time stamp, checksum or user id)
One round trip for these operations would be preferable.
We're not so worried about the batch SET/INCR/DECR operation returning value(s). I suppose, if needed, we can fire off the batch and then immediately GET any required values for a total of two round trips, which is still less than what we're currently doing.
Thanks for the feedback slegay, we're definitely planning on adding these kinds of features to our software...this is a great specific use case for us to build off of.
Perry
Forum support is great for free but sometimes you need a guaranteed response time and dedicated resources for your questions or issues.
Consider purchasing enterprise-level support from Membase: http://www.membase.com/products-and-services/overview
Call or email "sales -at- membase -dot- com" today!