nodejs sdk performance issue
hi,
I was also evaluating the SDK's and found the nodejs sdk seems to be far less performant at this moment. Maybe it is because of the asynch nature of the sdk?
I tried to run a code that uses a simple for loop to set x numbers of objects in a bucket.
1. With Java Synchronous set call, I could get over 8k sets per second
2. With Java Asynchronous set call, I could get about 4k sets per second
3. With PHP Synchronous set call, I could get about 4k sets per second
4. However, with nodejs set call, I could get the set to work with the iteration number of the loop set to about 50. Any number behind that just spikes the CPU usage and hangs
I am on Mac OS X Mt Lion. Also, this is just the set operation along without mixing in with any other operations. The couchbase client is only created once as well outside the for loop. I am using the latest sdk's and libcouchbase as of 1/27/2013. In reality we will probably never need to 'loop' set operations like this. However, I just wanted to do a comparison by languages especially asynch Java vs nodejs. The result is just way too different than I would expect and I feel shocked. That being said, we might scratch the original idea to use nodejs and use Java or Php instead.
Just wondering if anyone else is experiencing this issue?
Thanks for the update Tug. I look forward to the future updates.
Hey Bob, tgrall,
We're working on a nodeJS application and ran into the same problem. We don't however have the option of switching to PHP or Java, unfortunately.
Node just hangs after trying to set too many data, with these last messages:
read(24, "9c6\r\n,\r\n{\"id\":\"invoices::#9968\","..., 65536) = 2520
getrusage(RUSAGE_SELF, {ru_utime={2, 8694}, ru_stime={0, 296954}, ...}) = 0
brk(0x27d2000) = 0x27d2000
futex(0x7f171c0008c8, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0x7f171c0008c8, FUTEX_WAKE_PRIVATE, 1) = 1
brk(0x27f7000) = 0x27f7000
brk(0x2819000) = 0x2819000
brk(0x283b000) = 0x283b000
brk(0x285d000) = 0x285d000
brk(0x28c3000) = 0x28c3000
brk(0x28e5000) = 0x28e5000
brk(0x2907000) = 0x2907000
brk(0x2929000) = 0x2929000
brk(0x294b000) = 0x294b000
brk(0x296d000) = 0x296d000
brk(0x298f000) = 0x298f000
brk(0x2a8e000) = 0x2a8e000
brk(0x2ab0000) = 0x2ab0000
brk(0x2ad2000) = 0x2ad2000
brk(0x2af4000) = 0x2af4000
brk(0x2911000) = 0x2911000
close(24) = 0
epoll_wait(3, {{EPOLLIN, {u32=5, u64=4294967301}}}, 64, 794) = 1
read(5, "x", 1024) = 1
write(23, "\200\5\0\17\24\0\3e\0\0\0#\\\2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1"..., 59) = 59
We'd be willing to deliver debug data, please keep me updated.
I have the same problem, on Mac OS X and also on Ubuntu. I'm gonna try on Amazon CentOS, though I suspect it will be the same...
I hand't found this post before I posted mine
http://www.couchbase.com/forums/thread/how-many-simultaneous-sets-can-li...
Hey Mladen,
We 'discovered' the RESTAPI and have modified https://github.com/cloudhead/cradle to work as our wrapper. Works perfectly.
You can steal all the web requests you need from the couchbase:8091 administrative web.
Thanks, I wanted to avoid using the REST API directly.
What did you modify in cradle, do you by any chance have a fork? Cradle is for CouchDB, although Couchbase uses CouchDB internally.
Removed some Cradle checks, changed how it requested the URL's. Pretty straightforward. No fork, unfortunately
I understand your reluctance of communicating directly with the RESTAPI, but believe me - you do not want nodejs crashing in production because libcouchbase/couchnode crashed for some weird reason. We were able to reproduce again and again the crashing of the library (under a little stress), did not want to move back to couchdb (bigcouch, cloudant, iriscouch - all have a reason to make sure to avoid them), so the RESTAPI was kind of the last option we had.
But the RESTAPI/cradle does not support metadata and CAS, does it?
I found a workaround which will do the job - create a pool of connections and iterate. Apparently the problem is per connection.
Not the perfect solution, but will take care of concurrent modifications (from another process) as I can pass meta & CAS.
Check out this gist
Cheers
Mladen
Mladen,
Nice workaround - if you don't overload the connections you will indeed not crash the library.
You can find our edited Cradle fork here:
https://github.com/renedx/cradle
Without CAS and meta, unfortunately. But we're not using it anyway.
Mladen,
Decided to take a look to see if we could get your workaround to work on our environment.
But.. how do you close the connections you open up?
Hello,
The big differences between the NodeJS SDK and other SDKs you are listed, is the fact that NodeJS is still in early stage development (0.0.11 release).
We have notice this and will be working on this soon.
Regards
Tug
@tgrall