timeout and connection failures under moderate load

Matt Ingenthron added a comment - 16/Dec/11 1:37 PM Output courtesy buzzy's Debian system. root@ip-10-48-43-67 :/opt/couchbase/bin# ./memcachetest -l -t 32 -h localhost:11211 Average with 32 threads Get operations:      #of ops. min max avg max90th max95th max99th          6655 108 us 18 ms 4457 us 9023 us 9384 us 11 ms Set operations:      #of ops. min max avg max90th max95th max99th          3345 116 us 18 ms 4284 us 8986 us 9311 us 11 ms Average with 32 threads Get operations:      #of ops. min max avg max90th max95th max99th          6758 114 us 26 ms 4995 us 9600 us 10 ms 13 ms Set operations:      #of ops. min max avg max90th max95th max99th          3242 116 us 26 ms 5053 us 9586 us 10 ms 14 ms Average with 32 threads Get operations:      #of ops. min max avg max90th max95th max99th          6696 109 us 20 ms 4878 us 9280 us 9811 us 12 ms Set operations:      #of ops. min max avg max90th max95th max99th          3304 117 us 15 ms 4585 us 9256 us 9691 us 12 ms Average with 32 threads Get operations:      #of ops. min max avg max90th max95th max99th          6720 133 us 20 ms 4946 us 9562 us 10 ms 13 ms Set operations:      #of ops. min max avg max90th max95th max99th          3280 115 us 16 ms 4872 us 9520 us 10 ms 13 ms Average with 32 threads Get operations:      #of ops. min max avg max90th max95th max99th          6752 125 us 14 ms 4581 us 9192 us 10 ms 12 ms Set operations:      #of ops. min max avg max90th max95th max99th          3248 156 us 14 ms 4387 us 9103 us 10 ms 12 ms Average with 32 threads Get operations:      #of ops. min max avg max90th max95th max99th          6694 116 us 13 ms 4501 us 9088 us 9431 us 11 ms Set operations:      #of ops. min max avg max90th max95th max99th          3306 127 us 13 ms 4357 us 9069 us 9365 us 11 ms Average with 32 threads Get operations:      #of ops. min max avg max90th max95th max99th          6749 110 us 15 ms 4365 us 8978 us 9269 us 11 ms Set operations:      #of ops. min max avg max90th max95th max99th          3251 114 us 15 ms 4255 us 8973 us 9236 us 10 ms Average with 32 threads Get operations:      #of ops. min max avg max90th max95th max99th          6766 109 us 23 ms 4538 us 9143 us 9691 us 11 ms Set operations:      #of ops. min max avg max90th max95th max99th          3234 117 us 23 ms 4557 us 9154 us 9773 us 12 ms <9749> isn't there anymore <3328> isn't there anymore <1644> isn't there anymore <638> isn't there anymore <9351> isn't there anymore <2917> isn't there anymore ASCII set failure: SERVER_ERROR proxy downstream timeout 10.48.43.67 ASCII set failure: SERVER_ERROR proxy out of downstreams <3944> isn't there anymore <6808> isn't there anymore ASCII set failure: SERVER_ERROR proxy out of downstreams <3570> isn't there anymore ASCII set failure: SERVER_ERROR proxy out of downstreams <8189> isn't there anymore <2863> isn't there anymore ASCII set failure: SERVER_ERROR proxy out of downstreams ASCII set failure: SERVER_ERROR proxy out of downstreams <3757> isn't there anymore <2961> isn't there anymore <1153> isn't there anymore <5042> isn't there anymore <8506> isn't there anymore ASCII set failure: SERVER_ERROR proxy out of downstreams <5451> isn't there anymore <9111> isn't there anymore <7916> isn't there anymore ASCII set failure: SERVER_ERROR proxy out of downstreams <1854> isn't there anymore <5432> isn't there anymore <9347> isn't there anymore <7558> isn't there anymore <5836> isn't there anymore ASCII set failure: SERVER_ERROR proxy out of downstreams <5681> isn't there anymore <2271> isn't there anymore

Daniele Testa added a comment - 18/Dec/11 2:27 AM I have found this: http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/concepts_micro_instances.html This might be the cause of how micro-instances at Amazon EC2 works. Seems like they limit the CPU after some time of high load. I will try to do the same thing on a "small"-instance and see if it gives the same result.

Keith Batten added a comment - 19/Dec/11 1:43 PM So far I have not been able to reproduce this with a large instance. Was this seen with the same memcachetest on both the micro and large?

Daniele Testa added a comment - 19/Dec/11 3:48 PM I have tried it with a "small" instance at Amazon EC2 now and it was even worse :( 1. Create a "small" instance in EC2 with ami-6bb5821f in the EU-region 2. Download the 32bit .deb package from couchbase and install with "dpkg -i" 3. Go to the HTTP admin and configure a 500MB membase bucket. 4. Run a test from another instance, like: ./memcachetest -l -t 32 -h 10.227.7.34:11211 When I do this, I see the numbers go up to 4-5k/sec for a couple of seconds, then errors: ( root@ip-10-48-125-234 )-(/opt/couchbase/bin)# ./memcachetest -l -t 32 -h 10.227.7.34:11211 ASCII set failure: SERVER_ERROR proxy downstream timeout Failed to set [8876]: during populate. ASCII set failure: SERVER_ERROR proxy downstream timeout Failed to set [1074]: during populate. ASCII set failure: SERVER_ERROR proxy downstream timeout Failed to set [4521]: during populate. ASCII set failure: SERVER_ERROR proxy downstream timeout Failed to set [6076]: during populate.

Daniele Testa added a comment - 19/Dec/11 3:55 PM Tried to run the test locally on the Couchbase server itself. It seems to work a little longer. I get about 3500 ops/sec for 2-3 min, then errors start. Maybe connections are not closed fast enough? ./memcachetest -l -t 32 -h localhost:11211 Set operations:      #of ops. min max avg max90th max95th max99th          3377 103 us 501 ms 7934 us 7156 us 36 ms 214 ms Average with 32 threads Get operations:      #of ops. min max avg max90th max95th max99th          6720 90 us1266874885 2745051 s 7031 us 43 ms 239 ms Set operations:      #of ops. min max avg max90th max95th max99th          3280 101 us1266874885 5624007 s 6580 us 37 ms 188 ms Average with 32 threads Get operations:      #of ops. min max avg max90th max95th max99th          6613 92 us1266874885 2789466 s 14 ms 40 ms 168 ms Set operations:      #of ops. min max avg max90th max95th max99th          3387 101 us1266874885 5446337 s 12 ms 42 ms 281 ms ASCII set failure: SERVER_ERROR proxy downstream timeout ASCII set failure: SERVER_ERROR proxy downstream timeout <1656> isn't there anymore Average with 32 threads

Daniele Testa added a comment - 09/Jan/12 12:49 PM I have done some more tests. This time with "memslap" ./memslap -s 127.0.0.1:11211 -S 5s -c 128 <84 SERVER_ERROR proxy downstream timeout <80 SERVER_ERROR proxy downstream timeout <92 SERVER_ERROR proxy downstream timeout <100 SERVER_ERROR proxy downstream timeout <104 SERVER_ERROR proxy downstream timeout ./memslap -s 127.0.0.1:11211 -S 5s -c 64 This gives less errors ./memslap -s 127.0.0.1:11211 -S 5s -c 16 This one works fine for hours It is pretty obvious that this has something to do with concurrent connections. The Proxy cannot take more than somewhere between 16-64 connection. Is someone looking at this? I would say this is kind of a big deal :)

Farshid Ghods added a comment - 09/Jan/12 12:53 PM small instance also has 1 ECU as well. couch base server needs at least 2 real physical cores. we couldn't reproduce this on a 64-bit large instance. have you seen the same issue there ?

Daniele Testa added a comment - 09/Jan/12 12:56 PM Yes, I see the same issue on "micro", "small" and "large" instances. But now I found this: http://www.couchbase.org/wiki/display/membase/Moxi+Performance+Tuning "To simplify, let's say you just have a single bucket (such as the "default" bucket), that downstream_max is 4 (the default, circa 1.6.0 moxi) and that NUM_MOXI_WORKER_THREADS is 4 (which is also the default -t command-line flag value, circa 1.6.0 moxi). So, in this example, there are 4 x 1 x 4 = 16 connections that a single moxi will make to memcached/membase." Is it true that the default connection-limit is only 16? That is waaaaaay too low in my opinion.

Farshid Ghods added a comment - 09/Jan/12 1:00 PM yes thats moxi's limitation. do you see the same behavior when running the load directly against memcached instead of maxi?

Daniele Testa added a comment - 09/Jan/12 1:04 PM I cannot run against memcached directly. I see that couchbase installed memcached on port 11210, but running memslap againt that port fails. ( root@ip-10-49-37-17 )-(/opt/couchbase/bin)# netstat -nlp|grep memcache tcp 0 0 0.0.0.0:11210 0.0.0.0:* LISTEN 1488/memcached tcp6 0 0 :::11210 :::* LISTEN 1488/memcached udp 0 0 0.0.0.0:11210 0.0.0.0:* 1488/memcached udp6 0 0 :::11210 :::* 1488/memcached ( root@ip-10-49-37-17 )-(/opt/couchbase/bin)# ./memslap -s 127.0.0.1:11210 -S 5s -c 16 servers : 127.0.0.1:11210 threads count: 1 concurrency: 16 run time: 600s windows size: 10k set proportion: set_prop=0.10 get proportion: get_prop=0.90 memslap: clients/memslap.c:793: ms_print_memslap_stats: Assertion `pos <= buf' failed. Aborted

Perry Krug added a comment - 09/Jan/12 1:05 PM Daniele, this is one of the many reasons why we recommend deploying a client-side Moxi as opposed to using the included server-side one.

Daniele Testa added a comment - 09/Jan/12 1:05 PM Why was Moxi added if it cripples memcached that much? Normal memcached has like 1024 connection limit by default. Moxi limits this to 16? Would make sense to follow memcached default limit to 1024 then.

Daniele Testa added a comment - 09/Jan/12 1:08 PM Perry, but I am running all this on one machine. What difference would a "client-side" moxi do then? Do you have a link how to just turned the moxi off all together and rely on the clients hashing instead?

Farshid Ghods added a comment - 09/Jan/12 1:11 PM Daniele, memslap is not a supported tool or a binary against recent membase or couchbase servers. It would b best to try out with a smart client or use the client side moxi where you can change its configurations and increase the number of connections and etc as Perry mentioned. you should be able to use memcachetest to run smaple load against memcached directly if you have one node.

Perry Krug added a comment - 09/Jan/12 1:13 PM The "client-side" Moxi gives you the ability to change these parameters...the server-side has hardcoded defaults which are obviously too low for certain scenarios. There is no way to turn off Moxi altogether. You don't need to turn off the server-side one if you're using a client-side Moxi and/or smart client, the server-side one will just be sitting idle. Unless you are using a smart client (currently available for Java and .NET, soon to be released for PHP and Ruby), you must use Moxi.

Daniele Testa added a comment - 09/Jan/12 1:27 PM Ok, understood. I was looking for a way to change the config for the "server-side" moxi without success. I guess in my case, I will install another moxi beside the existing one and put it on another port and run my tests again. Btw, the "memslap" binary is included in your package so I assumed it was ok to use it :) I have also used memcachetest with the same result, but at least now I know why. I will let you know if the client-side moxi solves my issues. Thank you.