Scalability with Membase
Hi
I have played around with Membase, added/removed Membase nodes in order to see a difference in the ability to respond to requests and can not work out why there is no difference.
My setup is 2 Nginx servers loadbalanced with Memcached clients for PHP and Moxi server client side. I have 3 Membase nodes.
If I do a simple serving (just echo "ok", 2 bytes document) I'm able to get 2551.39 requests per second.
If I do a (125 bytes document):
$m = new Memcached();
$m->addServer('localhost', 11211) or die ("Could not connect");
$rv = mt_rand();
$m->set($rv, array('user_id' => 1, 'url' => 'http://something', 'name' => $rv));
var_dump($m->get($rv));
... I get 330-460 requests/second.
If I remove 1 Membase node I would assume that the cluster would not be able to handle as many request as before but to my surprise the 2 remaining nodes serve the same amount, how can that be?
Hi Perry
As stated earlier in another thread if I ping from one of the frontend servers to backend servers I get this output:
15 packets transmitted, 15 received, 0% packet loss, time 14024ms
rtt min/avg/max/mdev = 9.931/10.208/10.536/0.167 ms
... so an average of 10.208 ms could this be the bottleneck? But does this really have an impact as the Moxi client side description states "the client-side Moxi server can reuse long-running connections (saving on TCP connection setup handshake time)" ?
I have not done tuning of the Moxi client only passed the REST urls at startup. The commandline was as follows:
nohup /opt/moxi/1.6.5/bin/moxi -u root http://VPS1.host.com:8091/pools/default/bucketsStreaming/default,http://... &
... and that was done on each frontend server.
Thanks...
Yes, those 10ms will really have an impact on the performance that you can get between your frontend and backend servers. You're correct that Moxi will hold persistent connections, but you still have to wait at least 10ms for each request to go from your client to the server. Most environments will have only a few hundred microseconds of latency so you're essentially being slowed down 10x.
There's no tuning on Moxi that will help this. is it possible to have your frontend and backend servers on the same LAN?
Perry
Hi Perry
Ok, not glad to hear that the 10ms really is the problem here.
Since, the frontend and backend are residing in different datacenters I do not know how to accomplish that. The reason for the split up is that each datacenter provides different benefits and drawbacks.
Thanks...
That sounds fine. I don't foresee any major problems in what you have setup now, other than some less than desirable performance but the system should be able to continue working.
Take care, let me know if you have any other questions.
Perry
Hi Perry
Hm, an obvious reason for getting that low a number for requests/second is my own settings for the ab stress test. By raising the number of concurrent request and the number of request I can get the number to a maximum of Requests per second: 542.98 . Raising the number of request and concurrent requests even further does not have an impact on requests/second, so a bit better as you see. Adding another frontend server gave me about 800 request/second.
When you stated earlier that you would expect a bigger throughput than 340-460 requests/second what number did you have in mind?
It really varies quite a bit. I've personally seen a 5-node cluster running at over 100k ops/sec and I know that it can go much higher. It's really a matter of finding and eliminating bottlenecks in the whole environment. With 10ms latency though, I really wouldn't expect very high numbers for you.
It's also not necessarily a matter of raw ops/sec, whatever level satisfies your application's needs is "enough".
Make sense?
Perry
I would really expect you to be able to get much better throughput than 330-460 requests/sec, I wonder if there's something else going on in the environment. Can you send over the command-line you're using to confiugre the client-side Moxi?
As far as the 2 nodes being able to handle the same load, this level of traffic is not going to tax the Membase nodes and so it's likely that your bottleneck is elsewhere.
Perry
Forum support is great for free but sometimes you need a guaranteed response time and dedicated resources for your questions or issues.
Consider purchasing enterprise-level support from Couchbase: http://www.couchbase.com/products-and-services/overview
Call or email "sales -at- couchbase-dot- com" today!