2 Membases with 1 bucket (Replication enabled) slowness
Hey everyone,
We have a small Facebook game running, and we've noticed something very strange..
When we upgraded to 2 Membase servers running with 1 bucket (with replication enabled), our game suddenly started working incredibly slow - response times ran from as low as 100ms to as high as 8000ms!!
Once we've removed one of the servers from the cluster, the game "magically" fixed itself and all response times returned to a healthy 100-200ms response times.
What could be the issue here?
Obviously we want 2 servers to enable replication, but these slowness issues are out of the question.
The way we handled this from the code was in the configuration file, we added both hosts:
<add uri="http://SERVER_1_IP/pools/default" />
<add uri="http://SERVER_2_IP/pools/default" />
EDIT:
I've also ran a small performance test to further prove my point.
Running through 300,000 objects in Membase and saving them using 1 server took about 2 minutes.
Using 2 servers: took about 15 minutes!!!
We are using Enyim's .NET client.
And we're getting less than 1ms ping between the 2 membase servers.
Bump.
We need help with this - we are currently running without replication!
Are these by chance distributed between different LAN segments or some kind of WAN. For example, are the clients and servers in different EC2 availability zones?
It's really hard to tell what the issue might be here without more info. Can you post the basics of your test?
I don't believe so, they are 2 physical servers (not VPNs) hosted on the same host.
Should I ask them anything in particular?
You should check the latency (via a simple ping) between your client and all intended servers. I had seen, in one case, a scenario where the latency was about 1.2ms between the client and servers. The application code was doing many operations in serial and they would, as a result, see slowness from a high level. When they went to a single availability zone, latencies were closer to 300 microseconds, and things were much faster.
The real fix for them though was to start doing things in parallel. This allowed multiple operations to be on the wire to multiple servers simultaneously.
If you could post a bit about your test, I might be able to offer more advice. At the moment, I can't think of any reason it should be so dramatically slower when across multiple nodes. The only thought I have is maybe you're doing multiple operations and having to wait for each one to propagate over the wire and back before considering that higher level operation complete.
Hey,
I've already pinged all machines - all pings were below 1ms response time.
Regarding my test - I simply set a loop to go through about 350,000 items in Membase and load them, change something, and save back to Membase.
Regardless to the test; our game simply ran incredibly slow to a complete stand still when running 2 Membase servers with replication enabled.
BUMP!!!
This is a very serious issue and a deal breaker for us to use Membase.
We have 2 servers and only 1 of them is currently running Membase, the 2nd was detached due to the issue described!
This is not an acceptable state and personally I'm amazed at how bad it works when a simple and mandatory thing such as replication is involved.
PLEASE HELP - We currently have NO REPLICATION on our production server!
I agree, it's definitely not what we'd expect. At the moment, it's hard to say where the slowness is. It sounds like a possible misconfiguration or other client issue.
Can you possibly set up Enyim against a the bucket on a specific TCP port number, cluster your servers and try your workload against that? If that performs well, then it's a good workaround and we can then do some additional work to try to isolate where the issue is. In this case, you'd configure Enyim against the port (running moxi) as if it were using a memcached server on the port.
Also, on what OS/architecture are you running the Membase servers?
By the way, if you need direct support please do not hesitate to engage with Couchbase sales/support. They can do more to directly assist with diagnosis and resolution.
Hello,
I am also facing similar kind of issue.
I have 2 membase servers running single bucket each.
in my client config code i have updated both servers. req. is when one server is down, then it should automatically connect to another server.
But it is not working. do i need to do any other settings.
on both servers setup i have created new cluster.
Is there document how to setup multiple membase servers?
In order to provide advice on other settings, I'd need to know more about what client library configuration you're using.
If you're on Membase 1.7, you'll probably want to upgrade to Couchbase 1.8. There is great documentation on how to set that up at http://www.couchbase.com/docs
Thank you for quick response.
I am using couchbase 1.8, downloaded the software for windows 64 bit and Linux 64 bit. Successfully installed on both server. (ex – S1,S2)
If I am using one server at a time then everything works as expected.
Now requirement is, I need to configure in my client config to point both server. And when I store data in one server, I should be able to retrieve from another server, to do this what extra step I need to follow?
My client config looks like this –
On both server single node is created.
Can you please help me how I can achieve this.
Thank you in advance
First off, while it may work, mixing windows and linux servers in the same cluster is not tested or supported.
You'll probably want to read the manual for 1.8, in particular this section:
http://www.couchbase.com/docs/couchbase-manual-1.8/couchbase-deployment....
The main idea is that if you're going through the proxy port (a.k.a. moxi) you can configure your client against one or more servers and it will automatically loadbalance the traffic against all servers. I don't see a picture there of your client configuration though. Can you at least tell me what language, version of client library is?
Ok, I will go through the link you have sent.
I am using .NET 3.5 client and C# language. I don't know what is moxi.
So, if i setup couchbase server in 2 windows environment,
then will it load balance without doing any mapping in the server config?
Hello,
I modified the code little bit.
I installed couchbase for windows 64bit on 2 windows machines.
on both the servers i created new cluster and one default node is created. (followed steps given in the doc)
I am using .NET 3.5 as a client. my config looks like this
<couchbase> <servers bucket="default" bucketpassword=""> <add uri="http://10.1.1.91:8091/pools/default"></add> <add uri="http://192.168.0.179:8091/pools/default"></add> </servers> </couchbase>
When I telnet and check in the couchbase server, the data is stored in only 1 server.
I am using below command for telnet
cmd>telnet 10.1.191 11211 - Here I could see data
cmd>telnet 192.168.0.179 11211 - Here i am not able to see the data.
it is not storing in the 2nd server. Do I need to make any other config change?
What client are you using to connect to membase?
What latency do you see from the first server to the second server?