Requests do not distributed evenly across server nodes

I have a Couchbase cluster consisting of 2 nodes. On the web interface I noticed that requests do not distributed evenly on these nodes; actually one node take the 90% of the traffic / load.

Is it normal? I though that couchbase distributes requests evenly at the first place. If it isn't what should i check to fix it?

Regards
John

That is unlikely to happen under normal conditions. But if you are constantly updating a single key and not touching other data, you could certainly see skews. Lets tart with How many nodes you have and what type of workload and a working set are you throwing at the servers? could you give me an idea about that?
thanks
-cihan

Actually the workload is from several web applications, not just one. The bucket is the default for all these applications.

The working set is about 4-5 gb

Apologies for the delay and glad Jeff came to rescue.
For future reference, you can see the inbalance on the output with # active vbuckets on one node vs the other. That points out the great imbalance of vbucket distribution which points to rebalance issues.
You can also detect this in the couchbase web ui through the vbucket distribution - reported under active vbucket per server graph.
Thanks
-cihan

1 Answer

« Back to question.

As i said we have a cluster of 2 nodes. A typical key-value store. No, we do not update the same key all the time, we know what we are doing. The fact is that 90% of the load goes to one server no matter how many keys we touch (and we touch thousands). Couchbase is 2.5.1 and we use the. Net client 1.3.3.

could you send the output of the command below for each node;
cbstats node#1:11210 -b default vbucket
thanks
cihan

1. The follwing 3 images showing the cluster from the UI. Note that the 1st server gets the 90% of traffic and items distribution.

http://tinypic.com/r/11qmib5/8
http://tinypic.com/r/1688tbm/8
http://tinypic.com/r/2lazhxf/8

The output of the cbstats command:

Node1: http://www.fashionguide.gr/node1.txt
Node2: http://www.fashionguide.gr/node2.txt

Sorry, the cbstats output node1.txt referes to the node with low usage, and node2.txt on this with high usage.

Hi zizzop -

Which version of the .NET client are you using? Can you enable logging and send me the output? If you want, you can attach it to a jira ticket: http://www.couchbase.com/issues/browse/NCBC

-Jeff

We use v1.3.3.
Log file can be dound here:
http://www.fashionguide.gr/error-log.txt

zissop -

What I see when I look at your vbucketmap in the logs is a cluster that has not rebalanced entirely. It looks like perhaps a rebalance was started and perhaps stopped for some reason?

You should be able to initiate anther rebalance and once it's completed, your data should evenly distributed again, taking the load off the single server. Other than that, the client looks healthy, although I do see some key not found in your logs - this could be expected, however.

If you continue to cluster issues, Cihan can probably help out more than me given his experience there.

-Jeff

There is no pending rebalance on the ui. Actually we had removed entirely the first server and added again to the cluster. As soon as it was added, it took 90% of the load again in about 1 hour.

What are the steps to initiate a rebalance? I will do it and come back to you.

Jeff

That was the problem. A rebalance operation has failed. I rebalanced the cluster nd all seems to be OK now.

In order to rebalnce them I had to try at least 20 times! Rebalancing failed and i had to re-initiate it to continue further. I think rebalancing needs a lot of work as it has a great failure rate without obvious reasons and no auto retry mechanisms.

Zizzop -

Good to hear things are working correctly now. I am not sure exactly why you were having problems with rebalancing; I have done literally thousands of successful swap/failover and rebalances.

Thanks for the feedback,

Jeff