Couchbase cluster load balancing
Hello. I'm running Couchbase 2.0 beta on 2 linux instances. For accessing couhbase I use .net api. In it's config I set both server urls:
<cb-bucket1>
<httpClientFactory type="Couchbase.HammockHttpClientFactory, Couchbase" />
<servers bucket="bucket1" bucketPassword="qwerty" username="Admin" password="qwerty">
<add uri="http://10.0.0.1:8091/pools/default" />
<add uri="http://10.0.0.2:8091/pools/default" />
</servers>
</cb-bucket1>I've got some buckets that configured in same way.
So on the first server (10.0.0.1) CPU usage is much greater. Load average 2-2.5 times larger. Also 10.0.0.1 was firs node in cluster (so I think it is master). When I change servers order in config 10.0.0.2 cpu usage increased and now cpu utilization on both instances almost equal.
I've got several questions:
- What is better: master node first, or "slaves" first to equalize cpu load?
- Is this behavior connected with couchbase or it's connected .net library?
- Is it possible to use something like round-robin for those requests, to equalize cpu load of all cluster nodes?
I use the lastest lib, compiled from source.
Here is screenshot:
After red line the second node (called "slave", 10.0.0.2) moved to the frist line in servers config.
CPU utilization will always be somewhat uneven as there are some kinds of processing that run in one and only one place. Also, the .NET client does have a 'heartbeat' check that will use a little bit, but generally shouldn't get in the way.
Is this CPU usage getting in the way of any processing?
The cpu usage is almost equal when it processing something and idle, because the load isn't so big.
Do you know what frequency this 'heartbeat' check has? I'm asking because something (I don't exactly know what it is) in my web app began eating 100% cpu every 25 seconds. Couchbase lib was updated just that day when it began. Those cpu usage spikes appear even when noone using my app and sometimes they grow (i.e. in the begginning it eats 50% on each core of 8-core cpu and lasts in 1 sec, later it eat 100% on each core for 2 or more seconds). And it disappears only when I recycle IIS app pool. Also when I look through my network activity (on app server), in normal mode (whitout spikes) I saw that app server sends and then receives something over tcp, and when cpu usage jumps to 100% there are multiple "TCP Receive" from couchbase servers in a row without any tcp sending.
I'm seeing the same thing on Couchbase 1.8.1 on a 10 node cluster. Node1 has about double the CPU and Load average as the rest of the nodes.
Not sure how to relieve this pressure as its a bit concerning to me. Should i be changing my application configs to stagger the order of the couchbase nodes so they "even" out?
The best policy is to change the config to use simply /pools, but if you're using the latest .NET client library it should do that automatically. Which version of the client library are you using?