Home | Forums | Membase | Membase Server 1.7.x

Adding 2 new nodes and rebalancing hangs...

5 replies [Last post]
Tue, 11/01/2011 - 12:32
enkrypt3d
Offline
Joined: 11/01/2011
Groups: None

Trying to add 2 more nodes to an existing cluster... the rebalancing process is hanging and all I can do is stop and restart it which doesn't seem to help...

Should I try to add one server at a time? Please help!

All servers are using gigabit and are on the same network segment. Thanks

Top
Tue, 11/01/2011 - 15:07
enkrypt3d
Offline
Joined: 11/01/2011
Groups: None

Only trying to sync 4GB of data across the 2 new servers..... why does this keep hanging? I'm on the latest version 1.7.2...... Can anyone help me out here?

This forum seems dead...... :(

Top
Tue, 11/01/2011 - 21:34
enkryp3d
Offline
Joined: 10/28/2011
Groups: None

So for those that are wondering, so far I've shut down my web farm to do the rebalance and its taking FOREVER!

It goes about 2-3% and hangs each time. So I have to stop it and restart it again. Why is this so difficult so sync ~4GB of data? It shouldn't be that time consuming.......

Top
Wed, 11/02/2011 - 08:38
enkrypt3d
Offline
Joined: 11/01/2011
Groups: None

FINALLY got it to rebalance after trying it about 25 times... What seemed to help was restarting the membase-server service on the 2 new nodes... no idea why that helped but its now rebalanced.

Top
Thu, 11/03/2011 - 14:14
Mantle
Offline
Joined: 09/30/2010
Groups: None

Usually the determining factor of rebalancing time is more so the # of keys, rather than the size of data. How many keys do you have in the system? When there are many keys, disk write tends to be the bottleneck especially if you are on AWS using EBS.

Also, if memory serves me correctly, I've heard from from Couchbase support team that adding one server at a time is a cleaner way to go.

Top
Thu, 11/03/2011 - 14:17
enkryp3d
Offline
Joined: 10/28/2011
Groups: None

Mantle wrote:
Usually the determining factor of rebalancing time is more so the # of keys, rather than the size of data. How many keys do you have in the system? When there are many keys, disk write tends to be the bottleneck especially if you are on AWS using EBS.

Also, if memory serves me correctly, I've heard from from Couchbase support team that adding one server at a time is a cleaner way to go.

I had about 1.5M keys / items... I tried that too and it didn't seem to help. Oh well its balanced now and all is well! now if only i could figure out my memcache timeout issues :)

Top