Home | Forums | Couchbase | Couchbase Server 1.8.x

remove & rebalance fails after a server reboots

5 replies [Last post]
Thu, 01/31/2013 - 20:38
fahd
Offline
Joined: 01/29/2013
Groups: None

Server 10.153.253.183 rebooted, tried to remove said server & rebalance:

Rebalance exited with reason {wait_for_memcached_failed,"cookie-segments",
['ns_1@10.153.253.183']}
ns_orchestrator002	ns_1@10.153.253.180	22:32:43 - Thu Jan 31, 2013
Started rebalancing bucket cookie-segments	ns_rebalancer000	ns_1@10.153.253.180	22:32:32 - Thu Jan 31, 2013
Starting rebalance, KeepNodes = ['ns_1@10.153.253.180','ns_1@10.153.253.181',
'ns_1@10.153.253.182','ns_1@10.153.253.184',
'ns_1@10.153.253.185'], EjectNodes = ['ns_1@10.153.253.183']
ns_orchestrator004	ns_1@10.153.253.180	22:32:31 - Thu Jan 31, 2013
Node 'ns_1@10.153.253.185' saw that node 'ns_1@10.153.253.183' came up.	ns_node_disco004	ns_1@10.153.253.185	21:54:18 - Thu Jan 31, 2013
Node 'ns_1@10.153.253.184' saw that node 'ns_1@10.153.253.183' came up.	ns_node_disco004	ns_1@10.153.253.184	21:54:18 - Thu Jan 31, 2013
Node 'ns_1@10.153.253.182' saw that node 'ns_1@10.153.253.183' came up.	ns_node_disco004	ns_1@10.153.253.182	21:54:18 - Thu Jan 31, 2013
Node 'ns_1@10.153.253.181' saw that node 'ns_1@10.153.253.183' came up.	ns_node_disco004	ns_1@10.153.253.181	21:54:18 - Thu Jan 31, 2013
Node 'ns_1@10.153.253.180' saw that node 'ns_1@10.153.253.183' came up.	ns_node_disco004	ns_1@10.153.253.180	21:54:18 - Thu Jan 31, 2013
Node 'ns_1@10.153.253.182' saw that node 'ns_1@10.153.253.183' went down.	ns_node_disco005	ns_1@10.153.253.182	21:20:21 - Thu Jan 31, 2013
Node 'ns_1@10.153.253.180' saw that node 'ns_1@10.153.253.183' went down.	ns_node_disco005	ns_1@10.153.253.180	21:20:21 - Thu Jan 31, 2013
Node 'ns_1@10.153.253.181' saw that node 'ns_1@10.153.253.183' went down.	ns_node_disco005	ns_1@10.153.253.181	21:20:19 - Thu Jan 31, 2013
Node 'ns_1@10.153.253.184' saw that node 'ns_1@10.153.253.183' went down.	ns_node_disco005	ns_1@10.153.253.184	21:20:14 - Thu Jan 31, 2013

Top
Mon, 02/04/2013 - 05:14
tgrall
Offline
Joined: 09/05/2012
Groups: None

Hello,

Could you please give us more information about what happened, before rebooting?

Tug

__________________

Tug
@tgrall

Top
Mon, 02/04/2013 - 16:24
fahd
Offline
Joined: 01/29/2013
Groups: None

There was a hardware issue on the node the server is on. Nothing related to load, connections, or OS.

Top
Thu, 02/07/2013 - 09:10
tgrall
Offline
Joined: 09/05/2012
Groups: None

Hi

So now you are ok? everything is back to normal (from a Couchbase point of view)?

Regards

__________________

Tug
@tgrall

Top
Thu, 02/07/2013 - 14:22
fahd
Offline
Joined: 01/29/2013
Groups: None

lol! not really, as far as I'm concerned couchbase doesn't have a failover/rebalance feature. In two years, it has never worked.

Top
Fri, 02/08/2013 - 07:44
Neo-matrix
Offline
Joined: 10/15/2012
Groups: None

Hi,

Have you tried setting up a different set of nodes and tried the rebalance/Auto Failover feature.

Thanks,
Vijay

Top