Home | Forums | Membase | Membase Server 1.7.x

Unable to remove downed nodes

2 replies [Last post]
Tue, 09/06/2011 - 13:03
zaros
Offline
Joined: 08/25/2011
Groups: None

Hi,

There are two nodes which used to belong to a cluster that I have terminated (in Amazon EC2), however I am not able to remove them from the cluster.

The 'Remove Server' button is disabled, and when I press 'Fail Over' I get an error.

This is the log:

Module Code Server Node Time
Server error during processing: ["web request failed",
{path,"/controller/failOver"},
{type,exit},
{what,
{{function_clause,
[{ns_rebalancer,
'-promote_replicas/2-lc$^0/1-0-',
[undefined,['ns_1@x.x.x.x']]},
{ns_rebalancer,failover,2},
{lists,foreach,2},
{ns_orchestrator,idle,3},
{gen_fsm,handle_msg,7},
{proc_lib,init_p_do_apply,3}]},
{gen_fsm,sync_send_event,
[{global,ns_orchestrator},
{failover,'ns_1@x.x.x.x'},
infinity]}}},
{trace,
[{gen_fsm,sync_send_event,3},
{ns_cluster_membership,failover,1},
{menelaus_web,handle_failover,1},
{menelaus_web,loop,3},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]}]

How can I remove them please?

Thanks
zaros

Top
Tue, 09/06/2011 - 18:54
perry
Offline
Joined: 10/11/2010
Groups:

What version is this Zaros? I'd like to make sure to test this and get any bug fixed as needed. I'm pretty sure I've done this same thing before. The one thing I can think of being a problem here is if another instance has come up with the same IP address as the one you are trying to failover...and so that cluster thinks that there is a node out there but can't communicate with it.

You should be able to forcibly remove them by using the "eject" functionality directly from the REST API: http://www.couchbase.org/wiki/display/membase/Membase+Management+REST+API

Perry

__________________

Forum support is great for free but sometimes you need a guaranteed response time and dedicated resources for your questions or issues.
Consider purchasing enterprise-level support from Couchbase: http://www.couchbase.com/products-and-services/overview
Call or email "sales -at- couchbase-dot- com" today!

Top
Tue, 09/06/2011 - 23:15
zaros
Offline
Joined: 08/25/2011
Groups: None

Hi Perry,

I'm using Membase Server 1.7.1

-- Just to add: if I click on the node ip and the accordion opens to show the details of that node, the 'remove server' button becomes enabled, but it still won't work

Thanks for the link to the REST API ... would try that if it happens again. Right now I've gotten around the problem by just making a new cluster. Of course, not in production yet :)

zaros

Top