Couchbase cluster stuck after node failure

Couchbase Server
1

Hello,
today one of the servers in a small cluster we have incurred a hardware failure. The cluster had 4 nodes, running only ‘memcache’ type buckets. Version 4.1.1-5914 Community Edition (build-5914).

We fixed the server (required a full re-install), but now we are stuck in a situation where the cluster lists the old server as “pending rebalance”, and won’t come out of that state.

I tried (from the CLI) server-remove -force, it says “unable to rebalance cluster (400) Bad Request {“deltaRecoveryNotPossible”:1}” . If I try a rebalance command, it replies with the same error. I tried setting the recovery type to full, but it did not help.

Any idea how I can evict for good that node and rebalance the remaining three? As I mentioned all buckets are all memcached, there is very little data to rebalance, per se.

Many thanks.

2

You should be able to Failover the dead node on the UI (use hard failover), then add the new one in and Rebalance.

3

Yes, thanks for the reply. It worked, eventually. I had to ‘cancel’ the rebalance via CLI, then do a rebalance again to remove the node. Not terribly intuitive.
Thanks again.