Couchbase Rebalance Fails After Node Rejoin (inactiveAdded state)

Couchbase Server
1

I’m facing an issue in my Couchbase 3-node cluster (Community Edition 6.0.0). One of the nodes went down, and after restarting the service and trying to add it back to the cluster, the rebalance fails with the following error:

Rebalance exited with reason (badmatch, failed)
Error: Rebalance pid undefined
I’ve tried re-adding and rebalancing from the Web UI and CLI, but the rebalance consistently fails.

Steps already tried:

Restarted Couchbase service on problem nodes

Verified node communication

Observed no traffic handled by inactiveAdded nodes

Clearing cluster metadata on affected nodes, but it did not resolve the issue

Can someone please help me on this?

2

Do you have a support ticket open?

3

Hey there @Sneha can you provide the entire set of logs collected using the standard log collection tool (cbcollect_info), which you can collect through the web UI under the logs > collect information pages..? That will give us much more information about the error you encountered. Given just the snippet you have posted, it’s impossible to say what the issue is unfortunately.

1 Like
4

Hi @bryan.mccoid :waving_hand:,

Thanks for replying to my post. I’ve shared the error details below — could you please have a look? If you can suggest a solution, it would be a great help.

[ns_server:error,2025-04-20T04:35:00.592-04:00,ns_1@10.101.105.182:service_status_keeper-index<0.437.0>:service_status_keeper:handle_cast:119]Service service_index returned incorrect status

[ns_server:error,2025-04-20T04:35:05.593-04:00,ns_1@10.101.105.182:service_status_keeper_worker<0.436.0>:rest_utils:get_json_local:63]Request to (indexer) getIndexStatus failed: {error,

{econnrefused,

[{lhttpc_client,send_request,1,

[{file,

“/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/lhttpc/lhttpc_client.erl”},

{line,220}]},

{lhttpc_client,execute,9,

[{file,

“/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/lhttpc/lhttpc_client.erl”},

{line,169}]},

{lhttpc_client,request,9,

[{file,

“/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/lhttpc/lhttpc_client.erl”},

{line,92}]}]}}

[ns_server:error,2025-04-20T04:35:05.594-04:00,ns_1@10.101.105.182:service_status_keeper-index<0.437.0>:service_status_keeper:handle_cast:119]Service service_index returned incorrect status

[ns_server:error,2025-04-20T04:35:10.596-04:00,ns_1@10.101.105.182:service_status_keeper_worker<0.436.0>:rest_utils:get_json_local:63]Request to (indexer) getIndexStatus failed: {error,

{econnrefused,

[{lhttpc_client,send_request,1,

[{file,

“/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/lhttpc/lhttpc_client.erl”},

{line,220}]},

{lhttpc_client,execute,9,

[{file,

“/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/lhttpc/lhttpc_client.erl”},

{line,169}]},

{lhttpc_client,request,9,

[{file,

“/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/lhttpc/lhttpc_client.erl”},

{line,92}]}]}}

[ns_server:error,2025-04-20T04:35:10.596-04:00,ns_1@10.101.105.182:service_status_keeper-index<0.437.0>:service_status_keeper:handle_cast:119]Service service_index returned incorrect status

[ns_server:error,2025-04-20T04:35:15.598-04:00,ns_1@10.101.105.182:service_status_keeper_worker<0.436.0>:rest_utils:get_json_local:63]Request to (indexer) getIndexStatus failed: {error,

{econnrefused,

[{lhttpc_client,send_request,1,

[{file,

“/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/lhttpc/lhttpc_client.erl”},

{line,220}]},

{lhttpc_client,execute,9,

[{file,

“/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/lhttpc/lhttpc_client.erl”},

{line,169}]},

{lhttpc_client,request,9,

[{file,

“/home/couchbase/jenkins/workspace/couchbase-server-unix/couchdb/src/lhttpc/lhttpc_client.erl”},

{line,92}]}]}}

[ns_server:error,2025-04-20T04:35:15.598-04:00,ns_1@10.101.105.182:service_status_keeper-index<0.437.0>:service_status_keeper:handle_cast:119]Service service_index returned incorrect status

5

I haven’t raised a support ticket yet as I’m still trying to find a solution. I’ll definitely raise one.