Cluster rebalancing is Failed during cluster creation with data and query and index and eventing service

Couchbase Server
1

i was creating cluster using terraform script in aws. so when i was creating cluster with 24 nodes(16 data, 3 index, 3 query, 2 eventing), terraform scripts saying that rebalance is failed. when i checked cluster logs below is the error i see.

Rebalance exited with reason {service_rebalance_failed,eventing,
{worker_died,
{‘EXIT’,<0.9493.4>,
{{badmatch,
{error,
{unknown_error,
<<“failed to get active eventing nodes in the cluster”>>}}},
[{service_rebalancer,rebalance_worker,1,
[{file,“src/service_rebalancer.erl”},
{line,170}]},
{proc_lib,init_p,3,
[{file,“proc_lib.erl”},{line,232}]}]}}}}.
Rebalance Operation Id = a242924667b74fa0a440f1ec79f764e6

can anyone suggest how to resolve this?

2

Hello prasannakumar

We have seen this error message before in Couchbase Server 6.0 when a user’s environment had rate limiting on their DNS server, which during normal operations was fine but the extensive requests during a rebalance caused it to fail due to timeouts. If you’re not on Couchbase Server 6.5 or later, I would highly suggest you upgrade and take advantage of the configurable time out option that was added to prevent these types of issues: reference: https://issues.couchbase.com/browse/MB-32428 .

Thank you,
Ian McCloy (Principal Product Manager, Couchbase)