Rebalance failed. See logs for detailed reason. You can try again

Hello,

I’m setting up a cluster with two nodes. When i try to rebalance i receive the follow error:

Service ‘cbas’ exited with status 2. Restarting. Messages:
(0xc4201383c0, 0xc4201383d8, 0xc4203a71c0, 0x20, 0x0, 0x0, 0x0, 0xc4203a7240, 0x19, 0xc4205fe0e0, …)
goproj/src/github.com/couchbaselabs/cbas/cbas/manager.go:771 +0x2e5
main.(*Mgr).handleFailedCcRemoval.func1(0xc4201383d8)
goproj/src/github.com/couchbaselabs/cbas/cbas/manager.go:738 +0x73
main.(*Mgr).updateStateLOCKED(0xc4201383c0, 0xc420325ca8)
goproj/src/github.com/couchbaselabs/cbas/cbas/manager.go:1041 +0x3a
main.(*Mgr).updateState(0xc4201383c0, 0xc420325ca8)
goproj/src/github.com/couchbaselabs/cbas/cbas/manager.go:1037 +0x73
main.(*Mgr).handleFailedCcRemoval(0xc4201383c0, 0xc4203a71c0, 0x20, 0x0, 0x0, 0x0, 0xc4203a7240, 0x19, 0xc4205fe0e0, 0x2, …)
goproj/src/github.com/couchbaselabs/cbas/cbas/manager.go:741 +0x15a
main.(*Mgr).startAndMonitorRebalance(0xc4201383c0, 0xc4203a71c0, 0x20, 0x0, 0x0, 0x0, 0xc4203a7240, 0x19, 0xc4205fe0e0, 0x2, …)
goproj/src/github.com/couchbaselabs/cbas/cbas/manager.go:481 +0xcc6
created by main.(*Mgr).StartTopologyChange
goproj/src/github.com/couchbaselabs/cbas/cbas/manager.go:424 +0xa7b
[goport(/opt/couchbase/bin/cbas)] 2020/02/25 08:42:44 child process exited with status 2

When i check the logs in indexer.log i can see the following error:

[Error] PeerPipe.doRecieve() : ecounter error when received mesasage from Peer private_ip:41502. Error = EOF. Kill Pipe.

Rebalance is failing when is on 20%.

Currently i’m run out of ideas.

Thank you in advance,
Antonia

Hi,

What version of Couchbase Server is this? From the snippet of the error, it looks like it may be an issue which was fixed in 6.0.3, MB-35688.

If you are on a version newer than that, we will probably need to get our hands on cbcollect info logs from the nodes in order to help us figure out what’s going on…

Thanks!

-MDB