Haven’t heard from a higher priority node or a master, so I’m taking over

We are using a 2 nodes cluster on Couchbase version 3.0.1 Community Edition.
The node is dropping out of cluster and we have to add it back manually sometimes to the cluster.
We see that one of the node’s CPU utilisation is going high at about 90+ %. And we noticed that the node turns yellow for a couple of seconds during this time.
The operations per second for the buckets go down drastically.
The logs show the below line:
Haven’t heard from a higher priority node or a master, so I’m taking over. (Attached the log screenshot for reference).

We have not configured any XDCR.


Support will be appreciated.

Regards.

Hi Rajeev, The only scenario you receive that message “Haven’t heard from a higher priority node or a master, so I’m taking over.” is when nodes are not able to exchange heartbeats 10secs (default) and the nodes which is not sending the heartbeat is assigned orchestrator (master) node. Other node in the absence of orchestrator (master) node tries to takeover.

I would suggest you look if there is any network connectivity issues between these nodes.

HTP

Anil Kumar

Hi Anil,

Thanks for the reply.
Does this also happen because the servers maybe way too busy or this can happen only because of the network issue ?

Regards.

Hi Rajeev, Apologies for delay in my response. Yes, you’re right this could also happen due to server is overloaded and not able to respond.

Thanks!

Anil Kumar

Hi @anil

I’ve the same issue but I can’t find any network issues. I tested it with iperf during production and the performance is around 800Mb/s on a 1Gb/s line. Additionally I tested with netcat and telnet if all ports are open between the nodes.

Here is the issue I opened months ago:

I hope you can help me here.

Thanks
Mathias