We are using a 2 nodes cluster on Couchbase version 3.0.1 Community Edition.
The node is dropping out of cluster and we have to add it back manually sometimes to the cluster.
We see that one of the node’s CPU utilisation is going high at about 90+ %. And we noticed that the node turns yellow for a couple of seconds during this time.
The operations per second for the buckets go down drastically.
The logs show the below line:
Haven’t heard from a higher priority node or a master, so I’m taking over. (Attached the log screenshot for reference).
Hi Rajeev, The only scenario you receive that message “Haven’t heard from a higher priority node or a master, so I’m taking over.” is when nodes are not able to exchange heartbeats 10secs (default) and the nodes which is not sending the heartbeat is assigned orchestrator (master) node. Other node in the absence of orchestrator (master) node tries to takeover.
I would suggest you look if there is any network connectivity issues between these nodes.
I’ve the same issue but I can’t find any network issues. I tested it with iperf during production and the performance is around 800Mb/s on a 1Gb/s line. Additionally I tested with netcat and telnet if all ports are open between the nodes.