Cluster tries to access the removed node after a node was removed and cluster rebalanced.
I encountered the following behavior on one of my couchbase clusters:
After removing the node from cluster and rebalancing, the cluster continues sending data to the removed node
(when storing new cache keys) and the client fails with the following error:
"Failed to read from the socket 'REMOVED-NODE:11211'"
The cluster log contains the following records:
Node 'ns_1@REMOVED-NODE' is leaving cluster.
Node 'ns_1@OTHER-NODE1' saw that node 'ns_1@REMOVED-NODE' went down.
Node 'ns_1@OTHER-NODE2' saw that node 'ns_1@REMOVED-NODE' went down.
Started rebalancing bucket default.
Rebalance completed successfully.
This happens both when the removed node is up and down.
Any ideas how to solve this issue?