Cluster tries to access the removed node after a node was removed and cluster rebalanced.
I encountered the following behavior on one of my couchbase clusters:
After removing the node from cluster and rebalancing, the cluster continues sending data to the removed node
(when storing new cache keys) and the client fails with the following error:
"Failed to read from the socket 'REMOVED-NODE:11211'"
The cluster log contains the following records:
Node 'ns_1@REMOVED-NODE' is leaving cluster.
Node 'ns_1@OTHER-NODE1' saw that node 'ns_1@REMOVED-NODE' went down.
Node 'ns_1@OTHER-NODE2' saw that node 'ns_1@REMOVED-NODE' went down.
Started rebalancing bucket default.
Rebalance completed successfully.
This happens both when the removed node is up and down.
Any ideas how to solve this issue?
Hi,
We're using Enyim.Caching v2.12.
The rebalancing was successful, but after it was finished we stared getting errors from the removed node.
This problem occures on one of our 2 dev environments and this is the only wierd behaviour we encountered so far (no other errors/issues).
The node was removed for testing purposes.
The most wierd behavior was in the following scenario:
I stopped the memcache on one the nodes (without removing the node) and the cluster stopped sending data to this node.
Then I removed the stopped node and rebalanced, and the cluster suddenly "remembered" about the removed node and started sending data to it...
Apparently it's not possible to work with a cluster with such behavior.
Could you please help us investigate the problem.
Thanks a lot,
Marina
The world's best Recruitment Agencies in Pakistan, Candor Group (Pvt) Ltd. Manpower Recruitment Business Management Outsourcing Group the leaders in providing manpower recruitment services for professionals in Pakistan & Gulf.
http://www.candorrecruitmentgroup.com » http://manpoweragenciesinpakistan.com/ » http://staffingagenciesinpakistan.com/ » http://www.candorrecruitmentgroup.blogspot.com/ » http://candorrecruitmentgroup.wordpress.com/ »
Learn about bundle deals with WOW! wow gold Internet,
You can always learn something new everyday. Impressive health articles
Well this one is really out of my idea
Am sorry cant help you
Hi,
Please provide the answers for the following to better understand the issue.
a) What is the Client SDK you are using and the version of it?
b) Did you get the error while rebalance operation or after the removal of node?
c) Any other errors from the application/client apart from this?
d) Do you have any history of client errors/issues?
e) What is the reason for removing the node?
Thanks,
Neo