Single node fails causing the entire cluster to be unstable?
I’m testing failover with Membase 1.6.5 in a 3 node cluster without replication.
As expected, shutting down a single membase node causes a 1/3 of the cache to be lost and the client application to begin logging ‘set fail’ events.
But, it seems the entire cache might be unstable in this state. When the node is down there is a 10x increase in sets/sec on the remaining nodes, but no increase item count.
Anyone seen this type of behavior before or know if a single node failing will cause the cluster to be unstable until the node is failed over and rebalanced. I had expected the other nodes to be unaffected when one node was shut down.