Pesistence, replication & data loss
In our environment, there a 4 Couchbase servers in the cluster with bucket called "cm-config" configured to contain 3 replicas. From time to time, some of the nodes become unreachable which leads to nodes being automatically failovered. However, when 2 of the 4 nodes are failovered at the same time there appears to be some data loss. The data that was missing was set quite a number of weeks previously, so I would have expected the it would have been correctly persisted and replicated in that time which should have meant that once 2 nodes were failovered, the replicas on the other servers would now become replicas. Is this correct? Would you expect to see some data loss in the above scenario? If so, is there some way we can lessen the chances of data loss?
To continue from the comment discussion.
You cannot enable auto-failover on a 2 node cluster (and the auto-failover will not happen when you cluster is reduced to 2 nodes) because you cannot move to a 1 node cluster... in this case if you end up with a single node cluster you do not have any replicas so you may lose data.
This is just a safety net that force deployment to always have 2 node or more.