ok so in this context I think it makes sense. You use 2 replicas, but wait for just 1 replica to acknowledge the data (ReplicateTo.ONE). When you failover the node, one of the 2 replicas becomes active. That means that as long as you don’t rebalance, you have 1 replica less for that subset of data (eg. old master is down, old replica 1 is now master, old replica 2 is not replica 1).
When you’re inserting during the failover, you could run into 3 scenarios:
- replication has time to reach only replica1, but replica2 is promoted: replica2 never saw the document and so you get a durability exception
- replication has time to reach only replica1, and it is the same replica that is promoted: this case replica2 will be detected as “behind” and catch up. => success
- replication has time to reach both replicas before the failover happens. => success
Generally in order to be absolutely sure the best is to use ReplicateTo.TWO (in your case), that is wait for all replicas to ack.
You should indeed retry if you receive a DurabilityException