Distance between two nodes
Fri, 03/01/2013 - 00:49
We have two datacenters close to each other and few are apart. There are few nodes that are in same datacenter but not in same room, these nodes are also not in same vlan.
What distance (in ms) is good enough for nodes to be in same cluster?
Or
What is the distance (in ms) when using XDCR is essential?
Within a cluster , intra-cluster replication happens in memory, if network latencies are higher there is a chance replicas will start falling behind. This depends obviously on many factors, your write rate , is your workload heavy read or heavy write or mixed. for a write heavy, I'd advice a much better network latency.
The cluster managers also communicate with each other. The larger the latencies, you may see timeout issues during rebalance or other topology based operations. there are customers who literally have 7-8 nodes in one data center and 7-8 nodes in another completely different data center but have a 20 GB inter connect. And the cluster behaves as 1 seamlessly. So don't have a distance in ms answer for you, it will depend on your use case and what your SLA is.
For XDCR, it can be very small ( some customers have a fiber - 40 GB inter connect between DCs) or large ( 1GB ). the protocol handles network failures etc quite well.
We see extremely high latencies on Amazon for example 100-200ms and with lower bandwidths , so these may not meet your SLAs.