Couchbase Server

Every item sent twice in bi-directional XDCR

Details

  • Type: Improvement Improvement
  • Status: Resolved Resolved
  • Priority: Critical Critical
  • Resolution: Won't Fix
  • Affects Version/s: 2.0
  • Fix Version/s: 2.1
  • Component/s: cross-datacenter-replication
  • Security Level: Public
  • Labels:
    None

Description

From my understanding, the XDCR replication code does not yet have the ability to distinguish incoming data from a remote cluster versus incoming application data.

What this means is that every mutation sent from the remote cluster is attempted to be sent back. No data problems are encountered because of our conflict resolution, but this presents a significant performance and bandwidth limitation.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Junyi Xie added a comment -
Let me make it clear, we do not send item twice during bi-directional xdcr. The destination will only send getMeta request back to source for each key. The bounce-back traffic is key and metadata only, not the document body.

The reason behind this, during bi-dir xdcr, the replicator at destination side got notified about every persisted mutation from storage, but it has no idea where this mutation came from (say, from front-end workload, from source cluster, or from another cluster). Therefore, it just tried to replicate it to every destination which has been configured by user, including the source cluster if bi-dir xdcr.

Compared with uni-dir xdcr, the bandwidth cost we pay in bi-dir xdcr is just key+metadata in returning traffic from destination back to source.
Show
Junyi Xie added a comment - Let me make it clear, we do not send item twice during bi-directional xdcr. The destination will only send getMeta request back to source for each key. The bounce-back traffic is key and metadata only, not the document body. The reason behind this, during bi-dir xdcr, the replicator at destination side got notified about every persisted mutation from storage, but it has no idea where this mutation came from (say, from front-end workload, from source cluster, or from another cluster). Therefore, it just tried to replicate it to every destination which has been configured by user, including the source cluster if bi-dir xdcr. Compared with uni-dir xdcr, the bandwidth cost we pay in bi-dir xdcr is just key+metadata in returning traffic from destination back to source.
Hide
Permalink
Junyi Xie added a comment -
Link to discussion on CBSE-291
Show
Junyi Xie added a comment - Link to discussion on CBSE-291
Hide
Permalink
Perry Krug added a comment -
Thanks for the clarification Junyi. I understand that we're not sending the document body, but let's take a quick example of 800M items (which is what a customer is testing right now).

800M items at 120bytes each just for the metadata (not even including the key) is another 90GB of bandwidth being used. That is pure dollars down the drain for the customer. At 70byte keys, that's another 50GB of data sent...
Show
Perry Krug added a comment - Thanks for the clarification Junyi. I understand that we're not sending the document body, but let's take a quick example of 800M items (which is what a customer is testing right now). 800M items at 120bytes each just for the metadata (not even including the key) is another 90GB of bandwidth being used. That is pure dollars down the drain for the customer. At 70byte keys, that's another 50GB of data sent...
Hide
Permalink
Dipti Borkar added a comment -
this is what allows us to have a purely multi-master system with conflict resolution. eventually it will help adding conflict management as well.
This request simply falls into optimizing XDCR for performance. and will be folded into those bugs.

Show
Dipti Borkar added a comment - this is what allows us to have a purely multi-master system with conflict resolution. eventually it will help adding conflict management as well. This request simply falls into optimizing XDCR for performance. and will be folded into those bugs.
Hide
Permalink
Dipti Borkar added a comment -
this behaviorimplementation specific. the requirement is to reduce latency and network traffic and this is something which is being worked using different approaches.
Show
Dipti Borkar added a comment - this behaviorimplementation specific. the requirement is to reduce latency and network traffic and this is something which is being worked using different approaches.

People

  • Assignee:
    Dipti Borkar
    Reporter:
    Perry Krug
Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: