Couchbase Server

Compress XDCR traffic

Details

  • Type: Improvement Improvement
  • Status: Open Open
  • Priority: Major Major
  • Resolution: Unresolved
  • Affects Version/s: 2.0
  • Fix Version/s: 2.1
  • Component/s: cross-datacenter-replication
  • Security Level: Public
  • Labels:
    None

Description

As XDCR will almost always be used across an internet/WAN link, doing all we can to reduce the bandwidth will be critical. Given that the data is already compressed on disk, couldn't it be sent that way and uncompressed on the destination side?

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Junyi Xie added a comment -
Today in XDCR we use base64 encoding of all user-data and transmit them over internet/WAM link, we batch and package, but do not compress them within XDCR at this time. If the document size is huge, it may make some sense to compress them to save some bandwidth. But today, from Performance and QE team, mostly the the document is around several k bytes, the batch created by XDCR is at most 2M bytes by default. Given the overhead of compression and decompression, I am not sure how much benefit to compress them.

Show
Junyi Xie added a comment - Today in XDCR we use base64 encoding of all user-data and transmit them over internet/WAM link, we batch and package, but do not compress them within XDCR at this time. If the document size is huge, it may make some sense to compress them to save some bandwidth. But today, from Performance and QE team, mostly the the document is around several k bytes, the batch created by XDCR is at most 2M bytes by default. Given the overhead of compression and decompression, I am not sure how much benefit to compress them.
Hide
Permalink
Perry Krug added a comment -
To be clear, the benefit is not for performance but for saving network bandwidth on customer links that they will very likely have to pay for each bit they transfer.
Show
Perry Krug added a comment - To be clear, the benefit is not for performance but for saving network bandwidth on customer links that they will very likely have to pay for each bit they transfer.
Hide
Permalink
Junyi Xie added a comment -
In 2.0.2, we will introduce "optimistic XDCR" which allows users to continuously tune XDCR performance in the range between favoring bandwidth and favoring latency. Given this new feature and to prioritize tasks, IMHO at this time it is not very urgent to compress the XDCR traffic to save bandwidth, therefore I tentatively mark the fix version as 2.1.
 
Show
Junyi Xie added a comment - In 2.0.2, we will introduce "optimistic XDCR" which allows users to continuously tune XDCR performance in the range between favoring bandwidth and favoring latency. Given this new feature and to prioritize tasks, IMHO at this time it is not very urgent to compress the XDCR traffic to save bandwidth, therefore I tentatively mark the fix version as 2.1.  

People

  • Assignee:
    Junyi Xie
    Reporter:
    Perry Krug
Vote (0)
Watch (4)

Dates

  • Created:
    Updated: