Couchbase Server

XDCR: replication lag on ec2 periodically reaches 1 minute and more

Details

  • Type: Bug Bug
  • Status: In Progress In Progress
  • Priority: Critical Critical
  • Resolution: Unresolved
  • Affects Version/s: 2.0
  • Fix Version/s: 2.1
  • Component/s: cross-datacenter-replication
  • Security Level: Public
  • Labels:
    None
  • Environment:
    ec2 High-Memory Double Extra Large Instance, 4 east <-> 4 west, bidir
    build 1967

Description

While mutation rate is about 6K ops/sec replication queue reaches 400K items from time to time (see page 37, east coast (source) is worse for some reason).

I doesn't seem to be an issue, but I never saw requirements for delays. So wondering.

This is ec2 specific issue (taking into account laws of physics).

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Steve Yen added a comment -
per bug scrub, while the fix may not go into 2.0, need continued triaging -- looks like possible recent regression. assigning to junyi.
Show
Steve Yen added a comment - per bug scrub, while the fix may not go into 2.0, need continued triaging -- looks like possible recent regression. assigning to junyi.
Hide
Permalink
Junyi Xie added a comment -
The doc to replicate on page 37 is highly correlated with "doc disk size" (page 22, 23 27), doc fragmentation (page 28). Looks like an issue known to us for a while: the compaction competes I/O with ep_engine flusher and XDCR unable to replicate mutation as quickly as before compaction.


Pavel, to verify, you could turn off compaction, or change the compaction interval (say, compacting once per hour), and re-run the test.
Show
Junyi Xie added a comment - The doc to replicate on page 37 is highly correlated with "doc disk size" (page 22, 23 27), doc fragmentation (page 28). Looks like an issue known to us for a while: the compaction competes I/O with ep_engine flusher and XDCR unable to replicate mutation as quickly as before compaction. Pavel, to verify, you could turn off compaction, or change the compaction interval (say, compacting once per hour), and re-run the test.
Hide
Permalink
Farshid Ghods added a comment -
per bug scrub

Damien,
could you please have a look ?
Show
Farshid Ghods added a comment - per bug scrub Damien, could you please have a look ?
Hide
Permalink
Junyi Xie added a comment -
Known issue of compactor. Not a block IMHO.
Show
Junyi Xie added a comment - Known issue of compactor. Not a block IMHO.
Hide
Permalink
Junyi Xie added a comment -
I am not sure if there is any quick fix for 2.0.1 since it is a known performance issue: resource competition between different components.

Mark the fix version to 2.1 temporarily.
Show
Junyi Xie added a comment - I am not sure if there is any quick fix for 2.0.1 since it is a known performance issue: resource competition between different components. Mark the fix version to 2.1 temporarily.
Hide
Permalink
Jin Lim added a comment -
From XDCR meeting:

We need to rerun the test as follows
 a. If easier, rerun the test with on 2-2 EC2 (west to east)
 b. if 2-2, reduce the front load to 3k/ops as well
 c. use the latest 2.0.1 build (RC)
 d. run the test twice, one with compaction and the other without compaction
 e. goal of running this test is to verify that the lag is really from the io competition between xdcr and compaction
Show
Jin Lim added a comment - From XDCR meeting: We need to rerun the test as follows  a. If easier, rerun the test with on 2-2 EC2 (west to east)  b. if 2-2, reduce the front load to 3k/ops as well  c. use the latest 2.0.1 build (RC)  d. run the test twice, one with compaction and the other without compaction  e. goal of running this test is to verify that the lag is really from the io competition between xdcr and compaction
Hide
Permalink
Junyi Xie added a comment -
Thanks Jin for updates

Pavel, the 3K mutation rate ops/sec is per cluster, so if you are running the test on 2:2 configuration, it is 1.5K ops/sec per node. This is to be consistent with your original test where there was a 6K mutation rate per a 4-node cluster.


Show
Junyi Xie added a comment - Thanks Jin for updates Pavel, the 3K mutation rate ops/sec is per cluster, so if you are running the test on 2:2 configuration, it is 1.5K ops/sec per node. This is to be consistent with your original test where there was a 6K mutation rate per a 4-node cluster.
Hide
Permalink
Pavel Paulau added a comment -
Results look much better when compaction disabled.
Show
Pavel Paulau added a comment - Results look much better when compaction disabled.
Hide
Permalink
Junyi Xie added a comment - - edited
Brief summary:

In the latest test with build 2.0.1-170,

1) With compaction, the peak "docs to replicate" is reduced from 400K (2.0.0-1967) down to 60-70K.
2) After we return off compaction, the the peak "docs to replicate" is further reduced significantly to < 10K, which we shall be able to flush in 10 seconds, confirmed by the stats "XDCR lag diff(total − persistance)" in report
3) In 2.0.1-170, persistence time accounts most of the lag in both tests (w/ and w/o compaction), although in the test w/o compaction the latency is improved by >50% (see stats "XDCR lag time" in both reports)

I think

1) from 2.0.0 to 2.0.1, there seems significant improvement in storage layer because even with compaction, we see much reduced backlog.
2) in 2.0.1, compaction still competes I/O with XDCR, but the problem is alleviated quite a lot.

Show
Junyi Xie added a comment - - edited Brief summary: In the latest test with build 2.0.1-170, 1) With compaction, the peak "docs to replicate" is reduced from 400K (2.0.0-1967) down to 60-70K. 2) After we return off compaction, the the peak "docs to replicate" is further reduced significantly to < 10K, which we shall be able to flush in 10 seconds, confirmed by the stats "XDCR lag diff(total − persistance)" in report 3) In 2.0.1-170, persistence time accounts most of the lag in both tests (w/ and w/o compaction), although in the test w/o compaction the latency is improved by >50% (see stats "XDCR lag time" in both reports) I think 1) from 2.0.0 to 2.0.1, there seems significant improvement in storage layer because even with compaction, we see much reduced backlog. 2) in 2.0.1, compaction still competes I/O with XDCR, but the problem is alleviated quite a lot.
Hide
Permalink
Junyi Xie added a comment -
Not a functional bug. I am not sure what I can do at this time. Hand over back to PM.
Show
Junyi Xie added a comment - Not a functional bug. I am not sure what I can do at this time. Hand over back to PM.
Hide
Permalink
Maria McDuff added a comment -
assigning to anil per bug scrub. will discuss offline what other optimization work can be done within the 2.0.2 timeframe.
Show
Maria McDuff added a comment - assigning to anil per bug scrub. will discuss offline what other optimization work can be done within the 2.0.2 timeframe.
Hide
Permalink
Thuan Nguyen added a comment -
Integrated in ui-testing #35 (See [http://qa.hq.northscale.net/job/ui-testing/35/])
    
     Result = SUCCESS
Show
Thuan Nguyen added a comment - Integrated in ui-testing #35 (See [ http://qa.hq.northscale.net/job/ui-testing/35/ ])           Result = SUCCESS

People

  • Assignee:
    Anil Kumar
    Reporter:
    Pavel Paulau
Vote (0)
Watch (9)

Dates

  • Created:
    Updated:

Agile