Couchbase Server

XDC: worse performance with expired items

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Blocker Blocker
  • Resolution: Fixed
  • Affects Version/s: None
  • Fix Version/s: None
  • Component/s: cross-datacenter-replication
  • Security Level: Public
  • Labels:
  • Environment:
    VMs, CentOS 6.2, 4-to-4 nodes, build 1723

Description

Unidir symptoms (expiration ratio: 3%, expiration time: 5 minutes):
-- extremely high XDC ops/sec and XDC gets/sec rate on destination side. At the same time actual replication rate (new items/sec) is very low.
-- XDC queue grows all the time on source side (unlike case w/o expired items)

Activity

Ascending order - Click to sort in descending order
Steve Yen made changes -
Field Original Value New Value
Priority Major [ 3 ] Blocker [ 1 ]
Junyi Xie made changes -
Status Open [ 1 ] Resolved [ 5 ]
Resolution Duplicate [ 3 ]
Ketaki Gangal made changes -
Resolution Duplicate [ 3 ]
Status Resolved [ 5 ] Reopened [ 4 ]
Assignee Ketaki Gangal [ ketaki ] Junyi Xie [ junyi ]
Pavel Paulau made changes -
Description Unidir symptoms (3% of items expire every 5 minutes):
-- extremely high XDC ops/sec and XDC gets/sec rate on destination side. At the same time actual replication rate (new items/sec) is very low.
-- XDC queue grows all the time on source side (unlike case w/o expired items) 		Unidir symptoms (expiration ratio: 3%, expiration time: 5 minutes):
-- extremely high XDC ops/sec and XDC gets/sec rate on destination side. At the same time actual replication rate (new items/sec) is very low.
-- XDC queue grows all the time on source side (unlike case w/o expired items)
Junyi Xie made changes -
Attachment Junyi-dest Screen Shot 2012-09-14 at 6.26.39 PM.png [ 15053 ]
Attachment JUNYI-Source Screen Shot 2012-09-14 at 6.35.10 PM.png [ 15054 ]
Junyi Xie made changes -
Priority Blocker [ 1 ] Critical [ 2 ]
Junyi Xie made changes -
Assignee Junyi Xie [ junyi ] Ketaki Gangal [ ketaki ]
Pavel Paulau made changes -
Attachment debug.tar.gz [ 15055 ]
Junyi Xie made changes -
Attachment Junyi-Pavel-test Screen Shot 2012-09-14 at 7.39.21 PM.png [ 15056 ]
Junyi Xie made changes -
Priority Critical [ 2 ] Blocker [ 1 ]
Junyi Xie made changes -
Assignee Ketaki Gangal [ ketaki ] Chiyoung Seo [ chiyoung ]
Junyi Xie made changes -
Comment [ on VMs, set up a 4->4 unidirectional XDCR with about 8M items, on source side, loading it with 2-3K ops/sec using the same workload ( 3% of items expiration time: 5 minutes) in Pavel's test.

Concurrent mcsoda data loading:

lib/perf_engines/mcsoda.py 10.3.3.61:11211 vbuckets=1024 doc-gen=0 doc-cache=0 ratio-creates=1 ratio-sets=1 ratio-expirations=0.03 expiration=300 ratio-deletes=0.04 min-value-size=2,3 max-items=20000000 exit-after-creates=0 prefix=XDCR_10
lib/perf_engines/mcsoda.py 10.3.3.61:11211 vbuckets=1024 doc-gen=0 doc-cache=0 ratio-creates=1 ratio-sets=1 ratio-expirations=0.03 expiration=300 ratio-deletes=0.04 min-value-size=2,3 max-items=20000000 exit-after-creates=0 prefix=XDCR_9


Observation:
1. on destination side, the replication rate is stable, around 1-3k which is similar to the testcase w/o expired items. See the screenshot attached.
Need to figure out how to reproduce it.

2. on the source side, the XDC queue slowly increase to > 1M items, apparently the replication speed is hard to always catch up with front-end ops.

3. when compactor is running at dest, stats become choppy, not surprised, we know that.


]
Junyi Xie made changes -
Comment [ Downgrade the priority until we can reproduce it. ]
Junyi Xie made changes -
Comment [ Ketaki, please reproduce it on VM if possible, and point me to the clusters. Thanks. ]
Chiyoung Seo made changes -
Status Reopened [ 4 ] Resolved [ 5 ]
Resolution Fixed [ 1 ]

People

  • Assignee:
    Chiyoung Seo
    Reporter:
    Pavel Paulau
Vote (0)
Watch (2)

Dates

  • Created:
    Updated:
    Resolved: