[MB-6662] XDC: worse performance with expired items Created: 14/Sep/12  Updated: 19/Sep/12  Resolved: 17/Sep/12

Status: Resolved
Project: Couchbase Server
Component/s: cross-datacenter-replication
Affects Version/s: None
Fix Version/s: None
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Pavel Paulau Assignee: Chiyoung Seo
Resolution: Fixed Votes: 0
Labels: pblock
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: VMs, CentOS 6.2, 4-to-4 nodes, build 1723

Attachments: GZip Archive debug.tar.gz     PNG File Junyi-dest Screen Shot 2012-09-14 at 6.26.39 PM.png     PNG File Junyi-Pavel-test Screen Shot 2012-09-14 at 7.39.21 PM.png     PNG File JUNYI-Source Screen Shot 2012-09-14 at 6.35.10 PM.png     PDF File xperf-mixed-1-bi.loop_2.0.0-1723-rel-enterprise_2.0.0-1723-rel-enterprise_DEST_Sep-14-2012_11-45-45.pdf     PDF File xperf-mixed-1-bi.loop_2.0.0-1723-rel-enterprise_2.0.0-1723-rel-enterprise_SOURCE_Sep-14-2012_11-42-14.pdf     PDF File xperf-mixed-1-uni.loop_2.0.0-1723-rel-enterprise_2.0.0-1723-rel-enterprise_SOURCE_Sep-13-2012_20-27-31.pdf     PDF File xperf-read-1-uni.loop_2.0.0-1723-rel-enterprise_2.0.0-1723-rel-enterprise_DEST_Sep-13-2012_20-32-02.pdf    

 Description   
Unidir symptoms (expiration ratio: 3%, expiration time: 5 minutes):
-- extremely high XDC ops/sec and XDC gets/sec rate on destination side. At the same time actual replication rate (new items/sec) is very low.
-- XDC queue grows all the time on source side (unlike case w/o expired items)

 Comments   
Comment by Pavel Paulau [ 14/Sep/12 ]
Ketaki,
Please update/close your bug and assign this issue to Junyi.

http://www.couchbase.com/issues/browse/MB-6563
Comment by Pavel Paulau [ 14/Sep/12 ]
jic: reports.
Comment by Junyi Xie [ 14/Sep/12 ]
Known issue. Duplicate of MB-6563.
Comment by Ketaki Gangal [ 14/Sep/12 ]
Closed MB-6563, using this bug to track expired items.
Comment by Pavel Paulau [ 14/Sep/12 ]
Tried build 1723 w/o expired items and everything looks fine (at least as it was before).
Comment by Junyi Xie [ 14/Sep/12 ]
OK, Pavel reproduced the issue. Now it looks like it is a bug in ep_engine incorrectly account the getMeta ops.

I deleted replication on the source cluster at 19:29PM and XDCR activity is stopped. Look at logs at 10.2.2.190, there is no capi_replication trace after 19:29PM. However, on the UI it keeps showing very high rate of getMeta in XDC section.

Please look at screen shot: Junyi-Pavel-test Screen Shot 2012-09-14 at 7.39.21 PM.png from one of destination 10.2.2.190


By cbstats, you see the get_meta ops keep increasing (while del/set ops have no change) after I stopped the XDCR completely.

Junyis-MacBook-Pro:management junyi$ cbstats 10.2.2.190:11211 all | grep meta
 ep_num_ops_del_meta: 34329
 ep_num_ops_get_meta: 41910986
 ep_num_ops_set_meta: 14257849
Junyis-MacBook-Pro:management junyi$ cbstats 10.2.2.190:11211 all | grep meta
 ep_num_ops_del_meta: 34329
 ep_num_ops_get_meta: 44140162
 ep_num_ops_set_meta: 14257849
Junyis-MacBook-Pro:management junyi$ cbstats 10.2.2.190:11211 all | grep meta
 ep_num_ops_del_meta: 34329
 ep_num_ops_get_meta: 44226426
 ep_num_ops_set_meta: 14257849
Junyis-MacBook-Pro:management junyi$ cbstats 10.2.2.190:11211 all | grep meta
 ep_num_ops_del_meta: 34329
 ep_num_ops_get_meta: 44254887
 ep_num_ops_set_meta: 14257849
Junyis-MacBook-Pro:management junyi$ cbstats 10.2.2.190:11211 all | grep meta
 ep_num_ops_del_meta: 34329
 ep_num_ops_get_meta: 44273589
 ep_num_ops_set_meta: 14257849


Since no activity after 19:29PM, these getMeta should come from CAPI/XDCR, looks more likely the ep_engine incorrectly account the getMeta operations. It is a blocker IMHO, and hand over to Chiyoung for investigation.

Thanks.

Comment by Chiyoung Seo [ 17/Sep/12 ]
http://review.couchbase.org/#/c/20927/1
Comment by Pavel Paulau [ 19/Sep/12 ]
Verified.
Comment by Thuan Nguyen [ 19/Sep/12 ]
Integrated in github-ep-engine-2-0 #431 (See [http://qa.hq.northscale.net/job/github-ep-engine-2-0/431/])
    MB-6662 Ignore the item's expiration time for SET_WITH_META (Revision 5a9b619ae5c9c65c3a1e2902d57f60d0e4915c59)

     Result = SUCCESS
Chiyoung Seo :
Files :
* src/stored-value.hh
Generated at Wed May 22 20:40:40 CDT 2013 using JIRA 5.2.4#845-sha1:c9f4cc41abe72fb236945343a1f485c2c844dac9.