http://www.couchbase.com/issues This file is an XML representation of an issue en-us 5.2.4 845 26-12-2012 http://www.couchbase.com/issues/browse/MB-6662 Couchbase Server Unidir symptoms (expiration ratio: 3%, expiration time: 5 minutes): <br/> -- extremely high XDC ops/sec and XDC gets/sec rate on destination side. At the same time actual replication rate (new items/sec) is very low. <br/> -- XDC queue grows all the time on source side (unlike case w/o expired items) VMs, CentOS 6.2, 4-to-4 nodes, build 1723 MB-6662

XDC: worse performance with expired items

Bug Blocker Resolved Fixed Public Chiyoung Seo Pavel Paulau pblock Fri, 14 Sep 2012 11:17:25 -0500 Wed, 19 Sep 2012 21:01:14 -0500 Mon, 17 Sep 2012 20:17:39 -0500 cross-datacenter-replication 0 2 Ketaki, <br/> Please update/close your bug and assign this issue to Junyi. <br/> <br/> <a href="http://www.couchbase.com/issues/browse/MB-6563">http://www.couchbase.com/issues/browse/MB-6563</a> jic: reports. Known issue. Duplicate of <a href="http://www.couchbase.com/issues/browse/MB-6563" title="Replication ops/sec drops to 0, on stop/start load from source cluster. [Load with only creates looks okay, load with expired items--&gt; replication rate drops very low.]"><strike>MB-6563</strike></a>. Closed <a href="http://www.couchbase.com/issues/browse/MB-6563" title="Replication ops/sec drops to 0, on stop/start load from source cluster. [Load with only creates looks okay, load with expired items--&gt; replication rate drops very low.]"><strike>MB-6563</strike></a>, using this bug to track expired items. Tried build 1723 w/o expired items and everything looks fine (at least as it was before). OK, Pavel reproduced the issue. Now it looks like it is a bug in ep_engine incorrectly account the getMeta ops. <br/> <br/> I deleted replication on the source cluster at 19:29PM and XDCR activity is stopped. Look at logs at 10.2.2.190, there is no capi_replication trace after 19:29PM. However, on the UI it keeps showing very high rate of getMeta in XDC section. <br/> <br/> Please look at screen shot: Junyi-Pavel-test Screen Shot 2012-09-14 at 7.39.21 PM.png from one of destination 10.2.2.190 <br/> <br/> <br/> By cbstats, you see the get_meta ops keep increasing (while del/set ops have no change) after I stopped the XDCR completely. <br/> <br/> Junyis-MacBook-Pro:management junyi$ cbstats 10.2.2.190:11211 all | grep meta <br/> &nbsp;ep_num_ops_del_meta: 34329 <br/> &nbsp;ep_num_ops_get_meta: 41910986 <br/> &nbsp;ep_num_ops_set_meta: 14257849 <br/> Junyis-MacBook-Pro:management junyi$ cbstats 10.2.2.190:11211 all | grep meta <br/> &nbsp;ep_num_ops_del_meta: 34329 <br/> &nbsp;ep_num_ops_get_meta: 44140162 <br/> &nbsp;ep_num_ops_set_meta: 14257849 <br/> Junyis-MacBook-Pro:management junyi$ cbstats 10.2.2.190:11211 all | grep meta <br/> &nbsp;ep_num_ops_del_meta: 34329 <br/> &nbsp;ep_num_ops_get_meta: 44226426 <br/> &nbsp;ep_num_ops_set_meta: 14257849 <br/> Junyis-MacBook-Pro:management junyi$ cbstats 10.2.2.190:11211 all | grep meta <br/> &nbsp;ep_num_ops_del_meta: 34329 <br/> &nbsp;ep_num_ops_get_meta: 44254887 <br/> &nbsp;ep_num_ops_set_meta: 14257849 <br/> Junyis-MacBook-Pro:management junyi$ cbstats 10.2.2.190:11211 all | grep meta <br/> &nbsp;ep_num_ops_del_meta: 34329 <br/> &nbsp;ep_num_ops_get_meta: 44273589 <br/> &nbsp;ep_num_ops_set_meta: 14257849 <br/> <br/> <br/> Since no activity after 19:29PM, these getMeta should come from CAPI/XDCR, looks more likely the ep_engine incorrectly account the getMeta operations. It is a blocker IMHO, and hand over to Chiyoung for investigation. <br/> <br/> Thanks. <br/> <br/> <a href="http://review.couchbase.org/#/c/20927/1">http://review.couchbase.org/#/c/20927/1</a> Verified. Integrated in github-ep-engine-2-0 #431 (See [<a href="http://qa.hq.northscale.net/job/github-ep-engine-2-0/431/">http://qa.hq.northscale.net/job/github-ep-engine-2-0/431/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.couchbase.com/issues/browse/MB-6662" title="XDC: worse performance with expired items"><strike>MB-6662</strike></a> Ignore the item&#39;s expiration time for SET_WITH_META (Revision 5a9b619ae5c9c65c3a1e2902d57f60d0e4915c59)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> Chiyoung Seo : <br/> Files : <br/> * src/stored-value.hh<br/> Date of First Response Fri, 14 Sep 2012 16:56:43 -0500 Rank 3964 Time In Status