Couchbase Server

XDC: worse performance with expired items

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Blocker Blocker
  • Resolution: Fixed
  • Affects Version/s: None
  • Fix Version/s: None
  • Component/s: cross-datacenter-replication
  • Security Level: Public
  • Labels:
  • Environment:
    VMs, CentOS 6.2, 4-to-4 nodes, build 1723

Description

Unidir symptoms (expiration ratio: 3%, expiration time: 5 minutes):
-- extremely high XDC ops/sec and XDC gets/sec rate on destination side. At the same time actual replication rate (new items/sec) is very low.
-- XDC queue grows all the time on source side (unlike case w/o expired items)

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Pavel Paulau added a comment -
Ketaki,
Please update/close your bug and assign this issue to Junyi.

http://www.couchbase.com/issues/browse/MB-6563
Show
Pavel Paulau added a comment - Ketaki, Please update/close your bug and assign this issue to Junyi. http://www.couchbase.com/issues/browse/MB-6563
Hide
Permalink
Pavel Paulau added a comment -
jic: reports.
Show
Pavel Paulau added a comment - jic: reports.
Hide
Permalink
Junyi Xie added a comment -
Known issue. Duplicate of MB-6563.
Show
Junyi Xie added a comment - Known issue. Duplicate of MB-6563 .
Hide
Permalink
Ketaki Gangal added a comment -
Closed MB-6563, using this bug to track expired items.
Show
Ketaki Gangal added a comment - Closed MB-6563 , using this bug to track expired items.
Hide
Permalink
Pavel Paulau added a comment -
Tried build 1723 w/o expired items and everything looks fine (at least as it was before).
Show
Pavel Paulau added a comment - Tried build 1723 w/o expired items and everything looks fine (at least as it was before).
Hide
Permalink
Junyi Xie added a comment -
OK, Pavel reproduced the issue. Now it looks like it is a bug in ep_engine incorrectly account the getMeta ops.

I deleted replication on the source cluster at 19:29PM and XDCR activity is stopped. Look at logs at 10.2.2.190, there is no capi_replication trace after 19:29PM. However, on the UI it keeps showing very high rate of getMeta in XDC section.

Please look at screen shot: Junyi-Pavel-test Screen Shot 2012-09-14 at 7.39.21 PM.png from one of destination 10.2.2.190


By cbstats, you see the get_meta ops keep increasing (while del/set ops have no change) after I stopped the XDCR completely.

Junyis-MacBook-Pro:management junyi$ cbstats 10.2.2.190:11211 all | grep meta
 ep_num_ops_del_meta: 34329
 ep_num_ops_get_meta: 41910986
 ep_num_ops_set_meta: 14257849
Junyis-MacBook-Pro:management junyi$ cbstats 10.2.2.190:11211 all | grep meta
 ep_num_ops_del_meta: 34329
 ep_num_ops_get_meta: 44140162
 ep_num_ops_set_meta: 14257849
Junyis-MacBook-Pro:management junyi$ cbstats 10.2.2.190:11211 all | grep meta
 ep_num_ops_del_meta: 34329
 ep_num_ops_get_meta: 44226426
 ep_num_ops_set_meta: 14257849
Junyis-MacBook-Pro:management junyi$ cbstats 10.2.2.190:11211 all | grep meta
 ep_num_ops_del_meta: 34329
 ep_num_ops_get_meta: 44254887
 ep_num_ops_set_meta: 14257849
Junyis-MacBook-Pro:management junyi$ cbstats 10.2.2.190:11211 all | grep meta
 ep_num_ops_del_meta: 34329
 ep_num_ops_get_meta: 44273589
 ep_num_ops_set_meta: 14257849


Since no activity after 19:29PM, these getMeta should come from CAPI/XDCR, looks more likely the ep_engine incorrectly account the getMeta operations. It is a blocker IMHO, and hand over to Chiyoung for investigation.

Thanks.

Show
Junyi Xie added a comment - OK, Pavel reproduced the issue. Now it looks like it is a bug in ep_engine incorrectly account the getMeta ops. I deleted replication on the source cluster at 19:29PM and XDCR activity is stopped. Look at logs at 10.2.2.190, there is no capi_replication trace after 19:29PM. However, on the UI it keeps showing very high rate of getMeta in XDC section. Please look at screen shot: Junyi-Pavel-test Screen Shot 2012-09-14 at 7.39.21 PM.png from one of destination 10.2.2.190 By cbstats, you see the get_meta ops keep increasing (while del/set ops have no change) after I stopped the XDCR completely. Junyis-MacBook-Pro:management junyi$ cbstats 10.2.2.190:11211 all | grep meta  ep_num_ops_del_meta: 34329  ep_num_ops_get_meta: 41910986  ep_num_ops_set_meta: 14257849 Junyis-MacBook-Pro:management junyi$ cbstats 10.2.2.190:11211 all | grep meta  ep_num_ops_del_meta: 34329  ep_num_ops_get_meta: 44140162  ep_num_ops_set_meta: 14257849 Junyis-MacBook-Pro:management junyi$ cbstats 10.2.2.190:11211 all | grep meta  ep_num_ops_del_meta: 34329  ep_num_ops_get_meta: 44226426  ep_num_ops_set_meta: 14257849 Junyis-MacBook-Pro:management junyi$ cbstats 10.2.2.190:11211 all | grep meta  ep_num_ops_del_meta: 34329  ep_num_ops_get_meta: 44254887  ep_num_ops_set_meta: 14257849 Junyis-MacBook-Pro:management junyi$ cbstats 10.2.2.190:11211 all | grep meta  ep_num_ops_del_meta: 34329  ep_num_ops_get_meta: 44273589  ep_num_ops_set_meta: 14257849 Since no activity after 19:29PM, these getMeta should come from CAPI/XDCR, looks more likely the ep_engine incorrectly account the getMeta operations. It is a blocker IMHO, and hand over to Chiyoung for investigation. Thanks.
Hide
Permalink
Chiyoung Seo added a comment -
http://review.couchbase.org/#/c/20927/1
Show
Chiyoung Seo added a comment - http://review.couchbase.org/#/c/20927/1
Hide
Permalink
Pavel Paulau added a comment -
Verified.
Show
Pavel Paulau added a comment - Verified.
Hide
Permalink
Thuan Nguyen added a comment -
Integrated in github-ep-engine-2-0 #431 (See [http://qa.hq.northscale.net/job/github-ep-engine-2-0/431/])
    MB-6662 Ignore the item's expiration time for SET_WITH_META (Revision 5a9b619ae5c9c65c3a1e2902d57f60d0e4915c59)

     Result = SUCCESS
Chiyoung Seo :
Files :
* src/stored-value.hh
Show
Thuan Nguyen added a comment - Integrated in github-ep-engine-2-0 #431 (See [ http://qa.hq.northscale.net/job/github-ep-engine-2-0/431/ ])      MB-6662 Ignore the item's expiration time for SET_WITH_META (Revision 5a9b619ae5c9c65c3a1e2902d57f60d0e4915c59)      Result = SUCCESS Chiyoung Seo : Files : * src/stored-value.hh

People

  • Assignee:
    Chiyoung Seo
    Reporter:
    Pavel Paulau
Vote (0)
Watch (2)

Dates

  • Created:
    Updated:
    Resolved: