Couchbase Server

Unusual ops/sec ( spike) 40k ops/sec during rebalancing.

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: None
  • Fix Version/s: 2.0-beta
  • Component/s: couchbase-bucket, ns_server
  • Security Level: Public
  • Labels:
    None
  • Environment:
    2.0-1621

Description

i noticed this when running some of the tests which load data and then they do rebalancing. seems like during rebalancing there are some activities between nodes in the cluster that shows up in the UI as ops/sec.
do we have a bug for this ?
the set/get ops/sec should only show up for front end traffic

-Farshid

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Farshid Ghods added a comment -
we observed this again after rebalancing when replication was going on
Show
Farshid Ghods added a comment - we observed this again after rebalancing when replication was going on
Hide
Permalink
Farshid Ghods added a comment -
Chiyuong,

are there any recent changes to replication stats ( as you can see in the UI sets/gets/ per second is zero but the ops/sec in the UI shows as 40k )
Show
Farshid Ghods added a comment - Chiyuong, are there any recent changes to replication stats ( as you can see in the UI sets/gets/ per second is zero but the ops/sec in the UI shows as 40k )
Hide
Permalink
Farshid Ghods added a comment -
also i should have grabbed diags and stats from the cluster . so please close the bug as not complete or assign this back to me if this is something not known
Show
Farshid Ghods added a comment - also i should have grabbed diags and stats from the cluster . so please close the bug as not complete or assign this back to me if this is something not known
Hide
Permalink
Chiyoung Seo added a comment -
It seems to me that ops/sec in the UI also includes the replication rate, which is wrong.
Show
Chiyoung Seo added a comment - It seems to me that ops/sec in the UI also includes the replication rate, which is wrong.
Hide
Permalink
Farshid Ghods added a comment -
Andrei,

if you observe this can you grab cbstats all and cbstats raw from command line and post it here

can you also look at UI and see which stats it uses for showing the front end ops/sec
Show
Farshid Ghods added a comment - Andrei, if you observe this can you grab cbstats all and cbstats raw from command line and post it here can you also look at UI and see which stats it uses for showing the front end ops/sec
Hide
Permalink
Farshid Ghods added a comment -
chiyoung should i assign this to Andrei for more info or leave this assigned to Mike ?
Show
Farshid Ghods added a comment - chiyoung should i assign this to Andrei for more info or leave this assigned to Mike ?
Hide
Permalink
Chiyoung Seo added a comment -
Mike,

Please look at which stats are used for showing ops/sec in the UI. If those stats are also updated for replication stream, then this is a bug in ep-engine.
Show
Chiyoung Seo added a comment - Mike, Please look at which stats are used for showing ops/sec in the UI. If those stats are also updated for replication stream, then this is a bug in ep-engine.
Hide
Permalink
Mike Wiederhold added a comment -
http://review.couchbase.org/#/c/20146/
Show
Mike Wiederhold added a comment - http://review.couchbase.org/#/c/20146/
Hide
Permalink
Thuan Nguyen added a comment -
Integrated in github-ep-engine-2-0 #412 (See [http://qa.hq.northscale.net/job/github-ep-engine-2-0/412/])
    MB-6396: Remove unneeded includes from blackhole kv-store (Revision d416ccaa6531a0ad48b173c43baab2f312c970fc)
MB-6396: Remove unused epstats from couch-notifier (Revision a43e8a6dfab220702adc40bd83e3d6c31da44ac8)
MB-6396: Incr xdcr stats in ep_engine.cc command call (Revision 579195836de1ed743248eee031887fa465350725)

     Result = SUCCESS
Mike Wiederhold :
Files :
* src/kvstore.cc
* src/blackhole-kvstore/blackhole.hh
* src/blackhole-kvstore/blackhole.cc

Mike Wiederhold :
Files :
* src/couch-kvstore/couch-notifier.cc
* src/couch-kvstore/couch-notifier.hh

Mike Wiederhold :
Files :
* src/ep_engine.cc
* src/ep.cc
Show
Thuan Nguyen added a comment - Integrated in github-ep-engine-2-0 #412 (See [ http://qa.hq.northscale.net/job/github-ep-engine-2-0/412/ ])      MB-6396 : Remove unneeded includes from blackhole kv-store (Revision d416ccaa6531a0ad48b173c43baab2f312c970fc) MB-6396 : Remove unused epstats from couch-notifier (Revision a43e8a6dfab220702adc40bd83e3d6c31da44ac8) MB-6396 : Incr xdcr stats in ep_engine.cc command call (Revision 579195836de1ed743248eee031887fa465350725)      Result = SUCCESS Mike Wiederhold : Files : * src/kvstore.cc * src/blackhole-kvstore/blackhole.hh * src/blackhole-kvstore/blackhole.cc Mike Wiederhold : Files : * src/couch-kvstore/couch-notifier.cc * src/couch-kvstore/couch-notifier.hh Mike Wiederhold : Files : * src/ep_engine.cc * src/ep.cc
Hide
Permalink
Ketaki Gangal added a comment -
Verified on build 1701.
Show
Ketaki Gangal added a comment - Verified on build 1701.

People

  • Assignee:
    Mike Wiederhold
    Reporter:
    Ketaki Gangal
Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: