Couchbase Server

ns_server needs better way to calculate TAP stats.

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: 2.0
  • Fix Version/s: 2.0.2
  • Component/s: couchbase-bucket
  • Security Level: Public
  • Labels:
    None

Description

In order to calculate TAP stats ns_server requests "tapagg _" stats from ep-engine and differentiates them to calculate rates. But the problem with this approach is that ep-engine aggregates stats only for the TAP streams that currently exist. Hence ns_server is not able to account for part of the information because it's completely lost whenever some TAP stream is killed. What's more, it prevented us from implementing more detailed rebalance progress because the estimated number of transferred items was significantly different from the actual one. So we'll probably need some way to keep aggregated stats in ep-engine while rebalance is in progress.

Issue Links

blocks

Improvement - An improvement or enhancement to an existing feature or task. MB-8013 Implement detailed rebalance progress report

  • Blocker - Blocks development and/or testing work, production could not run.
  • Resolved - A resolution has been taken, and it is awaiting verification by reporter. From here issues are either reopened, or are closed.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Farshid Ghods added a comment -
per bug scrub.

moving this to 2.0.2 unless you think this is an important change required for 2.0.1 release . detailed progress report for rebalancing is not in 2.0.1 scope
Show
Farshid Ghods added a comment - per bug scrub. moving this to 2.0.2 unless you think this is an important change required for 2.0.1 release . detailed progress report for rebalancing is not in 2.0.1 scope
Hide
Permalink
Aliaksey Artamonau added a comment -
We need this to provide an estimate of transferred items during rebalance.
Show
Aliaksey Artamonau added a comment - We need this to provide an estimate of transferred items during rebalance.
Hide
Permalink
Aleksey Kondratenko added a comment -
Spotted removal from 2.0.2. CBD-184 (detailed rebalance progress report) is planned for 2.0.2 however.
Show
Aleksey Kondratenko added a comment - Spotted removal from 2.0.2. CBD-184 (detailed rebalance progress report) is planned for 2.0.2 however.
Hide
Permalink
Mike Wiederhold added a comment -
We might not have time for this which is why I moved it out. The issue is assigned to Dipti and she is welcome to move it back to 2.0.2, but things are already tight.
Show
Mike Wiederhold added a comment - We might not have time for this which is why I moved it out. The issue is assigned to Dipti and she is welcome to move it back to 2.0.2, but things are already tight.
Hide
Permalink
Aleksey Kondratenko added a comment -
Given that original tapagg was implemented by "us" (well Aaron, but he was part of ns_server back then), we may consider giving that ep-engine work to us too.
Show
Aleksey Kondratenko added a comment - Given that original tapagg was implemented by "us" (well Aaron, but he was part of ns_server back then), we may consider giving that ep-engine work to us too.
Hide
Permalink
Mike Wiederhold added a comment -
That's fine with me. Let me know when you start working progress report stuff. If I have time I will do the implementation otherwise if you guys can take care of it that would be great.
Show
Mike Wiederhold added a comment - That's fine with me. Let me know when you start working progress report stuff. If I have time I will do the implementation otherwise if you guys can take care of it that would be great.
Hide
Permalink
Maria McDuff added a comment -
per mike: it's currently under review. this is needed for rebalance progress indicator.
Show
Maria McDuff added a comment - per mike: it's currently under review. this is needed for rebalance progress indicator.
Hide
Permalink
Aleksey Kondratenko added a comment -
Note: I believe current set of patches are possibly confusing fix for a problem of losing tap stats on tap connection death and larger problem of implementing support for detailed rebalance progress. Those are related but in fact a bit separate. I.e. currently on UI we display near 0 value in "tap rebalance stats" column. And partly that's due to underlying problem of tap stats "annihilation" on tap connection death.
Show
Aleksey Kondratenko added a comment - Note: I believe current set of patches are possibly confusing fix for a problem of losing tap stats on tap connection death and larger problem of implementing support for detailed rebalance progress. Those are related but in fact a bit separate. I.e. currently on UI we display near 0 value in "tap rebalance stats" column. And partly that's due to underlying problem of tap stats "annihilation" on tap connection death.
Hide
Permalink
Maria McDuff added a comment -
so far, ns_server has what it needs to calculate the stats needed for rebalance progress. but for better accuracy, it still needs this issue to be addressed --- not a blocker. will revisit this issue if it turns out that the stats reporting is way inaccurate.
Show
Maria McDuff added a comment - so far, ns_server has what it needs to calculate the stats needed for rebalance progress. but for better accuracy, it still needs this issue to be addressed --- not a blocker. will revisit this issue if it turns out that the stats reporting is way inaccurate.
Hide
Permalink
Karen Zeller added a comment -
This is listed as "no docs needed" but when I read the ticket, it appears as if we will have different tap stats available in Web Console for 2.0.2. Please c
Show
Karen Zeller added a comment - This is listed as "no docs needed" but when I read the ticket, it appears as if we will have different tap stats available in Web Console for 2.0.2. Please c
Hide
Permalink
Jin Lim added a comment -
NS SRV team please confirm whether planning to have a new stat in the UI for the tap stats or planning to just update existing stats (like remaining on disks or backfill remaining, etc). Thanks.
Show
Jin Lim added a comment - NS SRV team please confirm whether planning to have a new stat in the UI for the tap stats or planning to just update existing stats (like remaining on disks or backfill remaining, etc). Thanks.
Hide
Permalink
Mike Wiederhold added a comment -
There will be no new stats in the UI Console. This was opened for the extra stats needed by ns_server for rebalance progress. There is still one more item for this ticket that is currently assigned to Chiyoung.
Show
Mike Wiederhold added a comment - There will be no new stats in the UI Console. This was opened for the extra stats needed by ns_server for rebalance progress. There is still one more item for this ticket that is currently assigned to Chiyoung.
Hide
Permalink
Karen Zeller added a comment -
Ok, removing as RN candidate.
Show
Karen Zeller added a comment - Ok, removing as RN candidate.
Hide
Permalink
Thuan Nguyen added a comment -
Integrated in github-ep-engine-2-0 #485 (See [http://qa.hq.northscale.net/job/github-ep-engine-2-0/485/])
    MB-7435: Add tap producer stats that shows num of sent items per vb (Revision e6ef69601e50ac026724bb60bc3a4d59c7e73bd5)
MB-7435: Add a command to get estimates of keys to move (Revision ad6475b0433fedc912753f36c6c7c0efeca2a14e)
MB-7435: Add tap-vbtakeover to cbstats (Revision 92df043ce6a02bbd75f3627093309d2826144979)
MB-7435: Fix bug related to tap cursor deregistration during backfill (Revision 9bcf346e37bcb4c592e333cede1d6ea9d7420360)
MB-7435: Add more stats into tap-vbtakeover command (Revision c21211cbb65d3136ff799c79303c2f77ad29a5e6)

     Result = SUCCESS
Mike Wiederhold :
Files :
* src/tapconnection.hh
* tests/ep_testsuite.cc
* src/tapconnection.cc

Mike Wiederhold :
Files :
* src/checkpoint.hh
* tests/ep_testsuite.cc
* src/checkpoint.cc
* src/tapconnmap.cc
* src/kvstore.hh
* src/couch-kvstore/couch-kvstore.cc
* tests/ep_test_apis.cc
* src/ep_engine.h
* tests/ep_test_apis.h
* src/tapconnmap.hh
* src/ep_engine.cc
* src/couch-kvstore/couch-kvstore.hh

Mike Wiederhold :
Files :
* management/cbstats

Mike Wiederhold :
Files :
* src/ep_engine.cc

Mike Wiederhold :
Files :
* src/ep_engine.cc
Show
Thuan Nguyen added a comment - Integrated in github-ep-engine-2-0 #485 (See [ http://qa.hq.northscale.net/job/github-ep-engine-2-0/485/ ])      MB-7435 : Add tap producer stats that shows num of sent items per vb (Revision e6ef69601e50ac026724bb60bc3a4d59c7e73bd5) MB-7435 : Add a command to get estimates of keys to move (Revision ad6475b0433fedc912753f36c6c7c0efeca2a14e) MB-7435 : Add tap-vbtakeover to cbstats (Revision 92df043ce6a02bbd75f3627093309d2826144979) MB-7435 : Fix bug related to tap cursor deregistration during backfill (Revision 9bcf346e37bcb4c592e333cede1d6ea9d7420360) MB-7435 : Add more stats into tap-vbtakeover command (Revision c21211cbb65d3136ff799c79303c2f77ad29a5e6)      Result = SUCCESS Mike Wiederhold : Files : * src/tapconnection.hh * tests/ep_testsuite.cc * src/tapconnection.cc Mike Wiederhold : Files : * src/checkpoint.hh * tests/ep_testsuite.cc * src/checkpoint.cc * src/tapconnmap.cc * src/kvstore.hh * src/couch-kvstore/couch-kvstore.cc * tests/ep_test_apis.cc * src/ep_engine.h * tests/ep_test_apis.h * src/tapconnmap.hh * src/ep_engine.cc * src/couch-kvstore/couch-kvstore.hh Mike Wiederhold : Files : * management/cbstats Mike Wiederhold : Files : * src/ep_engine.cc Mike Wiederhold : Files : * src/ep_engine.cc
Hide
Permalink
Maria McDuff added a comment -
assigning to anil to take a look, if it can be deferred.
Show
Maria McDuff added a comment - assigning to anil to take a look, if it can be deferred.
Hide
Permalink
Maria McDuff added a comment -
Deep,

this issue needs to be fixed if your rebalance api testing yields inaccurate result (see comment i made, 4/19).
pls update this bug with your test progress on CBD-184 (MB-8013).
We will wait for your update to decide if this bug needs to be fixed for 2.0.2.
Thanks,
maria
Show
Maria McDuff added a comment - Deep, this issue needs to be fixed if your rebalance api testing yields inaccurate result (see comment i made, 4/19). pls update this bug with your test progress on CBD-184 ( MB-8013 ). We will wait for your update to decide if this bug needs to be fixed for 2.0.2. Thanks, maria
Hide
Permalink
Deepkaran Salooja added a comment -
Assigning this to Iryna as she is working on the rebalance progress testing.

Iryna,

Please take a note of the information in this bug and update it with your findings about the accuracy of the rebalance progress stats.
Show
Deepkaran Salooja added a comment - Assigning this to Iryna as she is working on the rebalance progress testing. Iryna, Please take a note of the information in this bug and update it with your findings about the accuracy of the rebalance progress stats.
Hide
Permalink
Mike Wiederhold added a comment -
Iryna,

If you find any issues related to rebalance please file new issues and assign them to me.
Show
Mike Wiederhold added a comment - Iryna, If you find any issues related to rebalance please file new issues and assign them to me.
Hide
Permalink
Maria McDuff added a comment -
iryna,

you can go ahead and close this bug and file new ones for new issues.
Thanks.
Show
Maria McDuff added a comment - iryna, you can go ahead and close this bug and file new ones for new issues. Thanks.
Hide
Permalink
Iryna Mironava added a comment -
build 2.0.2-786 verified
Show
Iryna Mironava added a comment - build 2.0.2-786 verified

People

  • Assignee:
    Iryna Mironava
    Reporter:
    Aliaksey Artamonau
Vote (0)
Watch (10)

Dates

  • Created:
    Updated:
    Resolved: