Couchbase Server

Rebalance stuck, after adding back a failed over node, with XDCR enabled

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Critical Critical
  • Resolution: Fixed
  • Affects Version/s: 2.0
  • Fix Version/s: 2.0.1
  • Component/s: bucket-engine, cross-datacenter-replication
  • Security Level: Public
  • Labels:
    None
  • Environment:
    2.0.0 Build 1416
    Unidirectional replication
    1024 vBuckets

Description

Started over with a pair of single node clusters (10.1.3.71 and 10.1.3.72), threads=4, expiration=10
-- Added a node (10.1.3.235) on to the source cluster, rebalance.
- Fall in destination ops-per-sec towards the end of rebalance on the source`s side.
- But picks up once rebalance actually finishes.
-- Failover a node on source cluster
- Couldn`t add it back and rebalance for some reason, so removed it completely, rebalanced and then added another node again, but rebalance is now stuck (between 10.1.3.71 and 10.1.3.235).
  1. GZip Archive
    10.1.3.235-8091-diag.txt.gz
    13/Jul/12 5:29 PM
    10.10 MB
    Abhinav Dangeti
  2. GZip Archive
    10.1.3.71-8091-diag.txt.gz
    13/Jul/12 5:29 PM
    9.93 MB
    Abhinav Dangeti
  3. GZip Archive
    10.1.3.72-8091-diag.txt.gz
    13/Jul/12 5:29 PM
    14.56 MB
    Abhinav Dangeti
  1. Screen Shot 2012-07-13 at 3.28.51 PM.png
    111 kB
    13/Jul/12 5:29 PM

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Peter Wansch added a comment -
Can you take a first look at this?
Show
Peter Wansch added a comment - Can you take a first look at this?
Hide
Permalink
Aleksey Kondratenko added a comment -
Rebalance has in fact failed due to memcached timeout.

But post-rebalance cleanup is stuck. Found this to be caused by memcached completely ignoring our request(s) to deregister replica building tap names.

There's also badness in ns_server where there's no timeouts on those requests. I've fixed it in my tree.
Show
Aleksey Kondratenko added a comment - Rebalance has in fact failed due to memcached timeout. But post-rebalance cleanup is stuck. Found this to be caused by memcached completely ignoring our request(s) to deregister replica building tap names. There's also badness in ns_server where there's no timeouts on those requests. I've fixed it in my tree.
Hide
Permalink
Aleksey Kondratenko added a comment -
Not sure if it helps you as it's likely duplicate.

Or not. Note that in this case there's perhaps as storm of get-meta requests, so could be some sort of serious starvation problem instead.
Show
Aleksey Kondratenko added a comment - Not sure if it helps you as it's likely duplicate. Or not. Note that in this case there's perhaps as storm of get-meta requests, so could be some sort of serious starvation problem instead.
Hide
Permalink
Thuan Nguyen added a comment -
Integrated in github-ns-server-2-0 #412 (See [http://qa.hq.northscale.net/job/github-ns-server-2-0/412/])
    MB-5906: have finite timeouts for tap names unregistering (Revision b4a28e7e41bc70eaaca711a17ff7947cb7d90fd4)

     Result = SUCCESS
Aliaksey Artamonau :
Files :
* src/ns_replicas_builder.erl
Show
Thuan Nguyen added a comment - Integrated in github-ns-server-2-0 #412 (See [ http://qa.hq.northscale.net/job/github-ns-server-2-0/412/ ])      MB-5906 : have finite timeouts for tap names unregistering (Revision b4a28e7e41bc70eaaca711a17ff7947cb7d90fd4)      Result = SUCCESS Aliaksey Artamonau : Files : * src/ns_replicas_builder.erl
Hide
Permalink
Aleksey Kondratenko added a comment -
BTW, it's not clear at all why this ticket was closed. Have we fixed underlying memcached/ep-engine issue? Apparently not, given that there's no commits that refer to this bug.
Show
Aleksey Kondratenko added a comment - BTW, it's not clear at all why this ticket was closed. Have we fixed underlying memcached/ep-engine issue? Apparently not, given that there's no commits that refer to this bug.
Hide
Permalink
Jin Lim added a comment - - edited
Hi Trond, it appears to be that you had marked this bug fixed last time. For better bug tracking, we would like to require bare minimum info about why any bug is being closed/resolved. Can you please elaborate more on why it is being closed? Not complete, cannot reproduce, fix review #s, etc? Afterwards please assign it back to QE so they can properly close this bug. Thanks much!
Show
Jin Lim added a comment - - edited Hi Trond, it appears to be that you had marked this bug fixed last time. For better bug tracking, we would like to require bare minimum info about why any bug is being closed/resolved. Can you please elaborate more on why it is being closed? Not complete, cannot reproduce, fix review #s, etc? Afterwards please assign it back to QE so they can properly close this bug. Thanks much!
Hide
Permalink
Trond Norbye added a comment -
It doesn't look to me ( http://www.couchbase.com/issues/browse/MB-5906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ) that I've changed the status of this bug..
Show
Trond Norbye added a comment - It doesn't look to me ( http://www.couchbase.com/issues/browse/MB-5906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ) that I've changed the status of this bug..
Hide
Permalink
Jin Lim added a comment -
My apology, it was meant to address to Tony (Thuan). Reroute this to Tony ;)
Show
Jin Lim added a comment - My apology, it was meant to address to Tony (Thuan). Reroute this to Tony ;)
Hide
Permalink
Jin Lim added a comment -
Hi Tony, it appears to be that you had marked this bug fixed last time. For better bug tracking, we would like to require bare minimum info about why any bug is being closed/resolved. Can you please elaborate more on why it is being closed? Not complete, cannot reproduce, fix review #s, etc? Afterwards please re-close this bug if appropriate. Thanks much!
Show
Jin Lim added a comment - Hi Tony, it appears to be that you had marked this bug fixed last time. For better bug tracking, we would like to require bare minimum info about why any bug is being closed/resolved. Can you please elaborate more on why it is being closed? Not complete, cannot reproduce, fix review #s, etc? Afterwards please re-close this bug if appropriate. Thanks much!
Hide
Permalink
Jin Lim added a comment -
Per bug scrubs, it appears to be there was commit for this issue here:
Integrated in github-ns-server-2-0 #412 (See [http://qa.hq.northscale.net/job/github-ns-server-2-0/412/])
    MB-5906: have finite timeouts for tap names unregistering (Revision b4a28e7e41bc70eaaca711a17ff7947cb7d90fd4)

Alk & Aliaksey can you please verify if this commit had addressed the issue or not? Thanks.
Show
Jin Lim added a comment - Per bug scrubs, it appears to be there was commit for this issue here: Integrated in github-ns-server-2-0 #412 (See [ http://qa.hq.northscale.net/job/github-ns-server-2-0/412/ ])      MB-5906 : have finite timeouts for tap names unregistering (Revision b4a28e7e41bc70eaaca711a17ff7947cb7d90fd4) Alk & Aliaksey can you please verify if this commit had addressed the issue or not? Thanks.
Hide
Permalink
Aleksey Kondratenko added a comment -
As pointed out above this commit was _workaround_ for misbehavior elsewhere.

So this particular symptom is gone (worked around by ns_server), but apparently underlying issue is not fixed.
Show
Aleksey Kondratenko added a comment - As pointed out above this commit was _workaround_ for misbehavior elsewhere. So this particular symptom is gone (worked around by ns_server), but apparently underlying issue is not fixed.
Hide
Permalink
Jin Lim added a comment -
Based Alk's comment above we will close the issue since the symptom is gone by the workaround. We will open a separate bug for probably 2.0.2 and continue investigate why/how memcached completely ignoring ns_server request(s) to deregister replica building tap names.
Show
Jin Lim added a comment - Based Alk's comment above we will close the issue since the symptom is gone by the workaround. We will open a separate bug for probably 2.0.2 and continue investigate why/how memcached completely ignoring ns_server request(s) to deregister replica building tap names.
Hide
Permalink
Jin Lim added a comment -
MB-7717 has been created to track the mecached timeout (ignoring ns_server requests) issue.
Show
Jin Lim added a comment - MB-7717 has been created to track the mecached timeout (ignoring ns_server requests) issue.

People

  • Assignee:
    Thuan Nguyen
    Reporter:
    Abhinav Dangeti
Vote (0)
Watch (3)

Dates

  • Created:
    Updated:
    Resolved: