Couchbase Server

Rebalance failed due to wait_for_memcached on 181

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Blocker Blocker
  • Resolution: Fixed
  • Affects Version/s: 1.8.1-release-candidate
  • Fix Version/s: 1.8.1
  • Component/s: ns_server
  • Security Level: Public
  • Labels:
    None
  • Environment:
    CentOS
    1.8.1r-756

Description

swaprebalance.SwapRebalabanceStartStopTests.do_test,load-ratio:5,num-swap:1,replica:1,num-buckets:1

Wait_for_memcached failed on node 76

[ns_server:debug] [2012-04-12 17:12:51] [ns_1@10.1.3.74:ns_config_rep:ns_config_rep:handle_info:201] Pushing config
[ns_server:debug] [2012-04-12 17:12:51] [ns_1@10.1.3.74:ns_config_rep:ns_config_rep:handle_info:203] Pushing config done
[rebalance:info] [2012-04-12 17:12:52] [ns_1@10.1.3.74:<0.32703.3>:ns_rebalancer:wait_for_memcached:283] Waiting for ['ns_1@10.1.3.76']
[rebalance:info] [2012-04-12 17:12:53] [ns_1@10.1.3.74:<0.32703.3>:ns_rebalancer:wait_for_memcached:283] Waiting for ['ns_1@10.1.3.76']
[rebalance:info] [2012-04-12 17:12:54] [ns_1@10.1.3.74:<0.32703.3>:ns_rebalancer:wait_for_memcached:283] Waiting for ['ns_1@10.1.3.76']
[rebalance:info] [2012-04-12 17:12:55] [ns_1@10.1.3.74:<0.32703.3>:ns_rebalancer:wait_for_memcached:283] Waiting for ['ns_1@10.1.3.76']
[ns_server:info] [2012-04-12 17:12:55] [ns_1@10.1.3.74:<0.25182.3>:ns_orchestrator:handle_info:208] Skipping janitor in state rebalancing: {rebalancing_state,<0.32703.3>,
                                        {dict,4,16,16,8,80,48,
                                         {[],[],[],[],[],[],[],[],[],[],[],[],
                                          [],[],[],[]},
                                         {{[],[],[],[],
                                           [['ns_1@10.1.3.74'|0.0]],
                                           [['ns_1@10.1.3.75'|0.0]],
                                           [['ns_1@10.1.3.76'|0.0]],
                                           [['ns_1@10.1.3.77'|0.0]],
                                           [],[],[],[],[],[],[],[]}}}}
[rebalance:info] [2012-04-12 17:12:56] [ns_1@10.1.3.74:<0.32703.3>:ns_rebalancer:wait_for_memcached:283] Waiting for ['ns_1@10.1.3.76']
[rebalance:info] [2012-04-12 17:12:57] [ns_1@10.1.3.74:<0.32703.3>:ns_rebalancer:wait_for_memcached:283] Waiting for ['ns_1@10.1.3.76']
[rebalance:info] [2012-04-12 17:12:58] [ns_1@10.1.3.74:<0.32703.3>:ns_rebalancer:wait_for_memcached:283] Waiting for ['ns_1@10.1.3.76']
[rebalance:info] [2012-04-12 17:12:59] [ns_1@10.1.3.74:<0.32703.3>:ns_rebalancer:wait_for_memcached:283] Waiting for ['ns_1@10.1.3.76']
[rebalance:info] [2012-04-12 17:13:00] [ns_1@10.1.3.74:<0.32703.3>:ns_rebalancer:wait_for_memcached:283] Waiting for ['ns_1@10.1.3.76']
[rebalance:info] [2012-04-12 17:13:01] [ns_1@10.1.3.74:<0.32703.3>:ns_rebalancer:wait_for_memcached:283] Waiting for ['ns_1@10.1.3.76']
[rebalance:info] [2012-04-12 17:13:02] [ns_1@10.1.3.74:<0.32703.3>:ns_rebalancer:wait_for_memcached:283] Waiting for ['ns_1@10.1.3.76']
[user:info] [2012-04-12 17:13:02] [ns_1@10.1.3.74:<0.25182.3>:ns_orchestrator:handle_info:242] Rebalance exited with reason {wait_for_memcached_failed,"default",
                                 ['ns_1@10.1.3.76']}


Logs are present on MB-5060
  1. GZip Archive
    10.1.3.114-8091-diag.txt.gz
    19/Apr/12 3:30 PM
    338 kB
    Karan Kumar
  2. GZip Archive
    10.1.3.115-8091-diag.txt.gz
    19/Apr/12 3:30 PM
    281 kB
    Karan Kumar
  3. GZip Archive
    10.1.3.116-8091-diag.txt.gz
    19/Apr/12 3:30 PM
    284 kB
    Karan Kumar
  4. GZip Archive
    10.1.3.117-8091-diag.txt.gz
    19/Apr/12 3:30 PM
    250 kB
    Karan Kumar
  5. GZip Archive
    10.1.3.118-8091-diag.txt.gz
    19/Apr/12 3:30 PM
    242 kB
    Karan Kumar

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Karan Kumar added a comment -
This could be dupe of
http://www.couchbase.com/issues/browse/MB-5031
Show
Karan Kumar added a comment - This could be dupe of http://www.couchbase.com/issues/browse/MB-5031
Hide
Permalink
Karan Kumar added a comment -
Assigning to Chiyoung, for initial investigation.
Show
Karan Kumar added a comment - Assigning to Chiyoung, for initial investigation.
Hide
Permalink
Aleksey Kondratenko added a comment -
Real issue. But seemingly quite old. testsuite is trying to rebalance out node that's not part of cluster. And that's causing rebalance to fails instead of refusing it right away. Will fix.
Show
Aleksey Kondratenko added a comment - Real issue. But seemingly quite old. testsuite is trying to rebalance out node that's not part of cluster. And that's causing rebalance to fails instead of refusing it right away. Will fix.
Hide
Permalink
Aleksey Kondratenko added a comment -
Another interesting question is why were seeing this not ready vbucket at all. We shouldn't.
Show
Aleksey Kondratenko added a comment - Another interesting question is why were seeing this not ready vbucket at all. We shouldn't.
Hide
Permalink
Aliaksey Artamonau added a comment -
http://review.couchbase.org/14969
Show
Aliaksey Artamonau added a comment - http://review.couchbase.org/14969
Hide
Permalink
Aleksey Kondratenko added a comment -
Ignore my previous commit. It was about mb-5052. This one is fixed by http://review.couchbase.org/14969 which is now merged
Show
Aleksey Kondratenko added a comment - Ignore my previous commit. It was about mb-5052. This one is fixed by http://review.couchbase.org/14969 which is now merged
Hide
Permalink
Aleksey Kondratenko added a comment -
done
Show
Aleksey Kondratenko added a comment - done
Hide
Permalink
Karan Kumar added a comment -
Reopening this seeing the issue on the latest build
1.8.1.r-768
Show
Karan Kumar added a comment - Reopening this seeing the issue on the latest build 1.8.1.r-768
Hide
Permalink
Aleksey Kondratenko added a comment -
Thanks for reporting. This case seems to be different though.
Show
Aleksey Kondratenko added a comment - Thanks for reporting. This case seems to be different though.
Hide
Permalink
Aleksey Kondratenko added a comment -
this happens because of 2 things:

* node can be added back after failover prior to all buckets completed shutdown

* we're doing non-forced shutdown on failover (recent regression)
Show
Aleksey Kondratenko added a comment - this happens because of 2 things: * node can be added back after failover prior to all buckets completed shutdown * we're doing non-forced shutdown on failover (recent regression)
Hide
Permalink
Karan Kumar added a comment -
So the test is trying to failover the server 117.
Starting rebalance, KeepNodes = ['ns_1@10.1.3.114','ns_1@10.1.3.115',
'ns_1@10.1.3.116'], EjectNodes = ['ns_1@10.1.3.117']

Show
Karan Kumar added a comment - So the test is trying to failover the server 117. Starting rebalance, KeepNodes = [' ns_1@10.1.3.114 ',' ns_1@10.1.3.115 ', ' ns_1@10.1.3.116 '], EjectNodes = [' ns_1@10.1.3.117 ']
Hide
Permalink
Aleksey Kondratenko added a comment -
So recent reopening is exactly MB-5118. I'll close this and will fix MB-5118
Show
Aleksey Kondratenko added a comment - So recent reopening is exactly MB-5118 . I'll close this and will fix MB-5118
Hide
Permalink
Aleksey Kondratenko added a comment -
Done.
Show
Aleksey Kondratenko added a comment - Done.
Hide
Permalink
Thuan Nguyen added a comment -
Integrated in github-ns-server-2-0 #333 (See [http://qa.hq.northscale.net/job/github-ns-server-2-0/333/])
    don't allow attempts to "rebalance out" unknown nodes.MB-5062 (Revision bc2ca5f358f318387c5070ab8b192a837fe31c8e)

     Result = SUCCESS
Aliaksey Kandratsenka :
Files :
* src/ns_cluster_membership.erl
Show
Thuan Nguyen added a comment - Integrated in github-ns-server-2-0 #333 (See [ http://qa.hq.northscale.net/job/github-ns-server-2-0/333/ ])     don't allow attempts to "rebalance out" unknown nodes. MB-5062 (Revision bc2ca5f358f318387c5070ab8b192a837fe31c8e)      Result = SUCCESS Aliaksey Kandratsenka : Files : * src/ns_cluster_membership.erl

People

  • Assignee:
    Aleksey Kondratenko
    Reporter:
    Karan Kumar
Vote (0)
Watch (2)

Dates

  • Created:
    Updated:
    Resolved: