Couchbase Server

Rebalance fails with wait_for_memcached_failed (new nodes)

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Blocker Blocker
  • Resolution: Fixed
  • Affects Version/s: None
  • Fix Version/s: 2011-01-20, 1.6.5
  • Component/s: None
  • Security Level: Public
  • Labels:
    None

Description

With Perry's RightScale tests, he's getting this error when adding new nodes with no data. Log shows buckets taking 3 seconds to come up, which means 5 seconds is probably still too close.

I'll double the timeout to 10 seconds, since it used to be infinite.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Sean Lynch added a comment -
Fixed in http://review.membase.org/4234
Show
Sean Lynch added a comment - Fixed in http://review.membase.org/4234
Hide
Permalink
Farshid Ghods added a comment -
I am seeing the same behavior in the latest build : Version: 1.6.5.3r_6_g892cecc

to repro :
1- remove all the servers
2- delete all the buckets
3- create a default bucket
4- add a new node
5- rebalance

i see the wait_for_memcached_failed (new nodes)


logs from the ui :

Rebalance exited with reason wait_for_memcached_failed
ns_orchestrator002 16:25:56 - Wed Mar 23, 2011
Starting rebalance, KeepNodes = ['ns_1@10.1.5.177','ns_1@10.1.5.180'], EjectNodes = []
ns_orchestrator004 16:25:45 - Wed Mar 23, 2011
Node 'ns_1@10.1.5.177' saw that node 'ns_1@10.1.5.180' came up. ns_node_disco004 16:25:43 - Wed Mar 23, 2011
Created bucket "default" of type: membase
menelaus_web012 16:25:29 - Wed Mar 23, 2011
Deleted bucket "default"
menelaus_web011 16:25:18
Show
Farshid Ghods added a comment - I am seeing the same behavior in the latest build : Version: 1.6.5.3r_6_g892cecc to repro : 1- remove all the servers 2- delete all the buckets 3- create a default bucket 4- add a new node 5- rebalance i see the wait_for_memcached_failed (new nodes) logs from the ui : Rebalance exited with reason wait_for_memcached_failed ns_orchestrator002 16:25:56 - Wed Mar 23, 2011 Starting rebalance, KeepNodes = [' ns_1@10.1.5.177 ',' ns_1@10.1.5.180 '], EjectNodes = [] ns_orchestrator004 16:25:45 - Wed Mar 23, 2011 Node ' ns_1@10.1.5.177 ' saw that node ' ns_1@10.1.5.180 ' came up. ns_node_disco004 16:25:43 - Wed Mar 23, 2011 Created bucket "default" of type: membase menelaus_web012 16:25:29 - Wed Mar 23, 2011 Deleted bucket "default" menelaus_web011 16:25:18
Hide
Permalink
Farshid Ghods added a comment - - edited
I am able to repro this in the latest 1.7.0 build

reopening the bug.
Show
Farshid Ghods added a comment - - edited I am able to repro this in the latest 1.7.0 build reopening the bug.
Hide
Permalink
Aleksey Kondratenko added a comment -
Attaching diag from new node could help a lot in this case. Without logs its really hard to tell what's causing timeout
Show
Aleksey Kondratenko added a comment - Attaching diag from new node could help a lot in this case. Without logs its really hard to tell what's causing timeout

People

  • Assignee:
    Aleksey Kondratenko
    Reporter:
    Sean Lynch
Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: