Couchbase Server

Rebalance exited with reason {mover_failed,{abnormal,1}}

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: 1.6.5.2
  • Fix Version/s: 1.6.5.3
  • Component/s: couchbase-bucket
  • Security Level: Public
  • Labels:
    None
  • Environment:
    Ubuntu 10.04 64 bit

Description

Install ns 1.6.5.2r.15 on node 52
Load 6M keys with 2K/key
Run a load on node 52
Install ns 1.6.5.2r-15 on node 53, 55 and 56.
Add node 53 to node 52.
Rebalance
Add node 55 and 56 to cluster (node 52 and 53)
Rebalance. Failed
Rebalance exited with reason {mover_failed,{abnormal,1}}
  1. GZip Archive
    ns-diag-rebl-failed-node52.txt.gz
    09/Mar/11 4:19 PM
    1.64 MB
    Thuan Nguyen
  2. GZip Archive
    ns-diag-rebl-failed-node53.txt.gz
    09/Mar/11 4:19 PM
    1.48 MB
    Thuan Nguyen
  3. GZip Archive
    ns-diag-rebl-failed-node55.txt.gz
    09/Mar/11 4:19 PM
    268 kB
    Thuan Nguyen
  4. GZip Archive
    ns-diag-rebl-failed-node56.txt.gz
    09/Mar/11 4:19 PM
    275 kB
    Thuan Nguyen

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Matt Ingenthron added a comment -
Note that this seems to exhibit some of the same errors as seen as MB-3530

Rebalance exited with reason wait_for_memcached_failed
ns_orchestrator002 18:30:17 - Tue Mar 22, 2011
Starting rebalance, KeepNodes = ['ns_1@10.2.1.52','ns_1@10.2.1.53',
'ns_1@10.2.1.55'], EjectNodes = []
(repeated 2 times) ns_orchestrator004 18:30:14 - Tue Mar 22, 2011
Rebalance exited with reason wait_for_memcached_failed
(repeated 1 times) ns_orchestrator002 18:30:14 - Tue Mar 22, 2011
Rebalance exited with reason wait_for_memcached_failed
ns_orchestrator002 18:24:54 - Tue Mar 22, 2011
Starting rebalance, KeepNodes = ['ns_1@10.2.1.52','ns_1@10.2.1.53',
'ns_1@10.2.1.55'], EjectNodes = []
ns_orchestrator004 18:24:43 - Tue Mar 22, 2011
Control connection to memcached on 'ns_1@10.2.1.55' disconnected: {{badmatch,
{error,
timeout}},
[{mc_client_binary,
stats_recv,
4},
{mc_client_binary,
stats,4},
{ns_memcached,
handle_call,
3},
{gen_server,
handle_msg,
5},
{proc_lib,
init_p_do_apply,
3}]} (repeated 3 times) ns_memcached004 18:22:47 - Tue Mar 22, 2011
Bucket "default" loaded on node 'ns_1@10.2.1.55' in 0 seconds. (repeated 2 times) ns_memcached001 18:22:47 - Tue Mar 22, 2011
Bucket "default" loaded on node 'ns_1@10.2.1.55' in 0 seconds. ns_memcached001 18:17:06 - Tue Mar 22, 2011
Control connection to memcached on 'ns_1@10.2.1.55' disconnected: {{badmatch,
{error,
timeout}},
[{mc_client_binary,
stats_recv,
4},
{mc_client_binary,
stats,4},
{ns_memcached,
handle_call,
3},
{gen_server,
handle_msg,
5},
{proc_lib,
init_p_do_apply,
3}]} ns_memcached004 18:17:05 - Tue Mar 22, 2011
Rebalance exited with reason {mover_failed,{abnormal,1}}
ns_orchestrator002 18:16:47 - Tue Mar 22, 2011
Port server vbucketmigrator on node 'ns_1@10.2.1.55' exited with status 1. Restarting. Messages: Authenticating towards: {Sock 10.2.1.55:11210}
Authenticated towards: {Sock 10.2.1.55:11210}
Timed out on BinaryMessagePipe from 10.2.1.53:11210 on fd 7 ns_port_server000 18:16:47 - Tue Mar 22, 2011
Port server vbucketmigrator on node 'ns_1@10.2.1.55' exited with status 1. Restarting. Messages: Authenticating towards: {Sock 10.2.1.55:11210}
Authenticated towards: {Sock 10.2.1.55:11210}
Timed out on BinaryMessagePipe from 10.2.1.53:11210 on fd 7 ns_port_server000 18:11:45 - Tue Mar 22, 2011
Port server vbucketmigrator on node 'ns_1@10.2.1.55' exited with status 1. Restarting. Messages: Authenticated towards: {Sock 10.2.1.55:11210}
Starting to move bucket 7
Timed out on BinaryMessagePipe from 10.2.1.53:11210 on fd 7 ns_port_server000 18:06:44 - Tue Mar 22, 2011
Show
Matt Ingenthron added a comment - Note that this seems to exhibit some of the same errors as seen as MB-3530 Rebalance exited with reason wait_for_memcached_failed ns_orchestrator002 18:30:17 - Tue Mar 22, 2011 Starting rebalance, KeepNodes = [' ns_1@10.2.1.52 ',' ns_1@10.2.1.53 ', ' ns_1@10.2.1.55 '], EjectNodes = [] (repeated 2 times) ns_orchestrator004 18:30:14 - Tue Mar 22, 2011 Rebalance exited with reason wait_for_memcached_failed (repeated 1 times) ns_orchestrator002 18:30:14 - Tue Mar 22, 2011 Rebalance exited with reason wait_for_memcached_failed ns_orchestrator002 18:24:54 - Tue Mar 22, 2011 Starting rebalance, KeepNodes = [' ns_1@10.2.1.52 ',' ns_1@10.2.1.53 ', ' ns_1@10.2.1.55 '], EjectNodes = [] ns_orchestrator004 18:24:43 - Tue Mar 22, 2011 Control connection to memcached on ' ns_1@10.2.1.55 ' disconnected: {{badmatch, {error, timeout}}, [{mc_client_binary, stats_recv, 4}, {mc_client_binary, stats,4}, {ns_memcached, handle_call, 3}, {gen_server, handle_msg, 5}, {proc_lib, init_p_do_apply, 3}]} (repeated 3 times) ns_memcached004 18:22:47 - Tue Mar 22, 2011 Bucket "default" loaded on node ' ns_1@10.2.1.55 ' in 0 seconds. (repeated 2 times) ns_memcached001 18:22:47 - Tue Mar 22, 2011 Bucket "default" loaded on node ' ns_1@10.2.1.55 ' in 0 seconds. ns_memcached001 18:17:06 - Tue Mar 22, 2011 Control connection to memcached on ' ns_1@10.2.1.55 ' disconnected: {{badmatch, {error, timeout}}, [{mc_client_binary, stats_recv, 4}, {mc_client_binary, stats,4}, {ns_memcached, handle_call, 3}, {gen_server, handle_msg, 5}, {proc_lib, init_p_do_apply, 3}]} ns_memcached004 18:17:05 - Tue Mar 22, 2011 Rebalance exited with reason {mover_failed,{abnormal,1}} ns_orchestrator002 18:16:47 - Tue Mar 22, 2011 Port server vbucketmigrator on node ' ns_1@10.2.1.55 ' exited with status 1. Restarting. Messages: Authenticating towards: {Sock 10.2.1.55:11210} Authenticated towards: {Sock 10.2.1.55:11210} Timed out on BinaryMessagePipe from 10.2.1.53:11210 on fd 7 ns_port_server000 18:16:47 - Tue Mar 22, 2011 Port server vbucketmigrator on node ' ns_1@10.2.1.55 ' exited with status 1. Restarting. Messages: Authenticating towards: {Sock 10.2.1.55:11210} Authenticated towards: {Sock 10.2.1.55:11210} Timed out on BinaryMessagePipe from 10.2.1.53:11210 on fd 7 ns_port_server000 18:11:45 - Tue Mar 22, 2011 Port server vbucketmigrator on node ' ns_1@10.2.1.55 ' exited with status 1. Restarting. Messages: Authenticated towards: {Sock 10.2.1.55:11210} Starting to move bucket 7 Timed out on BinaryMessagePipe from 10.2.1.53:11210 on fd 7 ns_port_server000 18:06:44 - Tue Mar 22, 2011
Hide
Permalink
Matt Ingenthron added a comment -
Note that in the bug on MB-3530 (which is against 1.6.5.2.1), the other log entries are similar. Specifically this one pulled from the cluster under test in MB-3530.

Control connection to memcached on 'ns_1@10.1.5.122' disconnected: {{badmatch,
{error,
timeout}},
[{mc_client_binary,
stats_recv,
4},
{mc_client_binary,
stats,4},
{ns_memcached,
handle_call,
3},
{gen_server,
handle_msg,
5},
{proc_lib,
init_p_do_apply,
3}]} (repeated 1 times)
Show
Matt Ingenthron added a comment - Note that in the bug on MB-3530 (which is against 1.6.5.2.1), the other log entries are similar. Specifically this one pulled from the cluster under test in MB-3530 . Control connection to memcached on ' ns_1@10.1.5.122 ' disconnected: {{badmatch, {error, timeout}}, [{mc_client_binary, stats_recv, 4}, {mc_client_binary, stats,4}, {ns_memcached, handle_call, 3}, {gen_server, handle_msg, 5}, {proc_lib, init_p_do_apply, 3}]} (repeated 1 times)
Hide
Permalink
Sharon Barr added a comment -
http://review.membase.org/#change,5070
Show
Sharon Barr added a comment - http://review.membase.org/#change,5070

People

  • Assignee:
    Trond Norbye
    Reporter:
    Thuan Nguyen
Vote (0)
Watch (2)

Dates

  • Created:
    Updated:
    Resolved: