Home | Forums | Couchbase | Couchbase Server 1.8.x

Servers Crashing, Can't Rebalance

5 replies [Last post]
Wed, 03/21/2012 - 04:05
ericbrown
Offline
Joined: 09/08/2011
Groups: None

Today I can't get the cluster rebalance to execute after a node crashed. The server it appears fell out of the cluster and then came back in. When I got the alerts I tried to rebalance to put it back in and that failed. I've tried to remove the node and rebalance but that fails as well. How can I recover the cluster?

Could not auto-failover more nodes ('ns_1@10.206.117.77'). Maximum number of nodes that will be automatically failovered (1) is reached.
auto_failover002 ns_1@10.100.243.14 07:00:48 - Wed Mar 21, 2012
Could not auto-failover more nodes ('ns_1@10.206.117.77'). Maximum number of nodes that will be automatically failovered (1) is reached.
(repeated 71 times) auto_failover002 ns_1@10.100.243.14 07:00:47 - Wed Mar 21, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 10:53:13: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 10:53:13: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 06:56:39 - Wed Mar 21, 2012
Could not auto-failover more nodes ('ns_1@10.206.117.77'). Maximum number of nodes that will be automatically failovered (1) is reached.
auto_failover002 ns_1@10.100.243.14 06:54:48 - Wed Mar 21, 2012
Could not auto-failover more nodes ('ns_1@10.206.117.77'). Maximum number of nodes that will be automatically failovered (1) is reached.
(repeated 71 times) auto_failover002 ns_1@10.100.243.14 06:54:47 - Wed Mar 21, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 10:42:26: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 10:42:26: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 06:53:13 - Wed Mar 21, 2012
Rebalance exited with reason {wait_for_memcached_failed,"default",
['ns_1@10.206.117.77']}
ns_orchestrator002 ns_1@10.100.243.14 06:49:58 - Wed Mar 21, 2012
Starting rebalance, KeepNodes = ['ns_1@10.40.177.223','ns_1@10.90.145.211',
'ns_1@10.98.183.173','ns_1@10.100.243.14'], EjectNodes = ['ns_1@10.206.117.77']
ns_orchestrator004 ns_1@10.100.243.14 06:49:47 - Wed Mar 21, 2012
Rebalance exited with reason {wait_for_memcached_failed,"default",
['ns_1@10.206.117.77']}
(repeated 1 times) ns_orchestrator002 ns_1@10.100.243.14 06:49:47 - Wed Mar 21, 2012
Could not auto-failover more nodes ('ns_1@10.206.117.77'). Maximum number of nodes that will be automatically failovered (1) is reached.
auto_failover002 ns_1@10.100.243.14 06:48:48 - Wed Mar 21, 2012
Starting rebalance, KeepNodes = ['ns_1@10.40.177.223','ns_1@10.90.145.211',
'ns_1@10.98.183.173','ns_1@10.100.243.14',
'ns_1@10.206.117.77'], EjectNodes = []
(repeated 1 times) ns_orchestrator004 ns_1@10.100.243.14 06:48:47 - Wed Mar 21, 2012
Could not auto-failover more nodes ('ns_1@10.206.117.77'). Maximum number of nodes that will be automatically failovered (1) is reached.
(repeated 68 times) auto_failover002 ns_1@10.100.243.14 06:48:47 - Wed Mar 21, 2012
Rebalance exited with reason {wait_for_memcached_failed,"default",
['ns_1@10.206.117.77']}
ns_orchestrator002 ns_1@10.100.243.14 06:43:56 - Wed Mar 21, 2012
Starting rebalance, KeepNodes = ['ns_1@10.40.177.223','ns_1@10.90.145.211',
'ns_1@10.98.183.173','ns_1@10.100.243.14',
'ns_1@10.206.117.77'], EjectNodes = []
ns_orchestrator004 ns_1@10.100.243.14 06:43:45 - Wed Mar 21, 2012
Could not auto-failover more nodes ('ns_1@10.206.117.77'). Maximum number of nodes that will be automatically failovered (1) is reached.
auto_failover002 ns_1@10.100.243.14 06:43:03 - Wed Mar 21, 2012
Couchbase Server has started on web port 8091 on node 'ns_1@10.206.117.77'. menelaus_sup001 ns_1@10.206.117.77 06:42:26 - Wed Mar 21, 2012
Node 'ns_1@10.98.183.173' saw that node 'ns_1@10.206.117.77' came up. ns_node_disco004 ns_1@10.98.183.173 06:42:26 - Wed Mar 21, 2012
Node 'ns_1@10.90.145.211' saw that node 'ns_1@10.206.117.77' came up. ns_node_disco004 ns_1@10.90.145.211 06:42:26 - Wed Mar 21, 2012
Node 'ns_1@10.40.177.223' saw that node 'ns_1@10.206.117.77' came up. ns_node_disco004 ns_1@10.40.177.223 06:42:25 - Wed Mar 21, 2012
Node 'ns_1@10.100.243.14' saw that node 'ns_1@10.206.117.77' came up. ns_node_disco004 ns_1@10.100.243.14 06:42:25 - Wed Mar 21, 2012
Node 'ns_1@10.206.117.77' synchronized otp cookie qouhqbcugdhzjctj from cluster ns_node_disco002 ns_1@10.206.117.77 06:42:25 - Wed Mar 21, 2012
Node 'ns_1@10.90.145.211' saw that node 'ns_1@10.206.117.77' went down. ns_node_disco005 ns_1@10.90.145.211 06:42:23 - Wed Mar 21, 2012
Node 'ns_1@10.100.243.14' saw that node 'ns_1@10.206.117.77' went down. ns_node_disco005 ns_1@10.100.243.14 06:42:23 - Wed Mar 21, 2012
Node 'ns_1@10.40.177.223' saw that node 'ns_1@10.206.117.77' went down. ns_node_disco005 ns_1@10.40.177.223 06:42:22 - Wed Mar 21, 2012
Node 'ns_1@10.98.183.173' saw that node 'ns_1@10.206.117.77' went down. ns_node_disco005 ns_1@10.98.183.173 06:42:22 - Wed Mar 21, 2012
Could not auto-failover more nodes ('ns_1@10.206.117.77'). Maximum number of nodes that will be automatically failovered (1) is reached.
auto_failover002 ns_1@10.206.117.77 06:40:27 - Wed Mar 21, 2012
Could not auto-failover more nodes ('ns_1@10.206.117.77'). Maximum number of nodes that will be automatically failovered (1) is reached.
(repeated 71 times) auto_failover002 ns_1@10.206.117.77 06:40:23 - Wed Mar 21, 2012
Rebalance exited with reason {wait_for_memcached_failed,"default",
['ns_1@10.206.117.77']}
ns_orchestrator002 ns_1@10.206.117.77 06:36:38 - Wed Mar 21, 2012
Starting rebalance, KeepNodes = ['ns_1@10.40.177.223','ns_1@10.90.145.211',
'ns_1@10.98.183.173','ns_1@10.100.243.14'], EjectNodes = ['ns_1@10.206.117.77']
ns_orchestrator004 ns_1@10.206.117.77 06:36:27 - Wed Mar 21, 2012
Could not auto-failover more nodes ('ns_1@10.206.117.77'). Maximum number of nodes that will be automatically failovered (1) is reached.
auto_failover002 ns_1@10.206.117.77 06:34:27 - Wed Mar 21, 2012
Could not auto-failover more nodes ('ns_1@10.206.117.77'). Maximum number of nodes that will be automatically failovered (1) is reached.
(repeated 71 times) auto_failover002 ns_1@10.206.117.77 06:34:23 - Wed Mar 21, 2012
Could not auto-failover more nodes ('ns_1@10.206.117.77'). Maximum number of nodes that will be automatically failovered (1) is reached.
auto_failover002 ns_1@10.206.117.77 06:28:27 - Wed Mar 21, 2012
Could not auto-failover more nodes ('ns_1@10.206.117.77'). Maximum number of nodes that will be automatically failovered (1) is reached.
(repeated 71 times) auto_failover002 ns_1@10.206.117.77 06:28:23 - Wed Mar 21, 2012
Could not auto-failover more nodes ('ns_1@10.206.117.77'). Maximum number of nodes that will be automatically failovered (1) is reached.
auto_failover002 ns_1@10.206.117.77 06:22:27 - Wed Mar 21, 2012
Could not auto-failover more nodes ('ns_1@10.206.117.77'). Maximum number of nodes that will be automatically failovered (1) is reached.
(repeated 71 times) auto_failover002 ns_1@10.206.117.77 06:22:23 - Wed Mar 21, 2012
Starting rebalance, KeepNodes = ['ns_1@10.40.177.223','ns_1@10.90.145.211',
'ns_1@10.98.183.173','ns_1@10.100.243.14'], EjectNodes = ['ns_1@10.206.117.77']
(repeated 3 times) ns_orchestrator004 ns_1@10.206.117.77 06:19:23 - Wed Mar 21, 2012
Rebalance exited with reason {wait_for_memcached_failed,"default",
['ns_1@10.206.117.77']}
ns_orchestrator002 ns_1@10.206.117.77 06:18:48 - Wed Mar 21, 2012
Could not auto-failover more nodes ('ns_1@10.206.117.77'). Maximum number of nodes that will be automatically failovered (1) is reached.
auto_failover002 ns_1@10.206.117.77 06:16:27 - Wed Mar 21, 2012
Rebalance exited with reason {wait_for_memcached_failed,"default",
['ns_1@10.206.117.77']}
(repeated 4 times) ns_orchestrator002 ns_1@10.206.117.77 06:16:23 - Wed Mar 21, 2012
Could not auto-failover more nodes ('ns_1@10.206.117.77'). Maximum number of nodes that will be automatically failovered (1) is reached.
(repeated 71 times) auto_failover002 ns_1@10.206.117.77 06:16:23 - Wed Mar 21, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 10:02:47: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 10:02:47: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 06:15:03 - Wed Mar 21, 2012
Starting rebalance, KeepNodes = ['ns_1@10.40.177.223','ns_1@10.90.145.211',
'ns_1@10.98.183.173','ns_1@10.100.243.14'], EjectNodes = ['ns_1@10.206.117.77']
ns_orchestrator004 ns_1@10.206.117.77 06:14:19 - Wed Mar 21, 2012
Starting rebalance, KeepNodes = ['ns_1@10.40.177.223','ns_1@10.90.145.211',
'ns_1@10.98.183.173','ns_1@10.100.243.14',
'ns_1@10.206.117.77'], EjectNodes = []
ns_orchestrator004 ns_1@10.206.117.77 06:12:36 - Wed Mar 21, 2012
Starting rebalance, KeepNodes = ['ns_1@10.40.177.223','ns_1@10.90.145.211',
'ns_1@10.98.183.173','ns_1@10.100.243.14'], EjectNodes = ['ns_1@10.206.117.77']
(repeated 5 times) ns_orchestrator004 ns_1@10.206.117.77 06:11:23 - Wed Mar 21, 2012
Rebalance exited with reason {wait_for_memcached_failed,"default",
['ns_1@10.206.117.77']}
ns_orchestrator002 ns_1@10.206.117.77 06:11:04 - Wed Mar 21, 2012
Could not auto-failover more nodes ('ns_1@10.206.117.77'). Maximum number of nodes that will be automatically failovered (1) is reached.
auto_failover002 ns_1@10.206.117.77 06:10:27 - Wed Mar 21, 2012
Starting rebalance, KeepNodes = ['ns_1@10.40.177.223','ns_1@10.90.145.211',
'ns_1@10.98.183.173','ns_1@10.100.243.14',
'ns_1@10.206.117.77'], EjectNodes = []
(repeated 1 times) ns_orchestrator004 ns_1@10.206.117.77 06:10:23 - Wed Mar 21, 2012
Rebalance exited with reason {wait_for_memcached_failed,"default",
['ns_1@10.206.117.77']}
(repeated 6 times) ns_orchestrator002 ns_1@10.206.117.77 06:10:23 - Wed Mar 21, 2012
Could not auto-failover more nodes ('ns_1@10.206.117.77'). Maximum number of nodes that will be automatically failovered (1) is reached.
(repeated 64 times) auto_failover002 ns_1@10.206.117.77 06:10:23 - Wed Mar 21, 2012
Starting rebalance, KeepNodes = ['ns_1@10.40.177.223','ns_1@10.90.145.211',
'ns_1@10.98.183.173','ns_1@10.100.243.14'], EjectNodes = ['ns_1@10.206.117.77']
ns_orchestrator004 ns_1@10.206.117.77 06:05:44 - Wed Mar 21, 2012
Could not auto-failover more nodes ('ns_1@10.206.117.77'). Maximum number of nodes that will be automatically failovered (1) is reached.
auto_failover002 ns_1@10.206.117.77 06:05:02 - Wed Mar 21, 2012
Rebalance exited with reason {wait_for_memcached_failed,"default",
['ns_1@10.206.117.77']}
ns_orchestrator002 ns_1@10.206.117.77 06:04:36 - Wed Mar 21, 2012
Starting rebalance, KeepNodes = ['ns_1@10.40.177.223','ns_1@10.90.145.211',
'ns_1@10.98.183.173','ns_1@10.100.243.14',
'ns_1@10.206.117.77'], EjectNodes = []
ns_orchestrator004 ns_1@10.206.117.77 06:04:25 - Wed Mar 21, 2012
Shutting down bucket "default" on 'ns_1@10.206.117.77' for server shutdown ns_memcached002 ns_1@10.206.117.77 06:03:32 - Wed Mar 21, 2012
Node ('ns_1@10.206.117.77') was automatically failovered.
auto_failover001 ns_1@10.206.117.77 06:03:31 - Wed Mar 21, 2012
Failed over 'ns_1@10.206.117.77': ok ns_orchestrator006 ns_1@10.206.117.77 06:03:31 - Wed Mar 21, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 07:21:09: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 07:21:09: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 06:02:47 - Wed Mar 21, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 07:11:10: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 07:11:10: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 03:21:09 - Wed Mar 21, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 07:01:16: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 07:01:16: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 03:11:10 - Wed Mar 21, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 06:55:33: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 06:55:33: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 03:01:16 - Wed Mar 21, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 06:50:01: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 06:50:01: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 02:55:33 - Wed Mar 21, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 06:43:13: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 06:43:13: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 02:50:01 - Wed Mar 21, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 06:30:11: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 06:30:11: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 02:43:13 - Wed Mar 21, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 06:23:46: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 06:23:46: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 02:30:11 - Wed Mar 21, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 06:11:35: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 06:11:35: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 02:23:46 - Wed Mar 21, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 06:05:58: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 06:05:58: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 02:11:35 - Wed Mar 21, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 05:46:37: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 05:46:37: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 02:05:58 - Wed Mar 21, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 05:26:47: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 05:26:47: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 01:46:37 - Wed Mar 21, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 05:08:41: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 05:08:41: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 01:26:47 - Wed Mar 21, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 04:30:35: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 04:30:35: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 01:08:41 - Wed Mar 21, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 04:24:38: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 04:24:38: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 00:30:35 - Wed Mar 21, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 04:03:54: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 04:03:54: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 00:24:38 - Wed Mar 21, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 03:25:16: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 03:25:16: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 00:03:54 - Wed Mar 21, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 03:16:10: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 03:16:10: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 23:25:16 - Tue Mar 20, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 03:04:21: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 03:04:21: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 23:16:10 - Tue Mar 20, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 02:59:37: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 02:59:37: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 23:04:21 - Tue Mar 20, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 02:30:16: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 02:30:16: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 22:59:37 - Tue Mar 20, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 02:26:04: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 02:26:04: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 22:30:16 - Tue Mar 20, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 02:23:45: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 02:23:45: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 22:26:04 - Tue Mar 20, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 02:09:33: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 02:09:33: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 22:23:45 - Tue Mar 20, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 01:54:05: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 01:54:05: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 22:09:33 - Tue Mar 20, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 01:49:56: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 01:49:56: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 21:54:05 - Tue Mar 20, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 01:35:07: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 01:35:07: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 21:49:56 - Tue Mar 20, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 01:24:35: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 01:24:35: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 21:35:07 - Tue Mar 20, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 01:15:39: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 01:15:39: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 21:24:35 - Tue Mar 20, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 00:46:21: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 00:46:21: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 21:15:39 - Tue Mar 20, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 00:40:18: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 00:40:18: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 20:46:21 - Tue Mar 20, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 00:37:16: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 00:37:16: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 20:40:18 - Tue Mar 20, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 00:33:29: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 00:33:29: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 20:37:16 - Tue Mar 20, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 00:29:01: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 00:29:01: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 20:33:29 - Tue Mar 20, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-21 00:15:48: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-21 00:15:48: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 20:29:01 - Tue Mar 20, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-02-26 04:58:23: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-26 04:58:23: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 20:15:48 - Tue Mar 20, 2012

Top
Wed, 03/21/2012 - 16:54
alkondratenko
alkondratenko's picture
Offline
Joined: 12/01/2010
Groups: None

moxi is segfaulting (status code 139 tells me that). Please, enable core dumps and give run /opt/couchbase/bin/analyze_core on them.

Node ns_1@10.206.117.77 (where moxi is unhappy as well) is not healthy. Cluster tells you that it cannot automatically failover this node because limit is reached.

It's possible that memcached crashed on that node and it's now just warming up. Or it may be some worse problem.

Your options:

* wait until warmup is done (sadly we don't yet provide indication on UI, but you can grab stats via cbstats and observe if this is indeed warmup and if it makes any progress)

* fail over this node manually.

Top
Sun, 03/25/2012 - 17:08
ericbrown
Offline
Joined: 09/08/2011
Groups: None

Thanks for the reply. We've been having tons of problems over the past few days. How do I run that command?

Nodes keep falling out of the cluster or crashing or responding slowly. We've tried killing and restarting moxi to get this resolved.

ps ax | grep moxi
sudo kill -HUP PID

When a node crashes, it seems to be available again but requires a rebalance. To get a rebalance to work, we can remove the node then fail over, but it won't rebalance with remove only.

Here are more details from the past few days from the UI log.

Event Module Code Server Node Time
Port server moxi on node 'ns_1@10.40.177.223' exited with status 139. Restarting. Messages: 2012-03-25 23:56:21: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 23:56:21: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.40.177.223 19:57:42 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.40.177.223' exited with status 139. Restarting. Messages: 2012-03-25 23:54:54: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 23:54:54: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.40.177.223 19:56:21 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.40.177.223' exited with status 139. Restarting. Messages: 2012-03-25 23:53:31: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 23:53:31: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.40.177.223 19:54:54 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.40.177.223' exited with status 139. Restarting. Messages: 2012-03-25 23:51:50: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 23:51:50: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.40.177.223 19:53:31 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.40.177.223' exited with status 139. Restarting. Messages: 2012-03-25 23:50:35: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 23:50:35: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.40.177.223 19:51:50 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.40.177.223' exited with status 139. Restarting. Messages: 2012-03-25 23:49:09: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 23:49:09: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.40.177.223 19:50:35 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.40.177.223' exited with status 139. Restarting. Messages: 2012-03-25 23:31:43: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 23:31:43: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.40.177.223 19:49:09 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.40.177.223' exited with status 139. Restarting. Messages: 2012-03-25 11:36:16: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 11:36:16: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.40.177.223 19:31:43 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.90.145.211' exited with status 139. Restarting. Messages: 2012-03-25 11:51:26: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 11:51:26: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.90.145.211 15:16:48 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 17:37:11: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 17:37:11: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 13:40:00 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 17:33:41: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 17:33:41: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 13:37:11 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 17:31:28: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 17:31:28: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 13:33:41 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 17:17:44: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 17:17:44: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 13:31:28 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 17:15:02: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 17:15:02: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 13:17:43 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 17:01:40: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 17:01:40: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 13:15:02 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 16:53:24: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 16:53:24: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 13:01:40 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 16:47:09: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 16:47:09: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 12:53:24 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 16:40:33: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 16:40:33: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 12:47:09 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 16:33:18: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 16:33:18: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 12:40:33 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 16:22:53: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 16:22:53: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 12:33:18 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 16:18:18: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 16:18:18: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 12:22:53 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 16:16:53: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 16:16:53: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 12:18:18 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 16:15:42: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 16:15:42: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 12:16:53 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 15:48:49: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 15:48:49: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 12:15:42 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 15:48:12: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 15:48:12: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 11:48:49 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 15:47:27: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 15:47:27: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 11:48:12 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 15:46:01: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 15:46:01: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 11:47:27 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 15:43:40: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 15:43:40: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 11:46:01 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 15:30:52: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 15:30:52: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 11:43:40 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 15:26:42: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 15:26:42: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 11:30:52 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 15:22:19: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 15:22:19: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 11:26:42 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 15:15:46: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 15:15:46: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 11:22:19 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 15:12:27: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 15:12:27: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 11:15:46 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 15:09:13: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 15:09:13: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 11:12:27 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 15:05:40: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 15:05:40: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 11:09:13 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 14:49:12: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 14:49:12: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 11:05:40 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 14:46:53: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 14:46:53: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 10:49:12 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 14:45:10: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 14:45:10: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 10:46:53 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 14:43:49: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 14:43:49: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 10:45:10 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 14:41:59: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 14:41:59: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 10:43:49 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 14:37:50: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 14:37:50: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 10:41:59 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 14:32:23: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 14:32:23: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 10:37:50 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 14:30:55: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 14:30:55: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 10:32:23 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 14:22:21: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 14:22:21: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 10:30:55 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 14:16:57: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 14:16:57: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 10:22:21 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 14:01:40: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 14:01:40: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 10:16:57 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 13:51:30: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 13:51:30: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 10:01:39 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 13:44:38: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 13:44:38: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 09:51:30 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 13:31:29: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 13:31:29: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 09:44:38 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 13:20:01: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 13:20:01: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 09:31:29 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 13:18:19: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 13:18:19: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 09:20:01 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 13:16:45: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 13:16:45: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 09:18:19 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 13:15:51: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 13:15:51: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 09:16:45 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 13:13:25: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 13:13:25: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 09:15:51 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 13:11:07: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 13:11:07: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 09:13:25 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 12:29:11: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 12:29:11: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 09:11:07 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 12:19:41: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 12:19:41: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 08:29:11 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 12:18:19: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 12:18:19: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 08:19:41 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 12:13:50: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 12:13:50: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 08:18:19 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 12:11:31: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 12:11:31: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 08:13:50 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 12:07:30: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 12:07:30: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 08:11:31 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 11:44:28: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 11:44:28: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 08:07:30 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.90.145.211' exited with status 139. Restarting. Messages: 2012-03-25 10:26:14: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 10:26:14: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.90.145.211 07:51:26 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 11:41:48: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 11:41:48: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 07:44:28 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 11:36:16: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 11:36:16: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 07:41:48 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 11:32:13: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 11:32:13: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 07:36:16 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.40.177.223' exited with status 139. Restarting. Messages: 2012-03-25 08:48:18: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 08:48:18: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.40.177.223 07:36:16 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 11:28:49: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 11:28:49: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 07:32:13 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 11:20:36: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 11:20:36: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 07:28:48 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 11:18:45: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 11:18:45: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 07:20:36 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 11:16:13: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 11:16:13: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 07:18:45 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 11:12:54: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 11:12:54: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 07:16:13 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 11:07:43: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 11:07:43: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 07:12:54 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 11:04:17: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 11:04:17: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 07:07:43 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 10:59:18: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 10:59:18: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 07:04:17 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 10:17:28: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 10:17:28: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 06:59:18 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.90.145.211' exited with status 139. Restarting. Messages: 2012-03-25 08:59:58: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 08:59:58: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.90.145.211 06:26:14 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 10:13:39: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 10:13:39: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 06:17:28 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 09:49:01: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 09:49:01: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 06:13:39 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 09:42:53: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 09:42:53: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 05:49:01 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 09:36:19: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 09:36:19: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 05:42:53 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 09:27:15: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 09:27:15: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 05:36:19 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 09:21:26: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 09:21:26: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 05:27:15 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 09:03:33: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 09:03:33: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 05:21:26 - Sun Mar 25, 2012
Rebalance completed successfully.
ns_orchestrator001 ns_1@10.100.243.14 05:07:35 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 08:58:29: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 08:58:29: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 05:03:33 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.90.145.211' exited with status 139. Restarting. Messages: 2012-03-25 08:46:14: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 08:46:14: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.90.145.211 04:59:58 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 08:52:37: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 08:52:37: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 04:58:29 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 08:46:11: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 08:46:11: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 04:52:36 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.98.183.173' exited with status 139. Restarting. Messages: 2012-03-25 08:22:22: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 08:22:22: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.98.183.173 04:50:15 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.40.177.223' exited with status 139. Restarting. Messages: 2012-03-25 08:32:15: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 08:32:15: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.40.177.223 04:48:18 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.90.145.211' exited with status 139. Restarting. Messages: 2012-03-25 08:08:57: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 08:08:57: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.90.145.211 04:46:13 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 08:39:43: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 08:39:43: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 04:46:11 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 08:30:13: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 08:30:13: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 04:39:43 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.40.177.223' exited with status 139. Restarting. Messages: 2012-03-25 08:08:11: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 08:08:11: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.40.177.223 04:32:15 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 08:25:30: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 08:25:30: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 04:30:13 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 08:07:15: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 08:07:15: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 04:25:30 - Sun Mar 25, 2012
Bucket "default" loaded on node 'ns_1@10.98.183.173' in 0 seconds. ns_memcached001 ns_1@10.98.183.173 04:22:38 - Sun Mar 25, 2012
Starting rebalance, KeepNodes = ['ns_1@10.40.177.223','ns_1@10.90.145.211',
'ns_1@10.100.243.14','ns_1@10.206.117.77',
'ns_1@10.98.183.173'], EjectNodes = []
ns_orchestrator004 ns_1@10.100.243.14 04:22:36 - Sun Mar 25, 2012
Node 'ns_1@10.100.243.14' saw that node 'ns_1@10.98.183.173' came up. ns_node_disco004 ns_1@10.100.243.14 04:22:24 - Sun Mar 25, 2012
Node 'ns_1@10.98.183.173' saw that node 'ns_1@10.100.243.14' came up. ns_node_disco004 ns_1@10.98.183.173 04:22:24 - Sun Mar 25, 2012
Node 'ns_1@10.206.117.77' saw that node 'ns_1@10.98.183.173' came up. ns_node_disco004 ns_1@10.206.117.77 04:22:23 - Sun Mar 25, 2012
Node 'ns_1@10.98.183.173' saw that node 'ns_1@10.206.117.77' came up. ns_node_disco004 ns_1@10.98.183.173 04:22:23 - Sun Mar 25, 2012
Node 'ns_1@10.98.183.173' saw that node 'ns_1@10.40.177.223' came up. ns_node_disco004 ns_1@10.98.183.173 04:22:23 - Sun Mar 25, 2012
Node 'ns_1@10.40.177.223' saw that node 'ns_1@10.98.183.173' came up. ns_node_disco004 ns_1@10.40.177.223 04:22:23 - Sun Mar 25, 2012
Node ns_1@10.98.183.173 joined cluster ns_cluster003 ns_1@10.98.183.173 04:22:22 - Sun Mar 25, 2012
Couchbase Server has started on web port 8091 on node 'ns_1@10.98.183.173'. menelaus_sup001 ns_1@10.98.183.173 04:22:22 - Sun Mar 25, 2012
Node 'ns_1@10.90.145.211' saw that node 'ns_1@10.98.183.173' came up. ns_node_disco004 ns_1@10.90.145.211 04:22:22 - Sun Mar 25, 2012
Rebalance completed successfully.
ns_orchestrator001 ns_1@10.100.243.14 04:19:09 - Sun Mar 25, 2012
Node 'ns_1@10.100.243.14' saw that node 'ns_1@10.98.183.173' went down. (repeated 1 times) ns_node_disco005 ns_1@10.100.243.14 04:16:47 - Sun Mar 25, 2012
Node 'ns_1@10.40.177.223' saw that node 'ns_1@10.98.183.173' went down. (repeated 1 times) ns_node_disco005 ns_1@10.40.177.223 04:16:41 - Sun Mar 25, 2012
Node 'ns_1@10.90.145.211' saw that node 'ns_1@10.98.183.173' went down. (repeated 1 times) ns_node_disco005 ns_1@10.90.145.211 04:16:39 - Sun Mar 25, 2012
Node 'ns_1@10.206.117.77' saw that node 'ns_1@10.98.183.173' went down. (repeated 1 times) ns_node_disco005 ns_1@10.206.117.77 04:16:14 - Sun Mar 25, 2012
Rebalance exited with reason {wait_for_memcached_failed,"default",
['ns_1@10.98.183.173']}
(repeated 2 times) ns_orchestrator002 ns_1@10.100.243.14 04:14:47 - Sun Mar 25, 2012
Starting rebalance, KeepNodes = ['ns_1@10.40.177.223','ns_1@10.90.145.211',
'ns_1@10.98.183.173','ns_1@10.100.243.14',
'ns_1@10.206.117.77'], EjectNodes = []
(repeated 1 times) ns_orchestrator004 ns_1@10.100.243.14 04:13:47 - Sun Mar 25, 2012
Node 'ns_1@10.98.183.173' is leaving cluster. ns_cluster001 ns_1@10.98.183.173 04:11:46 - Sun Mar 25, 2012
Starting rebalance, KeepNodes = ['ns_1@10.40.177.223','ns_1@10.90.145.211',
'ns_1@10.100.243.14','ns_1@10.206.117.77'], EjectNodes = []
ns_orchestrator004 ns_1@10.100.243.14 04:11:46 - Sun Mar 25, 2012
Couchbase Server has started on web port 8091 on node 'ns_1@10.98.183.173'. menelaus_sup001 ns_1@10.98.183.173 04:11:08 - Sun Mar 25, 2012
Node 'ns_1@10.206.117.77' saw that node 'ns_1@10.98.183.173' came up. ns_node_disco004 ns_1@10.206.117.77 04:11:07 - Sun Mar 25, 2012
Node 'ns_1@10.206.117.77' saw that node 'ns_1@10.98.183.173' went down. ns_node_disco005 ns_1@10.206.117.77 04:11:07 - Sun Mar 25, 2012
Node 'ns_1@10.40.177.223' saw that node 'ns_1@10.98.183.173' came up. ns_node_disco004 ns_1@10.40.177.223 04:11:07 - Sun Mar 25, 2012
Node 'ns_1@10.40.177.223' saw that node 'ns_1@10.98.183.173' went down. ns_node_disco005 ns_1@10.40.177.223 04:11:07 - Sun Mar 25, 2012
Node 'ns_1@10.90.145.211' saw that node 'ns_1@10.98.183.173' came up. ns_node_disco004 ns_1@10.90.145.211 04:11:06 - Sun Mar 25, 2012
Node 'ns_1@10.90.145.211' saw that node 'ns_1@10.98.183.173' went down. ns_node_disco005 ns_1@10.90.145.211 04:11:06 - Sun Mar 25, 2012
Node 'ns_1@10.100.243.14' saw that node 'ns_1@10.98.183.173' came up. ns_node_disco004 ns_1@10.100.243.14 04:11:05 - Sun Mar 25, 2012
Node 'ns_1@10.100.243.14' saw that node 'ns_1@10.98.183.173' went down. ns_node_disco005 ns_1@10.100.243.14 04:11:05 - Sun Mar 25, 2012
Node 'ns_1@10.98.183.173' synchronized otp cookie qouhqbcugdhzjctj from cluster ns_node_disco002 ns_1@10.98.183.173 04:11:04 - Sun Mar 25, 2012
Node ('ns_1@10.98.183.173') was automatically failovered.
auto_failover001 ns_1@10.100.243.14 04:09:56 - Sun Mar 25, 2012
Failed over 'ns_1@10.98.183.173': ok ns_orchestrator006 ns_1@10.100.243.14 04:09:56 - Sun Mar 25, 2012
Shutting down bucket "default" on 'ns_1@10.98.183.173' for server shutdown ns_memcached002 ns_1@10.98.183.173 04:09:53 - Sun Mar 25, 2012
Starting rebalance, KeepNodes = ['ns_1@10.40.177.223','ns_1@10.90.145.211',
'ns_1@10.100.243.14','ns_1@10.206.117.77'], EjectNodes = ['ns_1@10.98.183.173']
ns_orchestrator004 ns_1@10.100.243.14 04:09:30 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.90.145.211' exited with status 139. Restarting. Messages: 2012-03-25 08:01:29: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 08:01:29: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.90.145.211 04:08:57 - Sun Mar 25, 2012
Rebalance exited with reason {wait_for_memcached_failed,"default",
['ns_1@10.98.183.173']}
ns_orchestrator002 ns_1@10.100.243.14 04:08:56 - Sun Mar 25, 2012
Starting rebalance, KeepNodes = ['ns_1@10.40.177.223','ns_1@10.90.145.211',
'ns_1@10.98.183.173','ns_1@10.100.243.14',
'ns_1@10.206.117.77'], EjectNodes = []
ns_orchestrator004 ns_1@10.100.243.14 04:08:45 - Sun Mar 25, 2012
Rebalance exited with reason {{{badmatch,{error,closed}},
[{mc_client_binary,cmd_binary_vocal_recv,5},
{mc_client_binary,get_vbucket,2},
{ns_memcached,do_handle_call,3},
{ns_memcached,handle_call,3},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]},
{gen_server,call,
[{'ns_memcached-default',
'ns_1@10.98.183.173'},
{get_vbucket,214},
30000]}}
ns_orchestrator002 ns_1@10.100.243.14 04:08:34 - Sun Mar 25, 2012
Control connection to memcached on 'ns_1@10.98.183.173' disconnected: {{badmatch,
{error,
closed}},
[{mc_client_binary,
cmd_binary_vocal_recv,
5},
{mc_client_binary,
get_vbucket,
2},
{ns_memcached,
do_handle_call,
3},
{ns_memcached,
handle_call,
3},
{gen_server,
handle_msg,
5},
{proc_lib,
init_p_do_apply,
3}]} ns_memcached004 ns_1@10.98.183.173 04:08:34 - Sun Mar 25, 2012
Port server memcached on node 'ns_1@10.98.183.173' exited with status 129. Restarting. Messages: Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs
Suspend eq_tapq:replication_ns_1@10.100.243.14 for 5.00 secs ns_port_server000 ns_1@10.98.183.173 04:08:34 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.40.177.223' exited with status 139. Restarting. Messages: 2012-03-25 08:02:26: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 08:02:26: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.40.177.223 04:08:11 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.100.243.14' exited with status 139. Restarting. Messages: 2012-03-25 08:00:44: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 08:00:44: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.100.243.14 04:07:15 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.98.183.173' exited with status 139. Restarting. Messages: 2012-03-25 08:02:03: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 08:02:03: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.98.183.173 04:07:15 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 08:03:40: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 08:03:40: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 04:07:15 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 08:00:54: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 08:00:54: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 04:03:40 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.40.177.223' exited with status 139. Restarting. Messages: 2012-03-25 07:58:07: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:58:07: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.40.177.223 04:02:26 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.98.183.173' exited with status 139. Restarting. Messages: 2012-03-25 07:57:49: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:57:49: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.98.183.173 04:02:03 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.90.145.211' exited with status 139. Restarting. Messages: 2012-03-25 07:51:20: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:51:20: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.90.145.211 04:01:29 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 07:57:04: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:57:04: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 04:00:54 - Sun Mar 25, 2012
Starting rebalance, KeepNodes = ['ns_1@10.40.177.223','ns_1@10.90.145.211',
'ns_1@10.98.183.173','ns_1@10.100.243.14',
'ns_1@10.206.117.77'], EjectNodes = []
(repeated 5 times) ns_orchestrator004 ns_1@10.100.243.14 04:00:47 - Sun Mar 25, 2012
Rebalance exited with reason stopped
(repeated 5 times) ns_orchestrator002 ns_1@10.100.243.14 04:00:47 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.100.243.14' exited with status 139. Restarting. Messages: 2012-03-25 07:50:28: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:50:28: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.100.243.14 04:00:44 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.40.177.223' exited with status 139. Restarting. Messages: 2012-03-25 07:50:12: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:50:12: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.40.177.223 03:58:07 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.98.183.173' exited with status 129. Restarting. Messages: 2012-03-25 07:54:22: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:54:22: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.98.183.173 03:57:49 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 07:52:51: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:52:51: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 03:57:04 - Sun Mar 25, 2012
Starting rebalance, KeepNodes = ['ns_1@10.40.177.223','ns_1@10.90.145.211',
'ns_1@10.98.183.173','ns_1@10.100.243.14',
'ns_1@10.206.117.77'], EjectNodes = []
ns_orchestrator004 ns_1@10.100.243.14 03:55:01 - Sun Mar 25, 2012
Rebalance exited with reason stopped
ns_orchestrator002 ns_1@10.100.243.14 03:54:57 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.98.183.173' exited with status 139. Restarting. Messages: 2012-03-25 07:47:28: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:47:28: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.98.183.173 03:54:22 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 07:49:10: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:49:10: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 03:52:51 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.90.145.211' exited with status 139. Restarting. Messages: 2012-03-25 07:43:18: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:43:18: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.90.145.211 03:51:20 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.100.243.14' exited with status 139. Restarting. Messages: 2012-03-25 07:34:46: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:34:46: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.100.243.14 03:50:28 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.40.177.223' exited with status 139. Restarting. Messages: 2012-03-25 07:42:41: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:42:41: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.40.177.223 03:50:12 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 07:45:36: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:45:36: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 03:49:10 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.98.183.173' exited with status 139. Restarting. Messages: 2012-03-25 07:37:35: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:37:35: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.98.183.173 03:47:28 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 07:40:43: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:40:43: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 03:45:36 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.90.145.211' exited with status 139. Restarting. Messages: 2012-03-25 07:33:51: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:33:51: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.90.145.211 03:43:18 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.40.177.223' exited with status 139. Restarting. Messages: 2012-03-25 07:35:24: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:35:24: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.40.177.223 03:42:41 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 07:36:53: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:36:53: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 03:40:42 - Sun Mar 25, 2012
Starting rebalance, KeepNodes = ['ns_1@10.40.177.223','ns_1@10.90.145.211',
'ns_1@10.98.183.173','ns_1@10.100.243.14',
'ns_1@10.206.117.77'], EjectNodes = []
ns_orchestrator004 ns_1@10.100.243.14 03:40:06 - Sun Mar 25, 2012
Rebalance exited with reason stopped
ns_orchestrator002 ns_1@10.100.243.14 03:39:56 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.98.183.173' exited with status 139. Restarting. Messages: 2012-03-25 07:28:33: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:28:33: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.98.183.173 03:37:35 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 07:32:59: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:32:59: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 03:36:53 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.40.177.223' exited with status 139. Restarting. Messages: 2012-03-25 07:27:56: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:27:56: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.40.177.223 03:35:24 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.100.243.14' exited with status 139. Restarting. Messages: 2012-03-25 06:47:43: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 06:47:43: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.100.243.14 03:34:46 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.90.145.211' exited with status 139. Restarting. Messages: 2012-03-25 07:20:29: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:20:29: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.90.145.211 03:33:51 - Sun Mar 25, 2012
Starting rebalance, KeepNodes = ['ns_1@10.40.177.223','ns_1@10.90.145.211',
'ns_1@10.98.183.173','ns_1@10.100.243.14',
'ns_1@10.206.117.77'], EjectNodes = []
ns_orchestrator004 ns_1@10.100.243.14 03:33:27 - Sun Mar 25, 2012
Rebalance exited with reason stopped
ns_orchestrator002 ns_1@10.100.243.14 03:33:09 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 07:29:21: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:29:21: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 03:32:59 - Sun Mar 25, 2012
Starting rebalance, KeepNodes = ['ns_1@10.40.177.223','ns_1@10.90.145.211',
'ns_1@10.98.183.173','ns_1@10.100.243.14',
'ns_1@10.206.117.77'], EjectNodes = []
(repeated 2 times) ns_orchestrator004 ns_1@10.100.243.14 03:31:47 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 07:24:39: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:24:39: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 03:29:21 - Sun Mar 25, 2012
Rebalance exited with reason stopped
(repeated 2 times) ns_orchestrator002 ns_1@10.100.243.14 03:28:47 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.98.183.173' exited with status 129. Restarting. Messages: 2012-03-25 07:20:55: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:20:55: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.98.183.173 03:28:33 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.40.177.223' exited with status 129. Restarting. Messages: 2012-03-25 07:20:29: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:20:29: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.40.177.223 03:27:55 - Sun Mar 25, 2012
Starting rebalance, KeepNodes = ['ns_1@10.40.177.223','ns_1@10.90.145.211',
'ns_1@10.98.183.173','ns_1@10.100.243.14',
'ns_1@10.206.117.77'], EjectNodes = []
ns_orchestrator004 ns_1@10.100.243.14 03:26:37 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 07:20:51: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:20:51: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 03:24:39 - Sun Mar 25, 2012
Rebalance exited with reason stopped
ns_orchestrator002 ns_1@10.100.243.14 03:22:52 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.98.183.173' exited with status 139. Restarting. Messages: 2012-03-25 07:11:02: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:11:02: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.98.183.173 03:20:53 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 07:17:44: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:17:44: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 03:20:50 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.40.177.223' exited with status 139. Restarting. Messages: 2012-03-25 07:09:27: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:09:27: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.40.177.223 03:20:29 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.90.145.211' exited with status 139. Restarting. Messages: 2012-03-25 07:10:02: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:10:02: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.90.145.211 03:20:29 - Sun Mar 25, 2012
Starting rebalance, KeepNodes = ['ns_1@10.40.177.223','ns_1@10.90.145.211',
'ns_1@10.98.183.173','ns_1@10.100.243.14',
'ns_1@10.206.117.77'], EjectNodes = []
(repeated 1 times) ns_orchestrator004 ns_1@10.100.243.14 03:18:47 - Sun Mar 25, 2012
Rebalance exited with reason stopped
(repeated 1 times) ns_orchestrator002 ns_1@10.100.243.14 03:18:47 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 07:14:30: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:14:30: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 03:17:44 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 07:09:15: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:09:15: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 03:14:30 - Sun Mar 25, 2012
Starting rebalance, KeepNodes = ['ns_1@10.40.177.223','ns_1@10.90.145.211',
'ns_1@10.98.183.173','ns_1@10.100.243.14',
'ns_1@10.206.117.77'], EjectNodes = []
ns_orchestrator004 ns_1@10.100.243.14 03:12:55 - Sun Mar 25, 2012
Rebalance exited with reason stopped
ns_orchestrator002 ns_1@10.100.243.14 03:12:50 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.98.183.173' exited with status 139. Restarting. Messages: 2012-03-25 07:00:49: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:00:49: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.98.183.173 03:11:00 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.90.145.211' exited with status 139. Restarting. Messages: 2012-03-25 06:48:54: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 06:48:54: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.90.145.211 03:10:02 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.40.177.223' exited with status 139. Restarting. Messages: 2012-03-25 06:57:24: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 06:57:24: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.40.177.223 03:09:27 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 07:03:59: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 07:03:59: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 03:09:15 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 06:59:44: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 06:59:44: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 03:03:59 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.98.183.173' exited with status 139. Restarting. Messages: 2012-03-25 06:47:15: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 06:47:15: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.98.183.173 03:00:49 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 06:55:27: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 06:55:27: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 02:59:44 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.40.177.223' exited with status 139. Restarting. Messages: 2012-03-25 06:47:40: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 06:47:40: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.40.177.223 02:57:24 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 06:51:21: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 06:51:21: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 02:55:27 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 06:47:29: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 06:47:29: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 02:51:21 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.90.145.211' exited with status 139. Restarting. Messages: 2012-03-25 06:27:59: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 06:27:59: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.90.145.211 02:48:54 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.100.243.14' exited with status 139. Restarting. Messages: 2012-03-25 06:31:58: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 06:31:58: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.100.243.14 02:47:42 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.40.177.223' exited with status 139. Restarting. Messages: 2012-03-25 06:38:41: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 06:38:41: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.40.177.223 02:47:40 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 06:42:36: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 06:42:36: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 02:47:29 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.98.183.173' exited with status 139. Restarting. Messages: 2012-03-25 06:36:55: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 06:36:55: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.98.183.173 02:47:15 - Sun Mar 25, 2012
Port server memcached on node 'ns_1@10.40.177.223' exited with status 71. Restarting. Messages: failed to listen on TCP port 11210: Address already in use (repeated 4 times) ns_port_server000 ns_1@10.40.177.223 02:44:41 - Sun Mar 25, 2012
Node 'ns_1@10.90.145.211' saw that node 'ns_1@10.40.177.223' came up. (repeated 1 times) ns_node_disco004 ns_1@10.90.145.211 02:43:39 - Sun Mar 25, 2012
Node 'ns_1@10.98.183.173' saw that node 'ns_1@10.40.177.223' went down. (repeated 1 times) ns_node_disco005 ns_1@10.98.183.173 02:43:29 - Sun Mar 25, 2012
Node 'ns_1@10.98.183.173' saw that node 'ns_1@10.40.177.223' came up. (repeated 1 times) ns_node_disco004 ns_1@10.98.183.173 02:43:29 - Sun Mar 25, 2012
Node 'ns_1@10.206.117.77' saw that node 'ns_1@10.40.177.223' went down. (repeated 1 times) ns_node_disco005 ns_1@10.206.117.77 02:43:14 - Sun Mar 25, 2012
Node 'ns_1@10.206.117.77' saw that node 'ns_1@10.40.177.223' came up. (repeated 1 times) ns_node_disco004 ns_1@10.206.117.77 02:43:14 - Sun Mar 25, 2012
Node 'ns_1@10.100.243.14' saw that node 'ns_1@10.40.177.223' went down. (repeated 1 times) ns_node_disco005 ns_1@10.100.243.14 02:42:47 - Sun Mar 25, 2012
Node 'ns_1@10.100.243.14' saw that node 'ns_1@10.40.177.223' came up. (repeated 1 times) ns_node_disco004 ns_1@10.100.243.14 02:42:47 - Sun Mar 25, 2012
Node 'ns_1@10.90.145.211' saw that node 'ns_1@10.40.177.223' went down. (repeated 1 times) ns_node_disco005 ns_1@10.90.145.211 02:42:39 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 06:36:59: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 06:36:59: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 02:42:36 - Sun Mar 25, 2012
Bucket "default" loaded on node 'ns_1@10.40.177.223' in 0 seconds. ns_memcached001 ns_1@10.40.177.223 02:39:07 - Sun Mar 25, 2012
Starting rebalance, KeepNodes = ['ns_1@10.90.145.211','ns_1@10.98.183.173',
'ns_1@10.100.243.14','ns_1@10.206.117.77',
'ns_1@10.40.177.223'], EjectNodes = []
ns_orchestrator004 ns_1@10.100.243.14 02:39:01 - Sun Mar 25, 2012
Service memcached exited on node 'ns_1@10.40.177.223' in 0.03s
supervisor_cushion001 ns_1@10.40.177.223 02:39:01 - Sun Mar 25, 2012
Service memcached exited on node 'ns_1@10.40.177.223' in 0.03s
supervisor_cushion001 ns_1@10.40.177.223 02:38:56 - Sun Mar 25, 2012
Service memcached exited on node 'ns_1@10.40.177.223' in 0.03s
supervisor_cushion001 ns_1@10.40.177.223 02:38:51 - Sun Mar 25, 2012
Service memcached exited on node 'ns_1@10.40.177.223' in 0.03s
supervisor_cushion001 ns_1@10.40.177.223 02:38:46 - Sun Mar 25, 2012
Node 'ns_1@10.40.177.223' saw that node 'ns_1@10.90.145.211' came up. ns_node_disco004 ns_1@10.40.177.223 02:38:44 - Sun Mar 25, 2012
Node 'ns_1@10.40.177.223' saw that node 'ns_1@10.98.183.173' came up. ns_node_disco004 ns_1@10.40.177.223 02:38:41 - Sun Mar 25, 2012
Service memcached exited on node 'ns_1@10.40.177.223' in 0.10s
supervisor_cushion001 ns_1@10.40.177.223 02:38:41 - Sun Mar 25, 2012
Port server memcached on node 'ns_1@10.40.177.223' exited with status 71. Restarting. Messages: failed to listen on TCP port 11210: Address already in use ns_port_server000 ns_1@10.40.177.223 02:38:41 - Sun Mar 25, 2012
Node ns_1@10.40.177.223 joined cluster ns_cluster003 ns_1@10.40.177.223 02:38:41 - Sun Mar 25, 2012
Couchbase Server has started on web port 8091 on node 'ns_1@10.40.177.223'. menelaus_sup001 ns_1@10.40.177.223 02:38:41 - Sun Mar 25, 2012
Node 'ns_1@10.40.177.223' saw that node 'ns_1@10.206.117.77' came up. ns_node_disco004 ns_1@10.40.177.223 02:38:41 - Sun Mar 25, 2012
Conflicting configuration changes to field {node,'ns_1@10.40.177.223',
memcached}:
[{'_vclock',[{'ns_1@10.40.177.223',{2,63499876718}}]},
{dbdir,"/opt/couchbase/var/lib/couchbase/data"},
{port,11210},
{admin_user,"_admin"},
{admin_pass,"_admin"},
{bucket_engine,"/opt/couchbase/lib/memcached/bucket_engine.so"},
{engines,
[{membase,
[{engine,"/opt/couchbase/lib/memcached/ep.so"},
{initfile,"/opt/couchbase/etc/couchbase/init.sql"},
{static_config_string,
"vb0=false;waitforwarmup=false;failpartialwarmup=false;shardpattern=%d/%b-%i.mb;db_strategy=multiMTVBDB"}]},
{memcached,
[{engine,"/opt/couchbase/lib/memcached/default_engine.so"},
{static_config_string,"vb0=true"}]}]},
{verbosity,[]}] and
[{'_vclock',
[{'ns_1@10.40.177.223',{1,63495720165}},
{'ns_1@127.0.0.1',{1,63495640288}}]},
{port,11210},
{dbdir,"/opt/couchbase/var/lib/couchbase/data"},
{admin_user,"_admin"},
{admin_pass,"_admin"},
{bucket_engine,"/opt/couchbase/lib/memcached/bucket_engine.so"},
{engines,
[{membase,
[{engine,"/opt/couchbase/lib/memcached/ep.so"},
{initfile,"/opt/couchbase/etc/couchbase/init.sql"},
{static_config_string,
"vb0=false;waitforwarmup=false;failpartialwarmup=false;shardpattern=%d/%b-%i.mb;db_strategy=multiMTVBDB"}]},
{memcached,
[{engine,"/opt/couchbase/lib/memcached/default_engine.so"},
{static_config_string,"vb0=true"}]}]},
{verbosity,[]}], choosing the former, which looks newer.
ns_config003 ns_1@10.40.177.223 02:38:41 - Sun Mar 25, 2012
Node 'ns_1@10.98.183.173' saw that node 'ns_1@10.40.177.223' came up. ns_node_disco004 ns_1@10.98.183.173 02:37:45 - Sun Mar 25, 2012
Node 'ns_1@10.90.145.211' saw that node 'ns_1@10.40.177.223' came up. ns_node_disco004 ns_1@10.90.145.211 02:37:45 - Sun Mar 25, 2012
Couchbase Server has started on web port 8091 on node 'ns_1@10.40.177.223'. menelaus_sup001 ns_1@10.40.177.223 02:37:45 - Sun Mar 25, 2012
Node 'ns_1@10.40.177.223' is leaving cluster. ns_cluster001 ns_1@10.40.177.223 02:37:45 - Sun Mar 25, 2012
Node 'ns_1@10.206.117.77' saw that node 'ns_1@10.40.177.223' came up. ns_node_disco004 ns_1@10.206.117.77 02:37:45 - Sun Mar 25, 2012
Node 'ns_1@10.100.243.14' saw that node 'ns_1@10.40.177.223' came up. ns_node_disco004 ns_1@10.100.243.14 02:37:45 - Sun Mar 25, 2012
Node 'ns_1@10.40.177.223' synchronized otp cookie qouhqbcugdhzjctj from cluster ns_node_disco002 ns_1@10.40.177.223 02:37:43 - Sun Mar 25, 2012
Node 'ns_1@10.206.117.77' saw that node 'ns_1@10.40.177.223' went down. ns_node_disco005 ns_1@10.206.117.77 02:37:38 - Sun Mar 25, 2012
Node 'ns_1@10.90.145.211' saw that node 'ns_1@10.40.177.223' went down. ns_node_disco005 ns_1@10.90.145.211 02:37:37 - Sun Mar 25, 2012
Node 'ns_1@10.98.183.173' saw that node 'ns_1@10.40.177.223' went down. ns_node_disco005 ns_1@10.98.183.173 02:37:36 - Sun Mar 25, 2012
Node 'ns_1@10.100.243.14' saw that node 'ns_1@10.40.177.223' went down. ns_node_disco005 ns_1@10.100.243.14 02:37:34 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-25 06:31:04: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 06:31:04: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 02:36:59 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.98.183.173' exited with status 134. Restarting. Messages: 2012-03-25 06:31:42: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 06:31:42: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
moxi: cproxy.c:1185: cproxy_release_downstream: Assertion `!found' failed. ns_port_server000 ns_1@10.98.183.173 02:36:55 - Sun Mar 25, 2012
Starting rebalance, KeepNodes = ['ns_1@10.40.177.223','ns_1@10.90.145.211',
'ns_1@10.98.183.173','ns_1@10.100.243.14',
'ns_1@10.206.117.77'], EjectNodes = []
(repeated 1 times) ns_orchestrator004 ns_1@10.100.243.14 02:33:47 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.100.243.14' exited with status 139. Restarting. Messages: 2012-03-25 06:26:27: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 06:26:27: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.100.243.14 02:31:58 - Sun Mar 25, 2012
Rebalance exited with reason {wait_for_memcached_failed,"default",
['ns_1@10.40.177.223']}
(repeated 2 times) ns_orchestrator002 ns_1@10.100.243.14 02:31:47 - Sun Mar 25, 2012
Could not auto-failover more nodes ('ns_1@10.40.177.223'). Maximum number of nodes that will be automatically failovered (1) is reached.
(repeated 39 times) auto_failover002 ns_1@10.100.243.14 02:31:47 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.40.177.223' exited with status 139. Restarting. Messages: 2012-03-25 06:27:21: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-25 06:27:21: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.40.177.223 02:31:42 - Sun Mar 25, 2012

Top
Mon, 03/26/2012 - 09:13
ericbrown
Offline
Joined: 09/08/2011
Groups: None

It looks like reducing the memory allocation per node in the UI instead of using the max available had a positive effect. Not sure if it's coincidence. We still get segfaults on the one node though.

Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-26 15:00:45: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-26 15:00:45: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 11:02:30 - Mon Mar 26, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-26 14:56:44: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-26 14:56:44: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 11:00:45 - Mon Mar 26, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-26 14:25:32: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-26 14:25:32: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 10:56:44 - Mon Mar 26, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-26 14:23:30: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-26 14:23:30: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 10:25:32 - Mon Mar 26, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-26 14:18:37: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-26 14:18:37: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 10:23:30 - Mon Mar 26, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-26 03:22:53: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-26 03:22:53: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 10:18:37 - Mon Mar 26, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-26 03:00:17: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-26 03:00:17: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 23:22:53 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-26 02:28:49: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-26 02:28:49: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 23:00:17 - Sun Mar 25, 2012
Port server moxi on node 'ns_1@10.206.117.77' exited with status 139. Restarting. Messages: 2012-03-26 02:23:26: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-03-26 02:23:26: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10) ns_port_server000 ns_1@10.206.117.77 22:28:49 - Sun Mar 25, 2012

Top
Mon, 03/26/2012 - 11:32
alkondratenko
alkondratenko's picture
Offline
Joined: 12/01/2010
Groups: None

We need backtrace from moxi. Is that possible that ram on this box is broken ?

Top
Mon, 03/26/2012 - 11:35
ericbrown
Offline
Joined: 09/08/2011
Groups: None

I can work on figuring out how to get that.

Top