Nodes Keep Crashing
We are seeing our nodes crash and auto-failover on a regular basis. It happens at least twice a day if not more. In the logs I am seeing the following message about once a minute.
Port server moxi on node 'ns_1@10.40.177.223' exited with status 139. Restarting. Messages: 2012-02-09 14:18:22: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-09 14:18:22: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
Port server moxi on node 'ns_1@10.100.243.14' exited with status 139. Restarting. Messages: 2012-02-09 14:20:08: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-09 14:20:08: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
When the nodes auto-failover it takes several tries to get a rebalance to kick off and that seems to produce restart messages like the one above as well.
Not sure if it's related, but appreciate being pointed at the right documentation. Thanks!
We're on EC2 on a standard linux AMI
2.6.34.7-56.40.amzn1.x86_64
Amazon Linux AMI release 2010.11.2 (beta)
Here's more log info:
Event Module Code Server Node Time
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 10:00:18 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 15:00:18: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 15:00:18: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 15:00:20: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 10:00:18 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 10:00:17 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 15:00:17: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 15:00:17: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 15:00:19: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 10:00:17 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 10:00:13 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 15:00:13: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 15:00:13: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 15:00:15: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 10:00:13 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 10:00:12 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 15:00:12: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 15:00:12: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 15:00:14: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 10:00:12 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 10:00:08 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 15:00:08: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 15:00:08: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 15:00:10: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 10:00:08 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 10:00:07 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 15:00:07: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 15:00:07: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 15:00:09: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 10:00:07 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 10:00:03 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 15:00:03: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 15:00:03: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 15:00:05: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 10:00:03 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 10:00:02 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 15:00:02: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 15:00:02: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 15:00:04: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 10:00:02 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:59:58 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:59:58: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:59:58: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 15:00:00: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:59:58 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:59:57 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:59:57: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:59:57: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:59:59: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:59:57 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:59:53 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:59:53: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:59:53: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:59:55: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:59:53 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.02s
supervisor_cushion001 ns_1@10.212.146.111 09:59:52 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:59:52: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:59:52: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:59:54: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:59:52 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:59:48 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:59:48: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:59:48: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:59:50: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:59:48 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:59:47 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:59:47: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:59:47: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:59:49: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:59:47 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:59:43 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:59:43: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:59:43: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:59:45: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:59:43 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:59:42 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:59:42: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:59:42: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:59:44: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:59:42 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:59:38 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:59:38: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:59:38: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:59:40: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:59:38 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:59:37 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:59:37: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:59:37: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:59:39: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:59:37 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:59:33 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:59:33: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:59:33: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:59:35: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:59:33 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:59:32 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:59:32: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:59:32: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:59:34: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:59:32 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:59:28 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:59:28: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:59:28: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:59:30: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:59:28 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:59:27 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:59:27: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:59:27: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:59:29: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:59:27 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:59:22 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:59:23: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:59:23: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:59:25: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:59:22 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:59:22 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:59:22: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:59:22: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:59:24: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:59:22 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:59:18 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:59:17: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:59:17: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:59:19: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:59:18 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:59:17 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:59:17: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:59:17: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:59:19: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:59:17 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:59:12 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:59:12: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:59:12: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:59:14: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:59:12 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:59:12 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:59:12: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:59:12: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:59:14: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:59:12 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:59:07 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:59:07: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:59:07: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:59:09: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:59:07 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:59:07 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:59:07: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:59:07: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:59:09: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:59:07 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:59:02 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:59:02: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:59:02: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:59:04: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:59:02 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:59:02 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:59:02: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:59:02: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:59:04: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:59:02 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:58:57 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:58:57: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:58:57: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:58:59: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:58:57 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.02s
supervisor_cushion001 ns_1@10.212.146.111 09:58:57 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:58:57: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:58:57: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:58:59: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:58:57 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:58:52 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:58:52: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:58:52: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:58:54: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:58:52 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:58:52 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:58:52: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:58:52: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:58:54: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:58:52 - Mon Feb 20, 2012
Node 'ns_1@10.212.146.111' saw that node 'ns_1@10.88.207.174' went down. ns_node_disco005 ns_1@10.212.146.111 09:58:51 - Mon Feb 20, 2012
Node 'ns_1@10.90.145.211' saw that node 'ns_1@10.88.207.174' went down. ns_node_disco005 ns_1@10.90.145.211 09:58:51 - Mon Feb 20, 2012
Node 'ns_1@10.98.183.173' saw that node 'ns_1@10.88.207.174' went down. ns_node_disco005 ns_1@10.98.183.173 09:58:51 - Mon Feb 20, 2012
Node 'ns_1@10.40.177.223' saw that node 'ns_1@10.88.207.174' went down. ns_node_disco005 ns_1@10.40.177.223 09:58:51 - Mon Feb 20, 2012
Node 'ns_1@10.37.65.189' saw that node 'ns_1@10.88.207.174' went down. ns_node_disco005 ns_1@10.37.65.189 09:58:51 - Mon Feb 20, 2012
Node 'ns_1@10.100.243.14' saw that node 'ns_1@10.88.207.174' went down. ns_node_disco005 ns_1@10.100.243.14 09:58:51 - Mon Feb 20, 2012
Node 'ns_1@10.205.49.9' saw that node 'ns_1@10.88.207.174' went down. ns_node_disco005 ns_1@10.205.49.9 09:58:51 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:58:47 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:58:47: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:58:47: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:58:49: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:58:47 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:58:47 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:58:47: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:58:47: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:58:49: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:58:47 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:58:42 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:58:42: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:58:42: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:58:44: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:58:42 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.02s
supervisor_cushion001 ns_1@10.212.146.111 09:58:42 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:58:42: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:58:42: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:58:44: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:58:42 - Mon Feb 20, 2012
Couchbase Server has started on web port 8091 on node 'ns_1@10.88.207.174'. menelaus_sup001 ns_1@10.88.207.174 09:58:40 - Mon Feb 20, 2012
Node 'ns_1@10.90.145.211' saw that node 'ns_1@10.88.207.174' came up. ns_node_disco004 ns_1@10.90.145.211 09:58:40 - Mon Feb 20, 2012
Node 'ns_1@10.88.207.174' is leaving cluster. ns_cluster001 ns_1@10.88.207.174 09:58:40 - Mon Feb 20, 2012
Node 'ns_1@10.98.183.173' saw that node 'ns_1@10.88.207.174' came up. ns_node_disco004 ns_1@10.98.183.173 09:58:40 - Mon Feb 20, 2012
Node 'ns_1@10.40.177.223' saw that node 'ns_1@10.88.207.174' came up. ns_node_disco004 ns_1@10.40.177.223 09:58:38 - Mon Feb 20, 2012
Node 'ns_1@10.212.146.111' saw that node 'ns_1@10.88.207.174' came up. ns_node_disco004 ns_1@10.212.146.111 09:58:38 - Mon Feb 20, 2012
Node 'ns_1@10.37.65.189' saw that node 'ns_1@10.88.207.174' came up. ns_node_disco004 ns_1@10.37.65.189 09:58:38 - Mon Feb 20, 2012
Node 'ns_1@10.205.49.9' saw that node 'ns_1@10.88.207.174' came up. ns_node_disco004 ns_1@10.205.49.9 09:58:37 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:58:37 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:58:37: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:58:37: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:58:39: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:58:37 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:58:37 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:58:37: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:58:37: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:58:39: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:58:37 - Mon Feb 20, 2012
Node 'ns_1@10.100.243.14' saw that node 'ns_1@10.88.207.174' came up. ns_node_disco004 ns_1@10.100.243.14 09:58:37 - Mon Feb 20, 2012
Node 'ns_1@10.88.207.174' synchronized otp cookie qouhqbcugdhzjctj from cluster ns_node_disco002 ns_1@10.88.207.174 09:58:36 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:58:32 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:58:32: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:58:32: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:58:34: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:58:32 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:58:32 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:58:32: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:58:32: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:58:34: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:58:32 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:58:27 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:58:27: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:58:27: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:58:29: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:58:27 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:58:27 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:58:27: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:58:27: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:58:29: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:58:27 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:58:22 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:58:22: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:58:22: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:58:24: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:58:22 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:58:22 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:58:22: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:58:22: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:58:24: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:58:22 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:58:17 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:58:17: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:58:17: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:58:19: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:58:17 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:58:17 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:58:17: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:58:17: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:58:19: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:58:17 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:58:12 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:58:12: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:58:12: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:58:14: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:58:12 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:58:12 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:58:12: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:58:12: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:58:14: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:58:12 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:58:07 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:58:07: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:58:07: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:58:09: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:58:07 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:58:07 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:58:07: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:58:07: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:58:09: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:58:07 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:58:02 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:58:02: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:58:02: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:58:04: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:58:02 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:58:02 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:58:02: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:58:02: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:58:04: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:58:02 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:57:57 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:57:57: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:57:57: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:57:59: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:57:57 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:57:57 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:57:57: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:57:57: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:57:59: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:57:57 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
(repeated 1 times) supervisor_cushion001 ns_1@10.205.49.9 09:57:56 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
(repeated 1 times) supervisor_cushion001 ns_1@10.205.49.9 09:57:56 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:57:52 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:57:52: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:57:52: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:57:54: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:57:52 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:57:52 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:57:52: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:57:52: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:57:54: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:57:52 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:57:47: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:57:47: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:57:49: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:57:47 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:57:47 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:57:47: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:57:47: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:57:49: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:57:47 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:57:42 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:57:42: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:57:42: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:57:44: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:57:42 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:57:42 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:57:42: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:57:42: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:57:44: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:57:42 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:57:37 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:57:37: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:57:37: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:57:39: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:57:37 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.02s
supervisor_cushion001 ns_1@10.212.146.111 09:57:37 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:57:37: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:57:37: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:57:39: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:57:37 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:57:32: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:57:32: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:57:34: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:57:32 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:57:32 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:57:32: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:57:32: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:57:34: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:57:32 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:57:27 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:57:27: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:57:27: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:57:29: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:57:27 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:57:27 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:57:27: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:57:27: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:57:29: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:57:27 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:57:22 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:57:22: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:57:22: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:57:24: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:57:22 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:57:22 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:57:22: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:57:22: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:57:24: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:57:22 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:57:17 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:57:17: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:57:17: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:57:19: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:57:17 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:57:17 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:57:17: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:57:17: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:57:19: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:57:17 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:57:12 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:57:12: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:57:12: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:57:14: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:57:12 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:57:12 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:57:12: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:57:12: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:57:14: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:57:12 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:57:07 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:57:07: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:57:07: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:57:09: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:57:07 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:57:07 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:57:07: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:57:07: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:57:09: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:57:07 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:57:02 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:57:02: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:57:02: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:57:04: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:57:02 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:57:02 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:57:02: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:57:02: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:57:04: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:57:02 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:56:57 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:56:57: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:56:57: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:56:59: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:56:57 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.02s
supervisor_cushion001 ns_1@10.212.146.111 09:56:57 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:56:57: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:56:57: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:56:59: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:56:57 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:56:52 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:56:52: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:56:52: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:56:54: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:56:52 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.02s
supervisor_cushion001 ns_1@10.212.146.111 09:56:52 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:56:52: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:56:52: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:56:54: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:56:52 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:56:47: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:56:47: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:56:49: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:56:47 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:56:47 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:56:47: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:56:47: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:56:49: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:56:47 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:56:42 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:56:42: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:56:42: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:56:44: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:56:42 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:56:42 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:56:42: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:56:42: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:56:44: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:56:42 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:56:37 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:56:37: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:56:37: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:56:39: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:56:37 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:56:37 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:56:37: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:56:37: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:56:39: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:56:37 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:56:32 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:56:32: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:56:32: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:56:34: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:56:32 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:56:32 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:56:32: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:56:32: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:56:34: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:56:32 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:56:27 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:56:27: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:56:27: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:56:29: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:56:27 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:56:27 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:56:27: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:56:27: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:56:29: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:56:27 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:56:22 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:56:22: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:56:22: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:56:24: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:56:22 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:56:22 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:56:22: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:56:22: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:56:24: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:56:22 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:56:17 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:56:17: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:56:17: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:56:19: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:56:17 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:56:17 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:56:17: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:56:17: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:56:19: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:56:17 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:56:12 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:56:12: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:56:12: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:56:14: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:56:12 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:56:12 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:56:12: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:56:12: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:56:14: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:56:12 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:56:07 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:56:07: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:56:07: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:56:09: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:56:07 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:56:07 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:56:07: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:56:07: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:56:09: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:56:07 - Mon Feb 20, 2012
Starting rebalance, KeepNodes = ['ns_1@10.37.65.189','ns_1@10.40.177.223',
'ns_1@10.90.145.211','ns_1@10.98.183.173',
'ns_1@10.100.243.14','ns_1@10.205.49.9',
'ns_1@10.212.146.111'], EjectNodes = []
(repeated 1 times) ns_orchestrator004 ns_1@10.100.243.14 09:56:06 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:56:02 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:56:02: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:56:02: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:56:04: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:56:02 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.07s
supervisor_cushion001 ns_1@10.212.146.111 09:56:02 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:56:02: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:56:02: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:56:04: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:56:02 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:55:57 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:55:57: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:55:57: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:55:59: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:55:57 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.07s
supervisor_cushion001 ns_1@10.212.146.111 09:55:57 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:55:57: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:55:57: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:55:59: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:55:57 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:55:52 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:55:52: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:55:52: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:55:54: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:55:52 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:55:51 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:55:51: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:55:51: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:55:53: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:55:51 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:55:47 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:55:47: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:55:47: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:55:49: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:55:47 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:55:46 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:55:46: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:55:46: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:55:48: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:55:46 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:55:42: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:55:42: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:55:44: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:55:42 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:55:41 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:55:41: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:55:41: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:55:43: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:55:41 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:55:37 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:55:37: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:55:37: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:55:39: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:55:37 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:55:36 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:55:36: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:55:36: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:55:38: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:55:36 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:55:32 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:55:32: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:55:32: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:55:34: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:55:32 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.212.146.111' in 0.01s
supervisor_cushion001 ns_1@10.212.146.111 09:55:31 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.212.146.111' exited with status 1. Restarting. Messages: 2012-02-20 14:55:31: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:55:31: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:55:33: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.212.146.111 09:55:31 - Mon Feb 20, 2012
Service moxi exited on node 'ns_1@10.205.49.9' in 0.01s
supervisor_cushion001 ns_1@10.205.49.9 09:55:27 - Mon Feb 20, 2012
Port server moxi on node 'ns_1@10.205.49.9' exited with status 1. Restarting. Messages: 2012-02-20 14:55:27: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (13)
2012-02-20 14:55:27: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (10)
2012-02-20 14:55:29: (cproxy.c.330) ERROR: could not listen on port 11211. Please use -Z port_listen=PORT_NUM to specify a different port number. ns_port_server000 ns_1@10.205.49.9 09:55:27 - Mon Feb 20, 2012
That log has a very different message than before. Specifically, it looks like there's something else running on port 11211, so moxi can't listen there. You can use lsof to determine which process on the system is already listening on 11211.
See this in the log:
ERROR: could not listen on port 11211.
A moxi failure shouldn't cause autofailover. Are you certain the cluster is doing an auto failover?
Status 139 is a segfault. I just checked through the known issues and there aren't any about moxi on 1.8.0 with a segfault.
Questions:
- On which OS/Architecture is this?
- If Linux, do you have a core from moxi on the system? (search for files with core in the name owned by user "couchbase")
We may need to file an issue to have a look.
Documentation for Couchbase Server 1.8 is here:
http://www.couchbase.com/docs/couchbase-manual-1.8/index.html