supervisor_cushion:1:warning:port exited too soon after restart - Service memcached exited on node 'ns_1@172.16.69.29'
We are running two clustered production instances of membase servers. One of the servers reports errors and cannot start. The log is filled with errors like these below.
Do you have any ideas which might be the source for these crashes?
Regards,
Robert
2011-04-27 17:07:15.339 supervisor_cushion:1:warning:port exited too soon after restart - Service memcached exited on node 'ns_1@172.16.69.29' in 0.00s
2011-04-27 17:07:20.345 supervisor_cushion:1:warning:port exited too soon after restart - Service memcached exited on node 'ns_1@172.16.69.29' in 0.00s
2011-04-27 17:07:25.351 supervisor_cushion:1:warning:port exited too soon after restart - Service memcached exited on node 'ns_1@172.16.69.29' in 0.00s
2011-04-27 17:07:29.219 ns_port_server:0:info:message - Port server memcached on node 'ns_1@172.16.69.29' exited with status 139. Restarting. Messages: (repeated 71 times)
2011-04-27 17:07:29.219 supervisor_cushion:1:warning:port exited too soon after restart - Service memcached exited on node 'ns_1@172.16.69.29' in 0.00s
(repeated 1 times)
2011-04-27 17:07:30.356 ns_port_server:0:info:message - Port server memcached on node 'ns_1@172.16.69.29' exited with status 139. Restarting. Messages:
2011-04-27 17:07:30.356 supervisor_cushion:1:warning:port exited too soon after restart - Service memcached exited on node 'ns_1@172.16.69.29' in 0.00s
2011-04-27 17:07:30.515 ns_orchestrator:4:info:message - Starting rebalance, KeepNodes = ['ns_1@172.16.69.29','ns_1@172.16.69.30'], EjectNodes = []
2011-04-27 17:07:35.362 supervisor_cushion:1:warning:port exited too soon after restart - Service memcached exited on node 'ns_1@172.16.69.29' in 0.00s
2011-04-27 17:07:40.367 supervisor_cushion:1:warning:port exited too soon after restart - Service memcached exited on node 'ns_1@172.16.69.29' in 0.00s
INFO REPORT <0.10793.34> 2011-05-10 21:34:43
===============================================================================
Cushion managed supervisor for memcached failed: {abnormal,139}
INFO REPORT <0.10793.34> 2011-05-10 21:34:43
===============================================================================
ns_log: logging supervisor_cushion:1:Service memcached exited on node 'ns_1@172.16.69.29' in 0.00s
ERROR REPORT <0.6524.0> 2011-05-10 21:34:44
===============================================================================
ns_1@172.16.69.29:ns_memcached:374: Unable to connect: {error,
{badmatch,
{error,
econnrefused}}}, retrying.
SUPERVISOR REPORT <0.100.0> 2011-05-10 21:34:44
===============================================================================
Reporting supervisor {local,menelaus_sup}
Child process
errorContext child_terminated
reason {noproc,{gen_server,call,['ns_memcached-default',topkeys,30000]}}
pid <0.10792.34>
name hot_keys_keeper
start_function {hot_keys_keeper,start_link,[]}
restart_type permanent
shutdown 5000
child_type worker
PROGRESS REPORT <0.100.0> 2011-05-10 21:34:44
===============================================================================
supervisor {local,menelaus_sup}
started
[{pid,<0.10796.34>},
{name,hot_keys_keeper},
{mfa,{hot_keys_keeper,start_link,[]}},
{restart_type,permanent},
{shutdown,5000},
{child_type,worker}]
ERROR REPORT <0.6524.0> 2011-05-10 21:34:45
===============================================================================
ns_1@172.16.69.29:ns_memcached:374: Unable to connect: {error,
{badmatch,
{error,
econnrefused}}}, retrying.
ERROR REPORT <0.6524.0> 2011-05-10 21:34:46
===============================================================================
ns_1@172.16.69.29:ns_memcached:374: Unable to connect: {error,
{badmatch,
{error,
econnrefused}}}, retrying.
ERROR REPORT <0.6524.0> 2011-05-10 21:34:47
===============================================================================
ns_1@172.16.69.29:ns_memcached:374: Unable to connect: {error,
{badmatch,
{error,
econnrefused}}}, retrying.
ERROR REPORT <0.6524.0> 2011-05-10 21:34:48
===============================================================================
ns_1@172.16.69.29:ns_memcached:374: Unable to connect: {error,
{badmatch,
{error,
econnrefused}}}, retrying.
ERROR REPORT <0.10793.34> 2011-05-10 21:34:48
===============================================================================
** Generic server <0.10793.34> terminating
** Last message in was {die,{error,cushioned_supervisor,{abnormal,139}}}
** When Server state == {state,memcached,5000,{1305,52483,203344},undefined}
** Reason for termination ==
** {error,cushioned_supervisor,{abnormal,139}}
CRASH REPORT <0.10793.34> 2011-05-10 21:34:48
===============================================================================
Crashing process
initial_call {supervisor_cushion,init,['Argument__1']}
pid <0.10793.34>
registered_name []
error_info
{exit,{error,cushioned_supervisor,{abnormal,139}},
[{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}]}
ancestors [ns_port_sup,ns_server_sup,ns_server_cluster_sup,<0.52.0>]
messages []
links [<0.105.0>]
dictionary []
trap_exit true
status running
heap_size 377
stack_size 24
reductions 571
SUPERVISOR REPORT <0.105.0> 2011-05-10 21:34:48
===============================================================================
Reporting supervisor {local,ns_port_sup}
Child process
errorContext child_terminated
reason {error,cushioned_supervisor,{abnormal,139}}
pid <0.10793.34>
name
{memcached,"./bin/memcached/memcached",
["-X","./bin/memcached/stdin_term_handler.so","-p",
"11210","-E","./bin/bucket_engine/bucket_engine.so","-B",
"binary","-r","-c","10000","-e",
"admin=_admin;default_bucket_name=default;auto_create=false",
[]],
[{env,[{"EVENT_NOSELECT","1"},
{"MEMCACHED_TOP_KEYS","100"},
{"ISASL_PWFILE",
"/var/opt/membase/1.6.5/data/ns_1/isasl.pw"},
{"ISASL_DB_CHECK_TIME","1"}]},
use_stdio,stderr_to_stdout,stream]}
start_function
{supervisor_cushion,start_link,
[memcached,5000,ns_port_server,start_link,
[memcached,"./bin/memcached/memcached",
["-X","./bin/memcached/stdin_term_handler.so",
"-p","11210","-E",
"./bin/bucket_engine/bucket_engine.so","-B",
"binary","-r","-c","10000","-e",
"admin=_admin;default_bucket_name=default;auto_create=false",
[]],
[{env,[{"EVENT_NOSELECT","1"},
{"MEMCACHED_TOP_KEYS","100"},
{"ISASL_PWFILE",
"/var/opt/membase/1.6.5/data/ns_1/isasl.pw"},
{"ISASL_DB_CHECK_TIME","1"}]},
use_stdio,stderr_to_stdout,stream]]]}
restart_type permanent
shutdown 10
child_type worker
It looks like the memcached process is exiting unexpectadly, but the logs don't contain the full messages.
Can you run this command and sent the to perry -at- couchbase -dot- com?
/opt/membase/bin/ns_server/collect_info
Forum support is great for free but sometimes you need a guaranteed response time and dedicated resources for your questions or issues.
Consider purchasing enterprise-level support from Couchbase: http://www.couchbase.com/products-and-services/overview
Call or email "sales -at- couchbase-dot- com" today!