SERVER_ERROR proxy write to downstream : Error while telnet
Hi,
I am using rpm of membase 1.6.4 membase-server-enterprise_x86_64_1.6.4.1.rpm. I have tested this in various redhat version. But when i tried to work with redhat EL5 kernal 2.6.18-238.5.1.el5 i am getting a strange issue.
I have done the below steps.
1. Installed the membase rpm
2. Created default bucket with membase option and replica as 1
3. restarted membase
4. telnet localhost 11211
I am getting the error
]# telnet localhost 11211
Trying 127.0.0.1...
Connected to localhost.localdomain (127.0.0.1).
Escape character is '^]'.
stats
SERVER_ERROR proxy write to downstream
===============================================================================
While checking the logs i am getting below error logs
INFO REPORT <5649.214.0> 2011-04-11 02:27:42
===============================================================================
ns_log: logging ns_port_server:0:Port server memcached on node 'ns_1@127.0.0.1' exited with status 139. Restarting. Messages:
ERROR REPORT <5649.214.0> 2011-04-11 02:27:42
===============================================================================
** Generic server <5649.214.0> terminating
** Last message in was {#Port<5649.2420>,{exit_status,139}}
** When Server state == {state,#Port<5649.2420>,memcached,
{[empty],[empty,empty]},
undefined,[],0}
** Reason for termination ==
** {abnormal,139}
CRASH REPORT <5649.214.0> 2011-04-11 02:27:42
===============================================================================
Crashing process
initial_call {ns_port_server,init,['Argument__1']}
pid <5649.214.0>
registered_name []
error_info
{exit,{abnormal,139},
[{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}]}
ancestors
[<5649.213.0>,ns_port_sup,ns_server_sup,ns_server_cluster_sup,
<5649.53.0>]
messages [{'EXIT',#Port<5649.2420>,normal}]
links [<5649.213.0>]
dictionary []
trap_exit true
status running
heap_size 610
stack_size 24
reductions 731
INFO REPORT <5649.213.0> 2011-04-11 02:27:42
===============================================================================
Cushion managed supervisor for memcached failed: {abnormal,139}
INFO REPORT <5649.213.0> 2011-04-11 02:27:42
===============================================================================
ns_log: logging supervisor_cushion:1:Service memcached exited on node 'ns_1@127.0.0.1' in 0.01s
INFO REPORT <5649.71.0> 2011-04-11 02:27:42
===============================================================================
ns_log: suppressing duplicate log ns_port_server:0("Port server memcached on node 'ns_1@127.0.0.1' exited with status 139. Restarting. Messages: ") because it's been seen 4 times in the past 20.039507 secs (last seen 5.007965 secs ago
ERROR REPORT <5649.111.0> 2011-04-11 02:27:43
===============================================================================
ns_1@127.0.0.1:ns_memcached:374: Unable to connect: {error,
{badmatch,
{error,econnrefused}}}, retrying.
INFO REPORT <5649.215.0> 2011-04-11 02:27:44
===============================================================================
ns_1@127.0.0.1:ns_janitor:269: Waiting for "default" on ['ns_1@127.0.0.1']
ERROR REPORT <5649.111.0> 2011-04-11 02:27:44
===============================================================================
ns_1@127.0.0.1:ns_memcached:374: Unable to connect: {error,
{badmatch,
{error,econnrefused}}}, retrying.
INFO REPORT <5649.215.0> 2011-04-11 02:27:45
===============================================================================
ns_1@127.0.0.1:ns_janitor:269: Waiting for "default" on ['ns_1@127.0.0.1']
ERROR REPORT <5649.111.0> 2011-04-11 02:27:45
===============================================================================
ns_1@127.0.0.1:ns_memcached:374: Unable to connect: {error,
{badmatch,
{error,econnrefused}}}, retrying.
*********************************************************************************************************
Please help me out with this issue
Thanks,
Pradeep
Hi Perry,
First of all thanks for your reply. I will mention the questions you have asked,
1. I am using swap memory
2. I have verified the /var/log/messges, i am getting below error while starting membase
kernel: memcached[6608]: segfault at ffffffffff700124 rip 00002af130e8929f rsp 00007fff13122590 error 4
kernel: memcached[6609]: segfault at ffffffffff700124 rip 00002ac006b0229f rsp 00007fffcdb7d9f0 error 4
kernel: memcached[6610]: segfault at ffffffffff700124 rip 00002b23a44b529f rsp 00007fff938697c0 error 4
kernel: memcached[6611]: segfault at ffffffffff700124 rip 00002ae82200329f rsp 00007ffff6c44b20 error 4
kernel: memcached[6612]: segfault at ffffffffff700124 rip 00002af2b801a29f rsp 00007fffe3411190 error 4
kernel: memcached[6631]: segfault at ffffffffff700124 rip 00002adbfe91829f rsp 00007fff9323d340 error 4
3. I tried with the latest version also, still i am getting the same error. Still i will use the latest verion
Thanks,
Pradeep
Thanks Pradeep.
-Are there any core files being created? They should be under /opt/membase I believe. If not, you can 'chown membase:membase /opt/membase' and it should begin putting any corefiles there (this is fixed in the upcoming 1.7). Once you have a core file, can you run the /opt/membase/bin/ep_engine/management/analyze_core against it and post the results?
-Can you send me the output of '/opt/membase/bin/ns_server/collect_info ' (send to perry -at- couchbase -dot- com)
Thank you
Perry
A few things here Pradeep...I would really recommend upgrading to the latest version of Membase (1.6.5.3) to avoid any known issues either here or farther down the line.
Secondly, it looks like memcached is exiting with status 139 which usually means the Linux OOM killer is coming along and shutting down the process. Can you look through the /var/log/messages to see if there is any indication this is happening? Do you have any swap configured on the system?
Perry
Forum support is great for free but sometimes you need a guaranteed response time and dedicated resources for your questions or issues.
Consider purchasing enterprise-level support from Couchbase: http://www.couchbase.com/products-and-services/overview
Call or email "sales -at- couchbase-dot- com" today!