SERVER_ERROR proxy write to downstream

 Hi Adverb, sorry to hear you're having trouble.

That error is generated when the proxy process can't connect to the underlying server and is usually caused when the server itself is either unavailable or constantly restarting (what I suspect here).

Can you look through the output of /opt/membase/bin/browse_logs for any errors standing out? You should be able to search for 'memcached' and find the messages that it is spitting out when it exits.

Perry

 Hi Perry,

Thanks for a quick response !!!

I didn't install from the binary package. I took the source and compiled it.

I don't seem to have the file (/opt/membase/bin/browse_logs) that you have mentioned. 

Where can I look for the logs in this case ? [In the logs directory under ns_server directory, I see some data files that are unreadable.

Here are the processes that run:

ubuntu:/opt/vts/membase/membase-server-community_1.6.5_src/ns_server/logs> ps axjf | grep 16547 16545 16547 16545  1635 pts/0    16545 Sl+   1000   0:05  |           \_ /usr/lib/erlang/erts-5.7.4/bin/beam.smp -A 16 -- -root /usr/lib/erlang -progname erl -- -home /home/bhalaji -- -pa ./deps/menelaus/deps/erlwsh/ebin ./deps/menelaus/deps/mochiweb/ebin ./deps/menelaus/ebin ./deps/gen_smtp/ebin ./ebin -setcookie nocookie -run ns_bootstrap -ns_server error_logger_mf_dir "logs" -ns_server error_logger_mf_maxbytes 10485760 -ns_server error_logger_mf_maxfiles 10 -kernel inet_dist_listen_min 21100 inet_dist_listen_max 21199 -- -noshell16547 16594 16594 16594 ?           -1 Ss    1000   0:00  |               \_ sh -s disksup16547 16595 16595 16595 ?           -1 Ss    1000   0:00  |               \_ /usr/lib/erlang/lib/os_mon-2.2.4/priv/bin/memsup16547 16597 16597 16597 ?           -1 Ss    1000   0:00  |               \_ /usr/lib/erlang/lib/os_mon-2.2.4/priv/bin/cpu_sup16547 16598 16598 16598 ?           -1 Ssl   1000   0:00  |               \_ ./bin/moxi/moxi -Z port_listen=11211,default_bucket_name=default,downstream_max=1024,downstream_conn_max=4,connect_max_errors=5,connect_retry_interval=30000,connect_timeout=400,auth_timeout=100,cycle=200,downstream_conn_queue_timeout=200,downstream_timeout=5000,wait_queue_timeout=200 -z url=http://127.0.0.1:8091/pools/default/saslBucketsStreaming -p 0 -Y y -O stderr 16547 16607 16607 16607 ?           -1 Ss    1000   0:00  |               \_ inet_gethost 412894 16875 16874  1704 pts/1    16874 S+    1000   0:00          \_ grep 16547---------------------------------------------------I used start.sh to start the server and the process tree is as below:start.sh───beam.smp─┬─cpu_sup                    ├─inet_gethost───inet_gethost                    ├─memsup                    ├─moxi───7*[{moxi}]                    ├─sh                    └─22*[{beam.smp}]  regardsAbverb 

 Yeah, it looks like there is no memcached process running, so that would definitely be part of the problem here.

The browse_logs script may be in a different path...do you have a 'bin' directory underneath where you installed into?

 Hi Perry,

This is what is there in the bin directory.

ubuntu:/opt/vts/membase/membase-server-community_1.6.5_src/ns_server/bin> ls -ltrtotal 20drwxr-xr-x 2 abverb abverb 4096 2011-01-28 13:36 vbucketmigratordrwxr-xr-x 2 abverb abverb 4096 2011-02-02 11:14 moxidrwxr-xr-x 2 abverb abverb 4096 2011-02-02 11:14 memcacheddrwxr-xr-x 2 abverb abverb 4096 2011-02-02 11:14 ep_enginedrwxr-xr-x 2 abverb abverb 4096 2011-02-02 11:14 bucket_engine Do I need to start the memcached process separately ?  regardsAbverb

Hi Perry,

This is what is there in the bin directory:

ubuntu:/opt/vts/membase/membase-server-community_1.6.5_src/ns_server/bin> ls -ltr
total 20
drwxr-xr-x 2 abverb abverb 4096 2011-01-28 13:36 vbucketmigrator
drwxr-xr-x 2 abverb abverb 4096 2011-02-02 11:14 moxi
drwxr-xr-x 2 abverb abverb 4096 2011-02-02 11:14 memcached
drwxr-xr-x 2 abverb abverb 4096 2011-02-02 11:14 ep_engine
drwxr-xr-x 2 abverb abverb 4096 2011-02-02 11:14 bucket_engine

Do I need to start the memcached process separately ?

regards
Abverb

 You shouldn't need to start it manually, no.

Can you do a 'find / -name browse_logs'?

Perry

Hi Perry,

Here is what I see:

ubuntu:/opt/vts/membase/membase-server-community_1.6.5_src> find . -name "browse_logs" -print
./ns_server/browse_logs
ubuntu:/opt/vts/membase/membase-server-community_1.6.5_src> cat ns_server/bro
browse_logs* browse_logs_cluster*
ubuntu:/opt/vts/membase/membase-server-community_1.6.5_src> cat ns_server/browse_logs
#!/bin/sh
# Copyright (c) 2010, NorthScale, Inc.
# All rights reserved.
cd `dirname $0`
exec erl -pa ebin -boot start_sasl -sasl errlog_type error -noshell -run ns_log_browser -report_dir logs -run init stop -- "$@"
ubuntu:/opt/vts/membase/membase-server-community_1.6.5_src> file ns_server/bro
browse_logs* browse_logs_cluster*
ubuntu:/opt/vts/membase/membase-server-community_1.6.5_src> file ns_server/browse_logs
ns_server/browse_logs: POSIX shell script text executable
ubuntu:/opt/vts/membase/membase-server-community_1.6.5_src>

regards
Abverb

 Now you should just be able to ./browse_logs and it will print out the logging information.  We store our logs in a binary format for better on-disk compression so you can't view them directly.

Feel free to zip up the output of that file and send it to perry -at- membase -dot- com and I'll take a look.

Perry

Hi Perry,

I see the following message suggesting to run as root in the log output:

ERROR REPORT <5676.103.0> 2011-02-02 16:17:59
===============================================================================

** Generic server <5676.103.0> terminating
** Last message in was {#Port<5676.1820>,{exit_status,71}}
** When Server state == {state,#Port<5676.1820>,memcached,
{["failed to set rlimit for open files. Try running as root or requesting smaller maxconns value.",
empty],
[empty]},
{ok,{1296692280431878,#Ref<5676.0.0.417>}},
["failed to set rlimit for open files. Try running as root or requesting smaller maxconns value."],
0}
** Reason for termination ==
** {abnormal,71}

But when I run as root, the moxi process also crashes with the following message:

Port server moxi on node 'ns_1@127.0.0.1' exited with status 64. Restarting. Messages: 2011-02-02 16:25:16: (memcached.c.4767) can't run as root without the -u switch

regards
Abverb

 Yes, unfortunately you can't run it as root...I think we should change that error message since its recommendation is not very useful ;-)

The problem is the rlimit on your system.  Take a look at this wiki page and let me know if you have any other questions on how to resolve: http://wiki.membase.org/display/membase/Common+Errors

Perry

Hi Perry,

Thank you very much !!!

I noticed that memcached was trying to set the no. of open files as 10000.
Everything started working after I increased the hard limit of open files.

regards
Abverb

 I'm also getting this exact same error, except I'm just running the Community version of membase on Windows XP for development testing.  I seem to get this error whenever I hibernate my machine and I wake it up in a place where it gets a new IP address.  In those scenarios, if I telnet to localhost with 11211, then call stats or whatever, I get the aforementioned "SERVER_ERROR proxy write to downstream" error.

I did notice, however, that in my

C:\Program Files\Membase\Server

directory, that I saw multiple folders with names like

Mnesia.ns_1@xxx.xxx.xxx.xxxMnesia.ns_1@yyy.yyy.yyy.yyy Where xxx.xxx.xxx.xxx and yyy.yyy.yyy.yyy represented 2 valid IP addresses that my machine had.  I was able to get things back up and running by just using the service_*.bat files in the bin directory.  However, I did notice that ip_addr.bat seems to be in charge of figuring out the IP address of the machine when doing the service registration.  So what I'm gonna try is replacing the content of that batch file with just: echo 127.0.0.1 to see if I can't get a consistent IP.  Hopefully that's not a horrible idea :-).

 You'll want to take a look at this wiki entry to bind Membase to localhost but keep in mind that this won't allow it to cluster to other nodes...you'll need static IP's for that but I don't think this will be a problem for you now

http://wiki.membase.org/display/membase/Using+Membase+in+the+Cloud