SERVER_ERROR proxy write to downstream

It looks like there is a port confusion.  The failure is coming from port 11213 while you are connecting to port 11211?

It also takes just a few seconds for Moxi to startup a new bucket, is it possible that you caught it before it was ready?  You should be able to manually telnet to that port and type 'stats' to get the stats output which will verify that Moxi is up and running properly.

Perry

Sorry. With ports all Ok.

When I create Membase on different port, it opens in netstat. When I create Memcache, nothing heppend.

When I use port of Membase, it says this error? fbout proxy.

When I trying to connect to closed port, it says: Coudn't connect.

I think the problem with connecting to default bucket

When I creating Memcache default bucket

http://gyazo.com/b0c387dc1b489bee6aa5fc2565ac3a3b.png

> telnet localhost 11211

stats
SERVER_ERROR unauthorized, null bucket

------------------------------------

When I create Membase default bucket, it says:

SERVER_ERROR proxy write to downstream

This looks like a configuration problem, I'm just not sure exactly what's wrong.

Can you grab the output of http://<host>:8091/diag where <host> is the IP address/hostname of your server and attach the output?  You'll probably want to zip it first.

That should let me look at the whole picture and see what the problem might be.

Thanks, take care.

Perry

 To follow up, I just reproduced this and filed a bug to have it addressed as soon as possible.  The problem only seems to affect the default bucket, can you confirm that your tests work properly otherwise?

http://dl.dropbox.com/u/585441/ns-diag-20101014110510.zip

How do I confirm my tests? I think I tried all the options and the probability of my mistakes is minimal.

 The bug we have seems to only surface when deleting and recreating buckets.  We'll be fixing it shortly, but you should be able to workaround it by setting Membase up from scratch and not deleting/recreating the default bucket.

Obviously not a long-term solution, but should allow you to continue using it for now.  Let me know if that's not the case.

Perry

when i uninstall membase,reinstall and contiune setup step by step ,still SERVER_ERROR proxy write to downstream  ::(

 Sorry you're still having trouble.  Would you mind describing exactly the step-by-step process you are going through?

Thanks

Perry

helo,you could set this http://59.188.14.68:8091/index.html

i install it newly.please help me to set a membase 1024M 1copy

if i set a memcached server it would display:SERVER_ERROR unauthorized, null bucket

and my servers is win32 2003 ,every serval servers had this problem.

Reinstalling the application did not produce any effect.

Hi, we posted an updated binary over the weekend to fix this and another bug.  Can you re-download the package and let me know if this is resolved for you?

Thanks

Perry

Now works correctly. Thank you

UPDATE:

When I recreate default backet, the error returns: "Server localhost (tcp 11211) failed with: SERVER_ERROR unauthorized, null bucket (0)"

Membase backet on 11212 works correctly.

I am download the newest membase-server-community_x86_1.6.0.1_setup.exe ,still have the same problem SERVER_ERROR proxy write to downstream

 Thanks, I just reproduced this in the lab and have updated the bug.  

Perry

we have successfully installed the latest version of membase on windows XP and fedora but we are experiencing the same issue on

Linux 2.6.18-128.el5 #1 SMP Wed Dec 17 11:41:38 EST 2008

Red Hat Enterprise Linux Server release 5.3 (Tikanga)
 

is there an ETA for a fix

here is what we are seeing in the diagnostic trace:

2010-10-19 21:25:26.404 ns_node_disco:2:info:cookie update - Node 'ns_1@127.0.0.1' synchronized otp cookie lltwobsmluateigg from cluster

2010-10-19 21:25:26.504 menelaus_app:1:info:web start ok - Membase Server has started on web port 8091 on node 'ns_1@127.0.0.1'.

2010-10-19 21:25:26.531 ns_port_server:0:info:message - Port server memcached on node 'ns_1@127.0.0.1' exited with status 71. Restarting. Messages: failed to ensure corefile creation

2010-10-19 21:25:26.534 supervisor_cushion:1:warning:port crashed too soon after restart - Service memcached crashed on node 'ns_1@127.0.0.1' in 0.01s

 Issue was due to the following setting in /etc/security/limits.conf

* hard core 0

When membase starts memcached it uses the -r parameter : "maximize core file limit". memcached tries to change the core file size limit and when it fails the service exits. So membase would be running without memcached.

 Thanks drvolks, is this resolution in response to the post directly above?  Just want to make sure everyone is all set since the issue and its resolution are coming from two different usernames...

I'll be adding this to the troubleshooting section of the wiki.  

Perry

 You are right, solracson is my colleague. The symptoms were:

Using the stats command on telnet localhost 11211:

SERVER_ERROR proxy write to downstream

In the web console log we were seeing:

Port server memcached on node 'ns_1@127.0.0.1' exited with status 71. Restarting. Messages: failed to ensure corefile creation They were caused by the hard code limit setted to 0. 

 Hi,

I installed membase from sources (membase-server-community_1.6.0.1_src.tar.gz).

At first I had problem 

Service memcached crashed on node. exited with code 71 (port address 11210 is already in use). But there is nothing on 11210 port. and I can manually start memcache on that port (/usr/bin/memcached -p 11210)

And for telnet localhost 11211 I have

set some_greeting 0 0 5

hello

SERVER_ERROR proxy write to downstream

  Do you have any ideas/tips?

Sorry for getting back to you so late on this...somehow the post got lost in the shuffle.

Can you attach the output of http://<host>:8091/diag where <host> is the Membase server?

This may be related to a known issue but I want to make sure.

Perry

Just went through this with my setup. Not sure if it applies as mine is kind of odd but here it is.

I was getting the same error with a 3 node cloud based system and after pairing it down to one node it turned out to be the fact I could not connect to the external address.

ie assume the server is a cloud based server with an external address unknown to it.

I went through the instructions here http://wiki.membase.org/display/membase/Using+Membase+in+the+Cloud to get the cluster working.

lets say the external address is 1.2.3.4 so - have -name ns_1@1.2.3.4

my internal addresses are 192.168.1.1 and 127.0.0.1

connecting to 192.168.1.1 or 127.0.0.1 on port 11211 and trying to set anything gave the downstream error.

adding an iptables rule like this cured it for me.

iptables -t nat -A OUTPUT  -d 1.2.3.4 -j DNAT --to-destination 127.0.0.1

Chatty

  hey,

 i want to join chatty on the same issue. installed membase 1.6.0.1, the web console loaded great on <DNS name>:8091. but when i logged in to the machine and telnet 127.0.0.1 11211, and hit "stats" i got SERVER_ERROR proxy write to downstream what am i doing wrong?   tal.

 This is a known issue in 1.6.0.1 and I believe we've fixed it in the next release.  Can you please download the RC2 of 1.6.4 (http://forums.membase.org/thread/membase-server-164-rc2-available) and let me know if this is resolved?  If not, please send the output of /opt/membase/bin/ns_server/collect_info <output_file> so we can look at the logs.

Thanks!

Perry

 thanks Perry.

it seems to be solved just by shutting does the membase-server and relaunching it.

however, I encountered another problem. i run a "membase" type bucket but cant manage to use the "standard port" as it require authentication.

in other words, when i run a standard 11211 port, i can connect neither from the telnet nor using from the PHP driver.

the only way i managed to do it was using a dedicated port (9990 for ex).

is there a way that i'm missing which will allow me to authenticate with the PHP driver? (i'm using $membase->addServer(ip,port)).

thanks so much.

tal.

Hi!

I have the same problem. When I installed the Membase Server 1.6.4.1 into my Windows XP (x86). I got the same error using telnet localhost 11211. 

C:\>telnet localhost 11211
stats
SERVER_ERROR proxy write to downstream

I have installed  Membase Server into 2 different computers (WinXP and WinXP). If I stop Membase service and start old Memcache, my code works correctly (so, problem not in PHP and its Memcache extenstion).  

Deleting bucket and creating new don't change anything. Creating a bucket for another port (11222) can't solve problem.

 $memcache->addServer(); works
 $memcache->getVersion(); works
 $memcache->set(args); does not work

After creating memcached bucket:

 $memcache->addServer(); works
 $memcache->getVersion(); does not work
 $memcache->set(args); does not work
 

Hi Perry,

I was trying to start memcached.exe manually.  It looks like memcached.exe unable to started successfully cause this problem.

C:\Program Files\Membase\Server\bin\memcached>memcached.exe
Can't create notify pipe: Result too large

Hope this message helps!

Thank you.

Hi -- I've just created jira issue MB-3253 to track this.

To help us diagnose this more, by the way, do you have any issues when trying to run moxi.exe manually?  eg...

  C:\Program Files\Membase\Server\bin\moxi>moxi.exe

Thanks,

Steve

 Hi,

An update on MB-3253 -- there was a 32-bit windows build issue that's been addressed, and a new 32-bit windows build of 1.6.4.1 has been posted for you to download.  Please see the download links off of membase.com or membase.org.  Also, my apologies for any inconvenience to folks!

Cheers,

Steve

Hi Steve,

Why are you using the same version number (1.6.4.1) as a newly build?  Shouldn't you use the new version number such as 1.6.4.2 as a bug fixed build?  Using the same version number for different build may confusing someone.  It's just my tiny suggestion.

Hi, we had a similar debate here in the engineering pit.  Turns out there were no source-code changes, no new git tag, but that only the 32-bit windows builder went through a build hiccup that we unfortunately didn't catch.  All the other builders in the farm (64-bit windows, redhat, ubuntu, ...) were fine and unchanged, so we made the call to just hit the rebuild button on the 32-bit windows builder and hope that not too much confusion would ensue.

Thanks for the note!

Steve

 Wanted to follow up here...

willh and insbrook, has this resolved any issues you were seeing?  

I'd like to close this particular thread since it is a bit too long and irrelevant to be useful (unless of course the problem's NOT solved...)

Thanks, take care.

Perry

 Tal, so sorry for the delay, I'm not sure why this post got lost in the shuffle.

Anyway, you can only connect unauthenticated to the default bucket on port 11211.  The bucket must be named 'default'.  If you want to connect (unauthenticated) to multiple buckets, you will have to have them each reside on separate ports (and different than 11211).  

Hope that clears things up.

Perry

Hi Perry,

The x86 problem has been solved.

Thank you.

 Great.  I'm going to close this thread and hope that anyone will open a new one if the problem persists or resurfaces.

Thanks all

Perry