Unable To Create Bucket

I've seen that error before when there are nodes down in the cluster (or when it thinks there are...)

How many nodes are in your cluster and do any show down right now? Can you give me the diag output from each of the nodes (http://:8080/diag)

Thanks

Perry

There are currently two nodes in the cluster neither of which are down. Diag attached.

Thanks, can you grab the diag from the other node as well?

Diag 2 attached.

Thanks for those. The diagnosis is showing that neither of the servers have any memory available. Obviously that's not the case...

Is this a new install or did you have it working for sometime and then this happened?

What specific OS are you running on? Also, can you try running these as a full admin user to eliminate any weird permissions issues? The problem really here is that we are asking the OS how much memory it has and we are getting back '0'.

Thanks

Perry

This is a new install.

Running Windows Server 2003 Service Pack 2 and I am a full administrator on this box.

Can you try removing one node from the cluster and verifying that each node sees it's own memory properly?

Removing one node from the cluster still results in negative memory available.

Is it negative on both of them or only one?

It is negative on both.

[B]Server 1:[/B] Total Cache Size/-67108864B
[B]Server 2:[/B] Total Cache Size/-67108864B

Hmm, this certainly is quite strange. I'm going to get our engineering team engaged...can you try reinstalling, or even installing fresh on a new system just to test it out?

I'll get back to you with some more information.

Thanks.

Perry

We have it running on many different machines already. These are our test middle tier servers that this is going on. This is the first time we've encountered this. My fear is that we are going to run into this when we install on production.

Sorry, I remember you mentioning that now...lots of things running around in my small brain.

I'd like to be able to reproduce this in house but have been unable to so far. Can you help me understand what makes these two machines different?

Thanks!

What would you like to know? The problematic servers are VMWare virtual servers running Windows Server 2003 SP2 with 4GB RAM per.

Thanks. Is that the same configuration as the non-problematic ones?

The non-problematic ones are running Hyper-V with the same version of Windows Server 2003 SP2 32-bit. We also have it running on our host machines which are Windows Server 2008 64-bit with 4GB memory.

Okay, let's get down and dirty and check out what the OS is really reporting.

On both a working, and a non-working machine can you perform the following:
1) Browse to C:\Program Files (x86)\NorthScale\Memcached Server\bin\erlang\bin (if you didn't install it in the default location, go to where you installed the Memcached server and go into bin\erlang\bin)
2) Open "erl.exe"
3) Type:
application:start(sasl).
application:start(os_mon).
memsup:get_system_memory_data().

Make sure you put a '.' at the end of each line, this is what tells erlang to execute the line.

Past the output and we'll have a look...

Thanks!

Perry

Screenshots attached.

Do you have Windows Firewall turned on?

Also, can you verify that this file exists: C:/Program Files/NorthScale/Developer Kit/bin/erlang/lib/os_mon-2.2.4/priv/bin/win32sysinfo.exe and that you can open/run it? (there won't be anything to get the ouput of, just make sure it runs)

No. Windows Firewall is turned off on both machines.

While the firewall matters, the erlang terminology for "port" here refers to starting and communicating with a subprocess, not the use of TCP/UDP ports.

Thanks dgottlieb, how about checking on the availability of that file that I pointed out? That's where Erlang is failing...

Thanks

Ah. Sorry about that. I missed that response in the thread. See attached.

No worries!

That's the expected behavior, but I'm still not sure why our Erlang process is unable to open a connection to that file, and why I haven't been able to reproduce it here in house. Seems like I should be able to. I'm running Windows 2003 R2 SP2 on a Vmware system...

Are these machines used for anything else at the moment? Would it be possible to start from scratch (either with these or with a new install) and see if we have the same issue? That way I can follow your steps exactly. It's possible that another application is interfering here...do you have any antivirus installed on these?

Thanks!

Perry

We do have antivirus running on these machines (Vipre Enterprise by Sunbelt Software). As far as a rebuild of these machines, that's not going to be possible at this time. We have installed this on other 2003 environments just as these without a problem.

Okay, good to know. As a test, could we disable the AV temporarily and restart the memcached service?

Okay, so starting our analysis from scratch (and thanks to Matt for his assistance) we've noticed some inconsistencies that are either the cause of the problem or at least throwing off our analysis.

Looking at the screenshot of the os_mon crash, it is unable to find a file at this path: "c:/PROGRA~1/NorthScale/Developer Kit/bin/erlang/lib/os_mon-2.2.4/priv/bin/win32sysinfo.exe" Looking at the screenshot of you running that file, the path is: "C:\Program Files (x86)\NorthScale\Memcached Server\bin\erlang\lib\os_mon-2.2.4\priv\bin\win32sysinfo.exe".

Two pieces of confusion:
-It appears that you installed the 32-bit version of our server on a 64-bit box. No big deal here, it should work fine but you may not have wanted to do that.
-The first path is looking in the "Developer Kit" directory, yet the second path is not. Do you know whether you downloaded the developer kit or the regular server? It appears that the server thinks it's running as the developer kit, but that it may not be installed as such. Sorry to go through this again, but can you uninstall NorthScale (and the developer kit if it's there), then make sure you've manually deleted any NorthScale directories under either "Program Files" or "Program Files (x86)" and reinstall?

Thanks, I hope that does it. BTW, at this point I doubt the AV software is having an adverse effects.

Perry

Due to restrictions on these boxes and how antivirus is set up, we can't disable it. It's a network thing. Note, however, that this same antivirus software is running on all the other machines running Northscale successfully.

Ah, now I've seen your latest posting. I'm re-installing the 32-bit version (these are 32-bit boxes). NOT developer kit but the server. I'll be back with results. Fingers crossed!

Understood, let's track down the path issues.

Let me know if you'd like to setup a screen sharing session, I can certainly make myself available to diagnose this live with you.

Perry

After a re-install of 32bit Memcached Server, I am still seeing the same results. I'm at a loss.

Can we setup a screen sharing session and I'll take a look live?

Yeah. That's fine. Contact me through email for details, please.

To recap for everyone's benefit...we found the problem here.

Apparently there is a bug in at least one version of Windows Server 2003 that does not handle spaces in path names very well. We were installing to C:\Program Files\NorthScale\Memcached Server and changing that path to C:\Program Files\NorthScale\MemcachedServer resolved the issue.

We'll be running this down in house, but there is a valid (and not too hard) workaround.

Thanks for your help and patience Dave.

Perry