Service memcached exited on node 'ns_1@IP' [windows Version: 1.7.0]
Dear all
I am using membase server version 1.7.0 on my windows 2003 machine. I use memcached just a single bucket. Server is not overloaded and performance seems fine. Till today everything worked fine. Now I cannot access server because it is dropping every connection on port 11211. I tried to delete and create new bucket but it didn't help. I tried to run mbcollect_info.exe but it seems that it hangs. Output:
C:\Program Files\NorthScale\Memcached Server\bin>mbcollect_info.exe hang.zip System information (systeminfo) - OK Computer system (wmic computersystem) - OK Computer OS (wmic os) - OK Service list (wmic service where state="running" GET caption, name, state) - OK Process list (wmic process) - OK Swap settings (wmic pagefile) - OK Disk partition (wmic partition) - OK Disk volumes (wmic volume) - OK Network configuration (ipconfig /all) - OK Taking sample 2 after 10.000000 seconds - OK Network status (netstat -an) - OK Network routing table (netstat -rn) - OK Arp cache (arp -a) - OK Network Interface Controller (wmic nic) - OK Network Adapter (wmic nicconfig) - OK Active network connection (wmic netuse) - OK Protocols (wmic netprotocol) - OK Cache memory (wmic memcache) - OK Physical memory (wmic memphysical) - OK Physical memory chip info (wmic memorychip) - OK Local storage devices (wmic logicaldisk) - OK membase config (escript.exe /mbdumpconfig.escript /../var/lib/membase/config/config.dat) - Exit code 1 memcached stats all (mbstats -a 127.0.0.1:11210 all _admin _admin) - Exit code255 memcached stats checkpoint (mbstats -a 127.0.0.1:11210 checkpoint _admin _admin) - Exit code 255 memcached stats dispatcher (mbstats -a 127.0.0.1:11210 dispatcher logs _admin _admin) - Exit code 255 memcached stats hash (mbstats -a 127.0.0.1:11210 hash detail _admin _admin) - Exit code 255 memcached stats tap (mbstats -a 127.0.0.1:11210 tap _admin _admin) - Exit code255 memcached stats tapagg (mbstats -a 127.0.0.1:11210 tapagg _admin _admin) - Exit code 255 memcached stats timings (mbstats -a 127.0.0.1:11210 timings _admin _admin) - Exit code 255 membase logs (mbbrowse_logs) -
Any ideas?
Yes I left mbbrowse_logs for a while - 10 mins. The CPU time didnt change so I killed it.
I figured out what was the rootcause of the problem. On this windows Server we have DNS server and it started listening on port UDP 11210. When I shut id down and restart memcache everything went back to normal.
Is there anyway to disable listening on that port in membase?
Regards
Szymon
Thanks for the follow up Szymon. Unfortunately 11210 is hard coded for the memcached process to listen on.
You could change those values in the code if you wanted to recompile, but unfortunately it's not easy to do otherwise.
Perry
Ok so the other way of avoiding that issue in future is to tell DNS server not to listen on specific port.
On windows system it can be done by tweaking registry:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\ReservedPorts
whole article can be found at:
http://support.microsoft.com/kb/812873
Enjoy :)
That 'mbbrowse_logs' might possibly take a long time to run, I presume you left it running for a while?
Is it possible that the IP address of the system changed while it was running? That's just off the top of my head, we'll have to look at logs to know better.
Perry
Forum support is great for free but sometimes you need a guaranteed response time and dedicated resources for your questions or issues.
Consider purchasing enterprise-level support from Couchbase: http://www.couchbase.com/products-and-services/overview
Call or email "sales -at- couchbase-dot- com" today!