Machine can't connect to cluster
Hi,
We have a very interesting issue using Membase server 1.6.5.3 connecting using the Enyim client dlls:
Enyim.Caching.dll: 2.10.1
Membase.dll: 2.13.1
In our dev environment we have 2 machine clustered and we can connect just fine. All the developers' workstations can and so does our dev web server.
It gets more complicated in QA and Production.
There no client server can connect to their respective Membase cluster. It doesn't bomb w/ an exception. It just won't SET values, returning false everytime.
We've tried upgrading the servers in QA and Production to 1.7.1 but we get this:
Unable to install and configure Membase server 1.7
So we reverted back to the 1.6.5.3 install that we had because we couldn't find the install package for 1.6.5.4.
From one of the client server in QA we've tried connecting to the Membase cluster using our Web App and a WinForm app to see if the IIS context mattered but none worked.
However if I try to connect to the QA Membase cluster from my dev box it will work just fine.
Just to clarify the Membase server services in QA and Production are installed on the same servers as our Web applications that can't connect to Membase.
Any idea??
Thanks.
Perry,
I've turned on logging and here is what it says:
================================================
2011-08-11 21:50:33,452 [132] DEBUG Membase.MessageStreamListener Starting the listener. Queue=True
2011-08-11 21:50:33,468 [121] DEBUG Membase.MessageStreamListener Started working.
2011-08-11 21:50:33,468 [121] DEBUG Membase.MessageStreamListener Looking for the first working node.
2011-08-11 21:50:33,843 [121] DEBUG Membase.ConfigHelper Found config for bucket Bucket1.
2011-08-11 21:50:33,858 [121] DEBUG Membase.MessageStreamListener Resolved pool url http://10.2.5.10:8091/pools/Bucket1 to http://10.2.5.10:8091/pools/Bucket1/bucketsStreaming/Bucket1
2011-08-11 21:50:33,858 [121] DEBUG Membase.MessageStreamListener Start receiving messages.
2011-08-11 21:50:34,061 [121] INFO Membase.MembasePool Received new configuration.
2011-08-11 21:50:34,077 [121] INFO Membase.MembasePool Has vbucket. Server count: 7
2011-08-11 21:50:34,108 [121] DEBUG Membase.MessageStreamListener Processing message: {"name":"Bucket1","bucketType":"membase","authType":"sasl","saslPassword":"Bucket12k11","proxyPort":0,"uri":"/pools/Bucket1/buckets/Bucket1","streamingUri":"/pools/Bucket1/bucketsStreaming/Bucket1","flushCacheUri":"/pools/Bucket1/buckets/Bucket1/controller/doFlush","nodes":[{"replication":1.0,"clusterMembership":"active","status":"healthy","hostname":"10.2.5.10:8091","clusterCompatibility":1,"version":"1.7.1","os":"windows","ports":{"proxy":11211,"direct":11210}},{"replication":1.0,"clusterMembership":"active","status":"healthy","hostname":"10.2.5.20:8091","clusterCompatibility":1,"version":"1.7.1","os":"windows","ports":{"proxy":11211,"direct":11210}},{"replication":1.0,"clusterMembership":"active","status":"healthy","hostname":"10.2.5.30:8091","clusterCompatibility":1,"version":"1.7.1","os":"windows","ports":{"proxy":11211,"direct":11210}},{"replication":1.0,"clusterMembership":"active","status":"healthy","hostname":"10.2.5.60:8091","clusterCompatibility":1,"version":"1.7.1","os":"windows","ports":{"proxy":11211,"direct":11210}},{"replication":1.0,"clusterMembership":"active","status":"healthy","hostname":"10.2.5.70:8091","clusterCompatibility":1,"version":"1.7.1","os":"windows","ports":{"proxy":11211,"direct":11210}},{"replication":1.0,"clusterMembership":"active","status":"healthy","hostname":"10.2.5.80:8091","clusterCompatibility":1,"version":"1.7.1","os":"windows","ports":{"proxy":11211,"direct":11210}},{"replication":1.0,"clusterMembership":"active","status":"healthy","hostname":"10.2.5.50:8091","clusterCompatibility":1,"version":"1.7.1","os":"windows","ports":{"proxy":11211,"direct":11210}}],"stats":{"uri":"/pools/Bucket1/buckets/Bucket1/stats","directoryURI":"/pools/Bucket1/buckets/Bucket1/statsDirectory","nodeStatsListURI":"/pools/Bucket1/buckets/Bucket1/nodes"},"nodeLocator":"vbucket","vBucketServerMap":{"hashAlgorithm":"CRC","numReplicas":1,"serverList":["10.2.5.10:11210","10.2.5.20:11210","10.2.5.30:11210","10.2.5.50:11210","10.2.5.60:11210","10.2.5.70:11210","10.2.5.80:11210"],"vBucketMap":[[5,6],[5,6],[5,6],[5,6],[5,6],[5,6],[5,6],[5,6],[5,6],[5,6],[5,6],[5,6],[5,6],[5,6],[5,6],[5,6],[5,6],[5,6],[5,6],[5,6],[5,6],[5,6],[5,6],[5,6],[5,6],[3,6],[3,6],[3,6],[3,6],[3,6],[3,6],[3,6],[3,6],[3,6],[2,6],[2,6],[2,6],[2,6],[2,6],[2,6],[2,6],[2,6],[2,6],[2,6],[2,6],[2,6],[2,6],[2,6],[2,6],[2,6],[2,6],[1,6],[1,6],[1,6],[1,6],[1,6],[1,6],[1,6],[1,6],[1,6],[1,6],[1,6],[1,6],[1,6],[1,6],[1,6],[1,6],[1,6],[1,6],[1,6],[1,6],[1,6],[1,6],[1,6],[1,6],[6,3],[6,3],[6,3],[6,3],[6,3],[6,3],[6,3],[6,3],[6,3],[6,3],[6,2],[0,6],[0,6],[0,6],[0,6],[0,6],[0,6],[0,6],[0,6],[0,6],[0,6],[0,6],[0,6],[0,6],[0,6],[0,6],[0,6],[0,6],[0,6],[0,6],[0,6],[0,6],[0,6],[0,6],[0,6],[0,6],[6,3],[6,3],[6,3],[6,3],[6,3],[6,3],[6,3],[6,3],[6,3],[6,2],[6,2],[6,2],[6,2],[6,2],[6,2],[6,2],[6,2],[6,2],[6,2],[6,2],[6,2],[6,2],[6,2],[6,2],[6,2],[6,2],[6,1],[6,1],[6,1],[6,1],[6,1],[6,1],[6,1],[6,1],[6,1],[6,1],[6,1],[6,1],[6,1],[6,1],[6,1],[6,1],[6,1],[6,1],[6,1],[6,1],[6,1],[6,1],[6,1],[6,1],[3,1],[3,1],[3,1],[3,1],[3,1],[3,1],[3,1],[3,1],[3,1],[6,4],[6,4],[6,4],[6,4],[6,4],[6,4],[6,4],[6,4],[6,4],[6,4],[6,4],[6,4],[6,4],[6,4],[6,4],[6,4],[6,4],[6,4],[6,4],[6,4],[6,4],[6,4],[6,4],[6,4],[3,4],[3,4],[3,4],[3,4],[3,4],[3,4],[3,4],[3,4],[3,4],[3,4],[2,4],[2,4],[2,4],[2,4],[2,4],[2,4],[2,4],[2,4],[2,4],[2,4],[2,4],[2,4],[2,4],[2,4],[2,4],[2,4],[2,4],[1,4],[1,4],[1,4],[1,4],[1,4],[1,4],[1,4],[1,4],[1,4],[1,4],[1,4],[1,4],[1,4],[1,4],[1,4],[1,4],[1,4],[1,4],[1,4],[1,4],[1,4],[1,4],[1,4],[1,4],[4,3],[4,3],[4,3],[4,3],[4,3],[4,3],[4,3],[4,3],[4,3],[4,3],[0,4],[0,4],[0,4],[0,4],[0,4],[0,4],[0,4],[0,4],[0,4],[0,4],[0,4],[0,4],[0,4],[0,4],[0,4],[0,4],[0,4],[0,4],[0,4],[0,4],[0,4],[0,4],[0,4],[0,4],[4,3],[4,3],[4,3],[4,3],[4,3],[4,3],[4,3],[4,3],[4,3],[4,3],[4,2],[4,2],[4,2],[4,2],[4,2],[4,2],[4,2],[4,2],[4,2],[4,2],[4,2],[4,2],[4,2],[4,2],[4,2],[4,2],[4,2],[4,1],[4,1],[4,1],[4,1],[4,1],[4,1],[4,1],[4,1],[4,1],[4,1],[4,1],[4,1],[4,1],[4,1],[4,1],[4,1],[4,1],[4,1],[4,1],[4,1],[4,1],[4,1],[4,1],[4,1],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[3,1],[3,1],[3,1],[3,1],[1,2],[4,6],[4,6],[4,6],[4,6],[4,6],[4,6],[4,6],[4,6],[4,6],[4,6],[4,6],[4,6],[4,6],[4,6],[4,6],[4,6],[4,6],[4,6],[4,6],[4,6],[4,6],[4,6],[4,6],[4,6],[3,6],[3,6],[3,6],[3,6],[3,6],[3,6],[3,6],[3,6],[3,6],[3,6],[2,6],[2,6],[2,6],[2,6],[2,6],[2,6],[2,6],[6,3],[6,3],[6,3],[6,3],[6,3],[3,6],[3,6],[3,6],[3,6],[3,6],[3,2],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[1,5],[6,2],[6,2],[6,2],[6,2],[6,2],[6,2],[3,2],[3,2],[3,2],[3,2],[3,2],[3,2],[3,2],[3,2],[3,2],[3,2],[6,0],[6,0],[6,0],[6,0],[6,0],[6,0],[6,0],[6,0],[6,0],[6,0],[6,0],[6,0],[6,0],[6,0],[6,0],[6,0],[6,0],[6,0],[6,0],[6,0],[6,0],[6,0],[6,0],[6,0],[6,0],[3,0],[3,0],[3,0],[3,0],[3,0],[3,0],[3,0],[3,0],[3,1],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[1,0],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[2,1],[6,5],[6,5],[6,5],[6,5],[6,5],[6,5],[6,5],[6,5],[6,5],[6,5],[6,5],[6,5],[6,5],[6,5],[6,5],[6,5],[6,5],[6,5],[6,5],[6,5],[6,5],[6,5],[6,5],[6,5],[6,5],[3,5],[3,5],[3,5],[3,5],[3,5],[3,5],[3,5],[3,5],[3,5],[2,5],[2,5],[2,5],[2,5],[2,5],[2,5],[2,5],[2,5],[2,5],[2,5],[2,5],[2,5],[2,5],[2,5],[2,5],[2,5],[2,5],[1,5],[1,5],[1,5],[1,5],[1,5],[1,5],[1,5],[1,5],[1,5],[1,5],[1,5],[1,5],[1,5],[1,5],[1,5],[1,5],[1,5],[1,5],[1,5],[1,5],[1,5],[1,5],[1,5],[5,3],[5,3],[5,3],[5,3],[5,3],[5,3],[5,3],[5,3],[5,3],[5,3],[5,2],[5,2],[0,5],[0,5],[0,5],[0,5],[0,5],[0,5],[0,5],[0,5],[0,5],[0,5],[0,5],[0,5],[0,5],[0,5],[0,5],[0,5],[0,5],[0,5],[0,5],[0,5],[0,5],[0,5],[0,5],[0,5],[5,3],[5,3],[5,3],[5,3],[5,3],[5,3],[5,3],[5,3],[5,3],[5,3],[5,2],[5,2],[5,2],[5,2],[5,2],[5,2],[5,2],[5,2],[5,2],[5,2],[5,2],[5,2],[5,2],[5,2],[5,2],[5,2],[5,2],[5,1],[5,1],[5,1],[5,1],[5,1],[5,1],[5,1],[5,1],[5,1],[5,1],[5,1],[5,1],[5,1],[5,1],[5,1],[5,1],[5,1],[5,1],[5,1],[5,1],[5,1],[5,1],[5,1],[5,1],[3,1],[3,1],[3,1],[3,1],[3,1],[3,1],[3,1],[3,1],[3,1],[5,4],[5,4],[5,4],[5,4],[5,4],[5,4],[5,4],[5,4],[5,4],[5,4],[5,4],[5,4],[5,4],[5,4],[5,4],[5,4],[5,4],[5,4],[5,4],[5,4],[5,4],[5,4],[5,4],[5,4],[5,4],[3,4],[3,4],[3,4],[3,4],[3,4],[3,4],[3,4],[3,4],[3,4],[4,2],[2,4],[2,4],[2,4],[2,4],[2,4],[2,4],[2,4],[4,3],[4,3],[4,3],[4,3],[4,3],[3,4],[3,4],[3,4],[3,4],[3,4],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[0,3],[4,2],[4,2],[4,2],[4,2],[4,2],[4,2],[2,3],[2,3],[2,3],[2,3],[2,3],[2,3],[2,3],[3,2],[3,2],[3,2],[4,0],[4,0],[4,0],[4,0],[4,0],[4,0],[4,0],[4,0],[4,0],[4,0],[4,0],[4,0],[4,0],[4,0],[4,0],[4,0],[4,0],[4,0],[4,0],[4,0],[4,0],[4,0],[4,0],[4,0],[0,3],[0,3],[0,3],[0,3],[3,0],[3,0],[3,0],[3,0],[3,0],[3,0],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[0,1],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[4,5],[4,5],[4,5],[4,5],[4,5],[4,5],[4,5],[4,5],[4,5],[4,5],[4,5],[4,5],[4,5],[4,5],[4,5],[4,5],[4,5],[4,5],[4,5],[4,5],[4,5],[4,5],[4,5],[4,5],[4,5],[3,5],[3,5],[3,5],[3,5],[3,5],[3,5],[3,5],[3,5],[3,5],[3,4],[2,5],[2,5],[2,5],[2,5],[2,5],[2,5],[2,5],[5,3],[5,3],[5,3],[5,3],[3,5],[3,5],[3,5],[3,5],[3,5],[3,5],[3,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[2,1],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[1,3],[3,2],[5,2],[5,2],[5,2],[5,2],[5,2],[3,2],[3,2],[3,2],[3,2],[3,2],[3,2],[3,2],[3,2],[3,2],[3,2],[5,0],[5,0],[5,0],[5,0],[5,0],[5,0],[5,0],[5,0],[5,0],[5,0],[5,0],[5,0],[5,0],[5,0],[5,0],[5,0],[5,0],[5,0],[5,0],[5,0],[5,0],[5,0],[5,0],[5,0],[3,0],[3,0],[3,0],[3,0],[3,0],[3,0],[3,0],[3,0],[3,0],[3,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[2,0],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[0,3],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[1,2],[2,3],[2,3],[2,3],[2,3],[2,3],[2,3],[2,3],[2,3],[2,3],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[0,2],[2,3],[2,3],[2,3],[2,3],[2,3],[2,3],[2,3],[2,3],[2,3],[1,3]]}}
2011-08-11 21:50:34,124 [120] INFO LeadManager.Business.Cache.MembaseClientProvider MembaseClientProvider started - default expiration time is 480 minutes.
2011-08-11 21:50:34,155 [132] ERROR Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl Could not init pool.
System.MissingMethodException: Method not found: 'Boolean System.Threading.WaitHandle.WaitOne(Int32)'.
at Enyim.Caching.Memcached.PooledSocket.ConnectWithTimeout(Socket socket, IPEndPoint endpoint, Int32 timeout)
at Enyim.Caching.Memcached.PooledSocket..ctor(IPEndPoint endpoint, TimeSpan connectionTimeout, TimeSpan receiveTimeout)
at Enyim.Caching.Memcached.MemcachedNode.CreateSocket()
at Enyim.Caching.Memcached.Protocol.Binary.BinaryNode.CreateSocket()
at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.CreateSocket()
at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.InitPool()
2011-08-11 21:50:34,171 [132] WARN Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl Marking node 10.2.5.30:11210 as dead
2011-08-11 21:50:34,171 [132] DEBUG Membase.MembasePool Node 10.2.5.30:11210 is dead.
2011-08-11 21:50:34,171 [132] DEBUG Membase.MembasePool We have a vbucket enabled bucket, skipping the timer.
2011-08-11 21:50:34,171 [132] DEBUG Membase.MembasePool Fail handler is finished.
2011-08-11 21:50:34,171 [132] ERROR Enyim.Caching.Memcached.MemcachedNode Acquire failed. Maybe we're already disposed?
System.MissingMethodException: Method not found: 'Boolean System.Threading.WaitHandle.WaitOne(System.TimeSpan)'.
at Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl.Acquire()
at Enyim.Caching.Memcached.MemcachedNode.Acquire()
================================================
After googling a little I found another discussion you were having w/ a different user about a similar issue way back in January. However no proper solution was in that particular thread:
http://www.couchbase.org/forums/thread/could-not-connect-11210
Someone suggested that it might be a permission issue on another board:
http://www.doexpert.net/a/4e08674b042f0a27da459350
What do you think? Plausible? Should the Membase service run w/ higher elevated rights?
Thanks.
We figured this one out... thanks to your suggestion on turning on the Enyim logs. It had to do w/ the .Net version installed.
To work properly the Enyim client library needs at least .Net 3.5 SP1. The SP1 is the important part here. We had the bare 3.5 version installed.
That's all it was. It's all working now. :)
Perry thank you very much. All your help was much appreciated.
Can you enable logging on the client to see what it is showing: https://github.com/enyim/EnyimMemcached/wiki/Configure-Logging
Forum support is great for free but sometimes you need a guaranteed response time and dedicated resources for your questions or issues.
Consider purchasing enterprise-level support from Couchbase: http://www.couchbase.com/products-and-services/overview
Call or email "sales -at- couchbase-dot- com" today!