IO Exception Error ConnectionReset
Hello,
i have problem that occurred intermittent. I build an application that can connect to a couchbase server and memcached server. the client I used are couchbase .net client and enyim.caching memcached client. The application is running fine but sometimes there are error like this :
“System.IO.IOException: Failed to write to the socket '10.0.10.109:11211'. Error: ConnectionReset
at Enyim.Caching.Memcached.PooledSocket.Write(IList`1 buffers)
at Enyim.Caching.Memcached.MemcachedNode.ExecuteOperation(IOperation op)”
btw my setup is like this
10.0.10.110:11211 => memcached server
10.0.10.109:8091 => couchbase server
192.168.123.60 => client application
But strangely if I put my client in the same class ip (10.0.10.###) the problem is not happening.
On my old application where I used MemcachedProvider as the client. there are no problem like this. Even the ip is not in the same class.
So the question is this problem is cause by the network or by the client ?
Thanks a lot
hello John,
thanks for the reply and sorry for the late respond :P
actually i only used 1 cache server. my future plan is to used couchbase as the cache server because i can separate my data into buckets.
memcached is my first used cache server. at that time i used MemcachedProvider as it's client. when i used memcached server and MemcachedProvider as it's client there are no problem at all. everythin is working fine. no connection error or reset at all. even when on a different class ip.
then i found out about membase/couchbase and interested to try them. i start migrate to membase/couchbase. when i first try to go live with membase/couchbase. i found this connection reset error happen intermittent when the server class ip is different.
at first i thought it's because of the membase/couchbase server that cause this problem. but then i try to code to connect to memcached server using the enyim.caching.memcachedclient. and this problem happen again.
i need to find out why the connection is always reset when the server is in a different class ip. without solving this problem i can not go live with couchbase :(. if there is a problem with the network. my old apps is working fine when i used memcached and MemcachedProvider.
for the network traffic and utilization, i think it can consider low. around 5-10% of my gigabit network.
bump.
Hi Hendra,
Could you try to enable logging?
Download the assemblies at http://packages.couchbase.com/clients/net/1.1/Couchbase-Net-Client-Loggi...
http://www.couchbase.com/docs/couchbase-sdk-net-1.1/couchbase-sdk-net-lo...
Hello john,
here is the the log i manage to gather http://www.sendspace.com/file/4tt7kl (the log is quite big so i zip it)
there are around 6 connection reset in this log. hope this can help identify the problem.
Thanks you
Hi Hendra,
Nothing is jumping out at me in the log. I don't see anything to suggest that there's a problem with the client. One suggestion that I think is worth trying would be to remove the Memcached server and the MemcachedProvider and replace it with a full CouchbaseClient solution.
On the server side, you would setup two buckets - one Couchbase bucket as you already have and a second Memcached bucket. In the configuration for the CouchbaseClient, you would need two bucket definitions as described in the wiki - http://www.couchbase.com/wiki/display/couchbase/Couchbase+.NET+Client+Li....
I suggest this because it will both simplify your deployment and reduce the number of variables on the network...
Let me know if that makes sense to you.
-- John
Hello John,
in this log i used full CouchvaseClient Solution. there are no memcached server or memcached provider in this test.
here i extract one of the error thread for an easy view. what i can see in the log somehow when the thread try to add the cache the connection is lost. after that the connection is marked dead for 3 min then it create a new connection again.
what i still wandering is what can cause the connection to be lost ? or anything that can cause an I/O error like this?
i already create a static client, so the client is shared for ever operation. for my detail caching code, you can check my code here http://www.couchbase.com/forums/thread/couchbase-server-httpserver8091po... (post no.3)
2012-07-16 00:06:36,580 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Acquiring stream from pool. 10.0.10.109:11210 2012-07-16 00:06:36,580 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.PooledSocket [(null)] <(null)> - Socket b1990bba-78ef-4c60-a9c7-24544020cd46 was reset 2012-07-16 00:06:36,580 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Socket was reset. b1990bba-78ef-4c60-a9c7-24544020cd46 2012-07-16 00:06:36,580 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Couchbase.VBucketAwareOperationFactory.VBStore [(null)] <(null)> - Key HndblLVCache_33 was mapped to 215 2012-07-16 00:06:36,533 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Acquiring stream from pool. 10.0.10.109:11210 2012-07-16 00:06:36,596 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.PooledSocket [(null)] <(null)> - Socket 2c3af599-539c-4fe8-9ecf-b3bd0e548908 was reset 2012-07-16 00:06:36,596 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Socket was reset. 2c3af599-539c-4fe8-9ecf-b3bd0e548908 2012-07-16 00:06:36,596 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Couchbase.VBucketAwareOperationFactory.VBStore [(null)] <(null)> - Key TensLVCache_33 was mapped to 748 ERROR HERE >>> 2012-07-16 00:06:36,596 [SBDB.Controller.CreateOthLVHist THREAD] ERROR Enyim.Caching.Memcached.MemcachedNode [(null)] <(null)> - System.IO.IOException: Failed to write to the socket '10.0.10.109:11210'. Error: ConnectionReset 2012-07-16 00:06:36,596 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Releasing socket 2c3af599-539c-4fe8-9ecf-b3bd0e548908 2012-07-16 00:06:36,596 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Are we alive? False 2012-07-16 00:06:36,596 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Acquiring stream from pool. 10.0.10.109:11210 2012-07-16 00:06:36,596 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Pool is dead or disposed, returning null. 10.0.10.109:11210 2012-07-16 00:06:36,596 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Releasing socket b1990bba-78ef-4c60-a9c7-24544020cd46 2012-07-16 00:06:36,596 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Are we alive? False 2012-07-16 00:06:36,596 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Acquiring stream from pool. 10.0.10.109:11210 2012-07-16 00:06:36,596 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Pool is dead or disposed, returning null. 10.0.10.109:11210 2012-07-16 00:06:36,611 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Acquiring stream from pool. 10.0.10.109:11210 2012-07-16 00:06:36,611 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Pool is dead or disposed, returning null. 10.0.10.109:11210 2012-07-16 00:06:36,611 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Acquiring stream from pool. 10.0.10.109:11210 2012-07-16 00:06:36,611 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Pool is dead or disposed, returning null. 10.0.10.109:11210 . . . ( the message is the same "Pool is dead or disposed, returning null. 10.0.10.109:11210" for around 3 minutes) . . 2012-07-16 00:08:34,018 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Acquiring stream from pool. 10.0.10.109:11210 2012-07-16 00:08:34,018 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Pool is dead or disposed, returning null. 10.0.10.109:11210 2012-07-16 00:08:37,060 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Acquiring stream from pool. 10.0.10.109:11210 2012-07-16 00:08:37,060 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.PooledSocket [(null)] <(null)> - Socket 520214de-a9e9-4d74-888d-4444928e890a was reset 2012-07-16 00:08:37,060 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Socket was reset. 520214de-a9e9-4d74-888d-4444928e890a 2012-07-16 00:08:37,060 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Couchbase.VBucketAwareOperationFactory.VBStore [(null)] <(null)> - Key HndblLVCache_21 was mapped to 962 2012-07-16 00:08:37,075 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Releasing socket 520214de-a9e9-4d74-888d-4444928e890a 2012-07-16 00:08:37,075 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Are we alive? True 2012-07-16 00:08:37,060 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Acquiring stream from pool. 10.0.10.109:11210 2012-07-16 00:08:37,075 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.PooledSocket [(null)] <(null)> - Socket 53c2c27f-eb42-4f27-9157-d49b41132470 was reset 2012-07-16 00:08:37,075 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Socket was reset. 53c2c27f-eb42-4f27-9157-d49b41132470 2012-07-16 00:08:37,075 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Couchbase.VBucketAwareOperationFactory.VBStore [(null)] <(null)> - Key TensLVCache_21 was mapped to 505 2012-07-16 00:08:37,106 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Acquiring stream from pool. 10.0.10.109:11210 2012-07-16 00:08:37,106 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.PooledSocket [(null)] <(null)> - Socket 5b5ec402-0e0f-4b6d-884e-4c4bf3b832f2 was reset 2012-07-16 00:08:37,106 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Socket was reset. 5b5ec402-0e0f-4b6d-884e-4c4bf3b832f2 2012-07-16 00:08:37,106 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Couchbase.VBucketAwareOperationFactory.VBStore [(null)] <(null)> - Key VlyblLVCache_21 was mapped to 328 2012-07-16 00:08:37,106 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Releasing socket 5b5ec402-0e0f-4b6d-884e-4c4bf3b832f2 2012-07-16 00:08:37,106 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Are we alive? True 2012-07-16 00:08:37,138 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Acquiring stream from pool. 10.0.10.109:11210 2012-07-16 00:08:37,169 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Acquiring stream from pool. 10.0.10.109:11210 2012-07-16 00:08:37,106 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Releasing socket 53c2c27f-eb42-4f27-9157-d49b41132470 2012-07-16 00:08:37,216 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.PooledSocket [(null)] <(null)> - Socket 520214de-a9e9-4d74-888d-4444928e890a was reset 2012-07-16 00:08:37,231 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Could not get a socket from the pool, Creating a new item. 10.0.10.109:11210 2012-07-16 00:08:37,247 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Are we alive? True 2012-07-16 00:08:37,278 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Socket was reset. 520214de-a9e9-4d74-888d-4444928e890a 2012-07-16 00:08:37,309 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Couchbase.VBucketAwareOperationFactory.VBStore [(null)] <(null)> - Key GlfLVCache_21 was mapped to 160 2012-07-16 00:08:37,294 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Done. 2012-07-16 00:08:37,309 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Couchbase.VBucketAwareOperationFactory.VBStore [(null)] <(null)> - Key cyclLVCache_21 was mapped to 703 2012-07-16 00:08:37,309 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Releasing socket 5298cf35-b239-4894-8abb-1899c6d94496 2012-07-16 00:08:37,309 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Are we alive? True 2012-07-16 00:08:37,309 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Releasing socket 520214de-a9e9-4d74-888d-4444928e890a 2012-07-16 00:08:37,309 [SBDB.Controller.CreateOthLVHist THREAD] DEBUG Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl [(null)] <(null)> - Are we alive? True
bump
I also have the same problem. My couchbase client keeps resetting the sockets and under load we spend a lot of time trying to recreate those sockets. Socketpooling did not help. I think the solution is to enable keep-alive on the connections so that we dont recreate them all the time.
Hi Hendra,
Just to confirm... You have a Memcached server and a Couchbase Server on two different physical machines? Not a single machine with two network cards or something like that?
It would seem that there's a problem on the network. What sort of traffic are you experiencing? Is the one Couchbase server being heavily utilized? When you click into the node on the cluster, do you see any resources being hammered?
Also, just out of curiosity, have you considered setting up a two node Couchbase cluster instead of using a mixed Memcached and Couchbase setup? If you ran Couchbase on both servers you could create both Memcached and Couchbase buckets.
-- John