.NET client error. Unable to read data from the transport connection

Hi all!

At any operation we have an error: Unable to read data from the transport connection: A blocking operation was interrupted by a call to WSACancelBlockingCall.

StackTrace example:

Start execute 22c5340420b15e81588939fa4f71e06c with bc30f3cd-0789-404a-be07-52a30b214eaa

Operation failed :i 22c5340420b15e81588939fa4f71e06c n:  t: cbc_thread-63 m: Exception reading response - X.X.X.X:11210 sc: InternalError r: 0

Operation failed :i 22c5340420b15e81588939fa4f71e06c n: X.X.X.X:11210 t: cbc_thread-63 m: Exception reading response - X.X.X.X:11210 sc: InternalError r: 0

End execute 22c5340420b15e81588939fa4f71e06c with bc30f3cd-0789-404a-be07-52a30b214eaa
m:Exception reading response - X.X.X.X:11210 s:22c5340420b15e81588939fa4f71e06c i:bc30f3cd-0789-404a-be07-52a30b214eaa

System.IO.IOException: Unable to read data from the transport connection: A blocking operation was interrupted by a call to WSACancelBlockingCall. ---> System.Net.Sockets.SocketException: A blocking operation was interrupted by a call to WSACancelBlockingCall
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at Couchbase.CouchbasePooledSocket.Read(Byte[] buffer, Int32 offset, Int32 count)
   at Enyim.Caching.Memcached.Protocol.Binary.BinaryResponse.Read(IPooledSocket socket)
   at Enyim.Caching.Memcached.Protocol.Binary.BinarySingleItemOperation.ReadResponse(IPooledSocket socket)
   at Couchbase.CouchbaseNode.Execute(IOperation op)

Operation timeout.

Cluster configuration: 5 Couchbase Server 2.2.0 Community Edition nodes on CentOS 6.5, .NET application on IIS 7.5 on Windows Server 2008 R2 SP1. We used client library of version 1.3.4.

What could be the cause of this error and how to fix it?

Thanks.

tx_user -

How often is this happening? What is the cluster’s state at this time: failover/rebalance in progress, add/remove node then rebalance, general healthy state, etc?

-Jeff

It happens often, about 60-100% of all requests.

Cluster have healthy state.

Hi tx_user -

Can you enable logging so that it writes to a file and then create an NCBC (jira ticket) and attach the log file? Set the log level to DEBUG and include the start of the application through to a shutdown.

If you can, please include your app.config or even better an example app so I can try to reproduce the issue.

-Jeff

Hi Jeff,

We debugged the networking communication between the application and the database server and found the problem. Error stopped after fine tuning OS limits of processes and opened files and TCP settings.

Thanks for answers.

tx_user -

Glad to hear your up and running! Would you mind sharing what you did with respect to “Error stopped after fine tuning OS limits of processes and opened files and TCP settings.” as an answer to your original question?

It really helps if you share your work around, so that the next person who encounters similar issues getting up and running has something to start with.

Thanks,

Jeff

Hi,

We set the max number of processes and the max number of open files (CentOS):

/etc/security/limits.conf
couchbase  soft    nproc   32000
couchbase  hard    nofile  64000
couchbase  soft    nofile  64000

And set default sysctl settings (earlier we had specific settings for other database).

Also we configure MaxUserPort and TCPTimedWaitDelay on Windows.

Now we process 15k+ rps without problems.

I am consistently getting this error after upgrading the CB ,net client library from 1.2.6.0 to 1.3.9.0. Here’s what I got in the log file after enabling logging. Any insight into how I can dig further? Thanks.

2014-10-30 10:53:19,376 [cbc_thread-1] ERROR Couchbase.CouchbaseNode [(null)] <(null)> - m:Exception reading response - 10.4.3.63:11210 s:Endpoints_http://inrix.incidents.iclosuressink/SHWETAS01/NA i:1a34cf74-af3d-4627-9a89-56f0ebcf7c30
System.IO.IOException: Unable to read data from the transport connection: A blocking operation was interrupted by a call to WSACancelBlockingCall. ---> System.Net.Sockets.SocketException: A blocking operation was interrupted by a call to WSACancelBlockingCall
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at Couchbase.CouchbasePooledSocket.Read(Byte[] buffer, Int32 offset, Int32 count) in c:\Users\jmorris\repos\CouchbaseNetClient-Build\couchbase-net-client\src\Couchbase\CouchbasePooledSocket.cs:line 102
   at Enyim.Caching.Memcached.Protocol.Binary.BinaryResponse.Read(IPooledSocket socket) in c:\Users\jmorris\repos\CouchbaseNetClient-Build\couchbase-net-client\src\Enyim.Caching\Memcached\Protocol\Binary\BinaryResponse.cs:line 61
   at Enyim.Caching.Memcached.Protocol.Binary.BinarySingleItemOperation.ReadResponse(IPooledSocket socket) in c:\Users\jmorris\repos\CouchbaseNetClient-Build\couchbase-net-client\src\Enyim.Caching\Memcached\Protocol\Binary\BinarySingleItemOperation.cs:line 24
   at Enyim.Caching.Memcached.Protocol.Operation.Enyim.Caching.Memcached.IOperation.ReadResponse(IPooledSocket socket) in c:\Users\jmorris\repos\CouchbaseNetClient-Build\couchbase-net-client\src\Enyim.Caching\Memcached\Protocol\Operation.cs:line 32
   at Couchbase.CouchbaseNode.Execute(IOperation op) in c:\Users\jmorris\repos\CouchbaseNetClient-Build\couchbase-net-client\src\Couchbase\CouchbaseNode.cs:line 95