Connection to the cluster fails unexpectedly (using Couchbase client library)
We're using Couchbase 1.8.1 community edition (build-937) on CentOS 6.3 64 bit. There are two identical machines in the cluster, and our Java app connects to the cluster via Couchbase client library 1.0.2.
The app handles about 1k requests per second, doing about 2k Couchbase queries per second (from 50 threads).
Once in a while app starts to get tons of the errors like
"Node exepcted to receive data is inactive. This could be due to afailure within the cluster. Will check for updated configuration. Key without a configured node is: XXXXXX."
and stacktraces like
Caused by: java.lang.IllegalArgumentException: Bucket name can not be blank.
and the application cannot make a single successful request to the cluster. After the app is restarted everything goes OK. If we try to execute similar queries from other JVM instance (while first one is failing), it goes fine as well.
There's nothing unusual in couchbase server logs.
Is this something known?