Intermittent Node Failure Events Using .NET Client
We are experiencing a intermittent case when using the Couchbase .NET client connecting to a 3-node Couchbase 1.8 cluster. At completely random times, our web application will hang from 30 to 90 seconds and then run like normal for the next hour or so before hitting another 30 to 90 seconds of hanging state. We debugged the issue by attaching logs to the Couchbase events and during the hangs, we are seeing node failure events from each of the three nodes. We will random see 1 to 3 errors per node which we believe corresponds to the 30 to 90 seconds that our application is hanging.
There is a firewall between the web servers (Windows 2008R2/IIS 7.0/MVC 3.5 web application) and we have opened up ports 8091, 11210 and 11211 per the node to client documentation at http://www.couchbase.com/docs/couchbase-manual-1.8/couchbase-network-por... and http://www.couchbase.com/docs/couchbase-manual-1.8/couchbase-bestpractic.... I do not believe we need port 11210 opened up but we figured we would open that up to see if that was causing the issues. We have looked at the firewall logs and do not believe we are blocking any traffic or ports that Couchbase needs.
Has anyone in the community seen this behavior. We have a 3-node cluster in our QA environment but we do not have a firewall between our web applications and the Couchbase cluster. We are wondering if we are missing a management or heartbeat port of sometime and causing the random node failure events.