Runaway worker process when cluster node shutdown - SessionState provider
I am experiencing a problem that has multiple symptoms and all in all seems very strange.
I have a single Windows Server 2012, IIS8 EC2 instance hosting an ASP .Net WebForms site using .Net Fx 4.0. This is configured to use a Couchbase 2.1.1 community edition (build-764-rel) cluster of 2 nodes.
The Client components are: CouchBaseNetClient 1.2.9 and CouchBaseAspNet 1.2.1 both installed from NuGet.
If I shutdown one of the nodes (either one, same problem) to simulate failure; then the following are observed.
The connected browser clients get redirected to a 404 page when they try to postback.
If I re-enter the home page url in the browser and navigate there, the server does not repsond to the get request and the server's w3wp process jumps to 50% and just sits there. If I browse to the site from another client, the w3wp processor usage jumps to 100% - eventually these requests timeout on the browser but the processor use remains the same on the server.
Stopping and starting the app pool seems to have no effect, a full iisreset is required. When I then access the site again after an iisreset, any pages that use session state timeout.
The system cannot be restored to working condition until the dropped Couchbase node is returned to the cluster.
This problem happens when exclusiveAccess is set to true or false.
The web.confg entries are included below:
<sessionState customProvider="Couchbase" mode="Custom" timeout="10"> <providers> <add name="Couchbase" type="Couchbase.AspNet.SessionState.CouchbaseSessionStateProvider, Couchbase.AspNet" exclusiveAccess="false" /> </providers> </sessionState> </system.web> <couchbase> <servers bucket="default" bucketPassword="passwd"> <add uri="http://cache1.domain.com:8091/pools" /> <add uri="http://cache2.domain.com:8091/pools" /> </servers> </couchbase> <code>
Thank you very much for prioritising this.
I'm not sure if this has any relevance, but just in case:
The set up is reliant on the configuration described in this article: http://alestic.com/2009/06/ec2-elastic-ip-internal
I.e. the dns referred to in the web.config cache1.domain.com is a cname to the ec2 public dns e.g. ec2-254-254-254-254-someregion-etc.com