ASP.NET worker process terminating due to Hammock NullReferenceException
We're getting a complete crash of the worker process after a short time of making request to the couchbase server.
Server version is 2.0.
Client is .net 1.2.0 SDK
An unhandled exception occurred and the process was terminated.
Application ID: /LM/W3SVC/5/ROOT
Process ID: 5160
Exception: System.NullReferenceException
Message: Object reference not set to an instance of an object.
StackTrace: at Hammock.RestClient.CompleteWithQuery(WebQuery query, RestRequest request, RestCallback callback, WebQueryAsyncResult result)
at Hammock.RestClient.<>c__DisplayClass18.b__15(Object sender, WebQueryResponseEventArgs args)
at System.EventHandler`1.Invoke(Object sender, TEventArgs e)
at Hammock.Web.WebQuery.OnQueryResponse(WebQueryResponseEventArgs args)
at Hammock.Web.WebQuery.HandleWebException(WebException exception)
at Hammock.Web.WebQuery.GetAsyncResponseCallback(IAsyncResult asyncResult)
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Net.ContextAwareResult.Complete(IntPtr userToken)
at System.Net.HttpWebRequest.SetResponse(Exception E)
at System.Net.ConnectionReturnResult.SetResponses(ConnectionReturnResult returnResult)
at System.Net.Connection.CompleteConnectionWrapper(Object request, Object state)
at System.Net.PooledStream.ConnectionCallback(Object owningObject, Exception e, Socket socket, IPAddress address)
at System.Net.ServicePoint.ConnectSocketCallback(IAsyncResult asyncResult)
at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
at System.Net.ContextAwareResult.Complete(IntPtr userToken)
at System.Net.Sockets.BaseOverlappedAsyncResult.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
Additionally we see this in the couchbase log generated by the SDK:
2013-01-08 14:11:52 [DEBUG] 5 Enyim.Caching.Memcached.PooledSocket - Socket 9cd1c3b9-e0f3-422d-9861-e940886ae63f was reset
2013-01-08 14:11:52 [DEBUG] 5 Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl - Socket was reset. 9cd1c3b9-e0f3-422d-9861-e940886ae63f
2013-01-08 14:11:52 [DEBUG] 5 Couchbase.VBucketAwareOperationFactory.VBGet - Key [appctx]ApplicationContext-6275-ProductDataSet-1-0__1 was mapped to 520
2013-01-08 14:11:53 [DEBUG] 5 Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl - Releasing socket 9cd1c3b9-e0f3-422d-9861-e940886ae63f
2013-01-08 14:11:53 [DEBUG] 5 Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl - Are we alive? True
2013-01-08 14:11:53 [DEBUG] 5 Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl - Acquiring stream from pool. 10.0.3.195:11210
2013-01-08 14:11:53 [DEBUG] 5 Enyim.Caching.Memcached.PooledSocket - Socket 9cd1c3b9-e0f3-422d-9861-e940886ae63f was reset
2013-01-08 14:11:53 [DEBUG] 5 Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl - Socket was reset. 9cd1c3b9-e0f3-422d-9861-e940886ae63f
2013-01-08 14:11:53 [DEBUG] 5 Couchbase.VBucketAwareOperationFactory.VBGet - Key [appctx]ApplicationContext-6275-ProductDataSet-1-0__1 was mapped to 520
2013-01-08 14:11:53 [WARN] 11 Couchbase.HammockHttpClient - Retrying
2013-01-08 14:11:53 [WARN] 11 Couchbase.HammockHttpClient - Retrying
2013-01-08 14:11:53 [WARN] 11 Couchbase.HammockHttpClient - Retrying
2013-01-08 14:11:53 [DEBUG] 5 Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl - Releasing socket 9cd1c3b9-e0f3-422d-9861-e940886ae63f
2013-01-08 14:11:53 [DEBUG] 5 Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl - Are we alive? True
2013-01-08 14:11:53 [DEBUG] 5 Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl - Acquiring stream from pool. 10.0.3.195:11210
2013-01-08 14:11:53 [DEBUG] 5 Enyim.Caching.Memcached.PooledSocket - Socket 9cd1c3b9-e0f3-422d-9861-e940886ae63f was reset
2013-01-08 14:11:53 [DEBUG] 5 Enyim.Caching.Memcached.MemcachedNode.InternalPoolImpl - Socket was reset. 9cd1c3b9-e0f3-422d-9861-e940886ae63f
We switched to a server that had couchbase 1.8 and the problem went away. It seems to be a problem with the 1.2 .net SDK and couchbase Server 2.0.
Makes sense. I un-updated the .NET SDK and the issue went away. This is with CB2.0.
I figured as much. Posted the issue to the Hammock repository also and it's not supported or being worked on any longer. Let's hope someone from couchbase is paying attention and replaces hammock with a more viable dependency.
"I'm not working on this codebase anymore. You're better off asking the Couchbase folks to handle the exception."
The Hammock error is a known issue with the 1.2.0 client and is noted in the release notes - http://www.couchbase.com/docs/couchbase-sdk-net-1.2/couchbase-sdk-net-rn....
For the upcoming 1.2.1 release, I've replaced Hammock as the default REST client in favor of RestSharp. That release will be official in early February, but you can grab a pre-release, signed build at http://www.couchbase.com/issues/browse/NCBC-172. The binaries are attached to the ticket.
If you have explicitly configured an httpClientFactory element in your config section, you'll need to update that. If you've been using the default values, then RestSharp will automatically take over as the new client.
Thanks John,
So does this mean the 1.2 client does not support the 2.0 server yet? Is there something we can do to prevent the Hammock library from consistently failing or should we stick with the older versions?
Hi Rob,
The client does support 2.0, but the error will surface when there's a timeout or network connection when executing a view, so you could wrap your view code in a try/catch block to avoid an app crash. Are you seeing the error at another point?
-- jz
Yes, we're not using views and just doing simple ExecuteGet/ExecuteStore. Our client environment is asp.net,.net4 and the server is an Linux AMI from amazon AWS if that helps.
When the client is instantiated, it will attempt to make a web request to initialize the ServicePoint. To disable that attempt, you could modify your config to include an httpClient element (sibling of the servers element).
<httpClient initializeConnection="false" timeout="00:00:45"/>
If possible, could you paste a stack trace?
Do you need something other than the stack from the exception I posted in the OP?
Oops, I should have scrolled up! That confirms it. The exception is occurring on an async request, which confirms that it's happening on initialization. I'm not sure what's happening to make Hammock throw an NRE, but disabling it should resolve the issue until 1.2.1 is officially released.
Thanks John...will give it a go.
No problem. Let me know if you need any more help.
problem solved
Awesome to hear, thanks for updating.
I've also been seeing this. Hard crash due to timeout from within Hammock.