CPU Spike when ASP.NET recycles

Thanks Brian. Does the CPU stay spiked or is it only for a period of time?

Perry

It seems like there has been a lot of reports of this issue
- [URL=http://stackoverflow.com/questions/1132657/enyim-memcached-provider-cpu-spike]http://stackoverflow.com/questions/1132657/enyim-memcached-provider-cpu-spike[/URL]
- [URL=http://github.com/enyim/EnyimMemcached/issues#issue/21]http://github.com/enyim/EnyimMemcached/issues#issue/21[/URL]

We also experience 100% CPU issue every time we deploy (asp.net process recycles), which renders the site unusable until we kill the process. Could someone please look in to this.

We are looking into this right now, but it has been notoriously hard to reproduce, let alone resolve.

Just to clarify, the first post of this thread says that the w3p.exe process is spiking the CPU. Is the problem here that the NorthScale server is spiking or the client side machines?

Perry

Thanks for the other links, MMGN.com. The github link is me posting the same issue to the Enyim issues.

Sorry for the delay in responding as well. Had a permissions issue that Perry took care of for me.

My setup has NorthScale and my ASP.NET application both running on the same server. The client machine does not see performance issues, only the server. And specifically the w3p.exe process is what spikes the CPU, and we do not see the CPU come down. Our resolution has been to recycle the w3p.exe process throuhg Task Manager, and the next request to our ASP.NET application successfully loads the page. Additionally, we have reverted back to Enyim.Caching 1.2.0.8 and we do not see this issue.

I am trying using an IKVM compiled version of NorthScale's client library with a test ASP.NET application to see if I can reproduce the issue. If I cannot, then I'm sure the issue is with Enyim.Caching's NorthScale.Store
library. I will let you know what I find.

-Brian

Thanks Brian & Perry.

Our w3p.exe and memcached are on the same server also. Our issue is exactly same as the one you described immediately above. We are able to resolve the issue by terminating the w3p.exe process in task manager, and asp.net process will automatically recycle.

I think I may have made some progress to my issue.
I stopped worked with the IKVM ported version of the NorthScale java client library because I found it was too low level for me and would take a while to simulate the same code that I have in my core app.
Instead I started working with the support tools that NorthScale & Enyim.Caching provide, such as dump_logs.bat and log4net

The other engineer I'm working with hasn't used log4net before, and didn't have the XmlConfigurator wired up correctly within our application, so log4net wasn't telling us anything.
I put my application in log4net debug mode, and with it wired up correctly I believe I am able to recreate the issue.

My application starts up just fine, and I browse a few pages. Then I touch the web.config to force an application recycle, and upon the next page request, my application pool starts up again, and I immediately see an ERROR when tailing the log4net logfile.

ERROR2010-09-21 10:26:02 û System.Runtime.Serialization.SerializationException: Type 'A.B.C.JobCollection' in Assembly 'A.B, Version=1.0.3916.37905, Culture=neutral, PublicKeyToken=null' is not marked as serializable.
at System.Runtime.Serialization.FormatterServices.InternalGetSerializableMembers(RuntimeType type)
at System.Runtime.Serialization.FormatterServices.GetSerializableMembers(Type type, StreamingContext context)
at System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo.InitMemberInfo()
at System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo.InitSerialize(Object obj, ISurrogateSelector surrogateSelector, StreamingContext context, SerObjectInfoInit serObjectInfoInit, IFormatterConverter converter, ObjectWriter objectWriter)
at System.Runtime.Serialization.Formatters.Binary.WriteObjectInfo.Serialize(Object obj, ISurrogateSelector surrogateSelector, StreamingContext context, SerObjectInfoInit serObjectInfoInit, IFormatterConverter converter, ObjectWriter objectWriter)
at System.Runtime.Serialization.Formatters.Binary.ObjectWriter.Serialize(Object graph, Header[] inHeaders, __BinaryWriter serWriter, Boolean fCheck)
at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Serialize(Stream serializationStream, Object graph, Header[] headers, Boolean fCheck)
at Enyim.Caching.Memcached.DefaultTranscoder.Enyim.Caching.Memcached.ITranscoder.Serialize(Object value) in d:\EnyimMemcached\Enyim.Caching\Memcached\Transcoders\DefaultTranscoder.cs:line 99
at NorthScale.Store.NorthScaleClient.PerformStore(StoreMode mode, String key, Object value, UInt32 expires, UInt64& cas) in d:\EnyimMemcached\Northscale.Store\NorthScaleClient.cs:line 137

I also then see an error within my application's logfile:

Unable to cast object of type 'Enyim.Caching.Memcached.Protocol.Binary.GetOperation' to type 'NorthScale.Store.IOperationWithState'.
at NorthScale.Store.NorthScaleClient.ExecuteWithRedirect(IMemcachedNode startNode, ISingleItemOperation op) in d:\EnyimMemcached\Northscale.Store\NorthScaleClient.cs:line 164
at NorthScale.Store.NorthScaleClient.PerformTryGet(String key, UInt64& cas, Object& value) in d:\EnyimMemcached\Northscale.Store\NorthScaleClient.cs:line 80
at Enyim.Caching.MemcachedClient.Get(String key) in d:\EnyimMemcached\Enyim.Caching\MemcachedClient.cs:line 98

Since recycling my application, my w3p.exe process has not dropped below 50%. Granted it's not peg'd at 100% CPU, but I know my application is sitting idle doing nothing, so I'm really curious where the CPU is being used. I'm running DebugDiag again to see what IIS thinks it's doing, but it *looks* like the issue is within my application's code.

I'll keep you posted as I fix what I found.
-Brian

2 things since my last post:
1.) I reverted NorthScale.Store.dll back to 2.5. Yesterday I tried using their latest release, 2.6, since it just came out on 9/28. I wasn't sure if that possibly contributed to the second error message I received about Binary.GetOperation, and I don't need it anyway since I'm not working with MemBase Server. So I'm back to NorthScale.Store.dll 2.5 and Enyim.Caching.dll 2.4

2.) I commented out the code that was failing above with the JobCollection not being serializable.

With these 2 changes, I no longer see the CPU spike after recycling my application pool.
I am going to re-introduce the JobCollection code as it currently is now to see if I can reproduce the issue again. If I can reproduce it, then that is definitely the issue, and I will resolve that within my own code.

However, if this does prove to be the root cause of my problems, then I think it merits an issue to Enyim.Caching to better handle the exception within their NorthScaleClient.cs class.

More to come later tonight or tomorrow morning...

Hey doublestakked,

Did you have further issues with this? We see production failure every other day, really puzzled on how to resolve this.

 We are experiencing the same issue in production, doesn't happen on every recycle but it is quite frequently.  We get w3wp.exe process running at 100% cpu usage, proving very difficult to track down the cause.  We are using latest Enyim 2.7 and Membase 2.10 client.  We are working with the cache and w3wp.exe process running on the same machine.  Has anyone found a work around for this issue?

The Engineers are looking into this as I write.

-bhawana

 Just to add to my previous post, and assist in any debugging effort thought it might be helpful to give a few more specifics, as this issue might be only opccuring on a setup similar to ours....

We are running our Application Pool as a web garden with a maximum of 5 worker processes, and have managed pipeline mode set to Classic.  This is a Asp.Net 3.5 application running on a Windows Server 2008 (64 bit)

Thanks,

Matt

Thanks for more information.

Please send me an email on bhawana -at-membase-com. I would like you to try a patch for memcached to see if you still see this issue.

Thanks

Bhawana

Your client instances may not have been disposed. The MembaseClient has a background thread for the configuration management, and this must be stopped by disposing the client.

The easiest solution is to add the following to your Global.asax:

protected void Application_End(object sender, EventArgs e)
{
// dispose the clients here
}

Thanks

Bhawana