Memcached.exe cpu consumption

Thanks Tom. We're digging into this now.

A few questions...

Is the "high cpu" server always the same server? Out of the three, is it the first one you setup, the second, the third?

Can you upload a zip of the output of dump_logs.bat from the sever that is experiencing the high cpu?

Thanks!

Perry

Yes, the "high cpu" server is always (twice) the same.

I'm not sure which one I setup first. I thought I'd look on the creation date of any file under the NorthScale folder, but all the creation dates are the same between the servers. How can I find that out?

I've had some problem uploading the logs to the forum, so I put them here:
[url]http://rapidshare.com/files/400312287/nslogs.zip.html[/url]

Note about the logs: I've restarted the NorthScale service on that machine approximately 24 hours ago, and the cpu consumption returned to normal.

Thank you!

Sorry for the delay Tom. I did grab your logs, but nothing immediately popped out at me. I'll try to review this in the next day or so.

It's odd that it's on only one system. Is there a windows service pack/hotfix difference between the two?

All three servers have Windows 2008 Datacenter SP 2 and have the same fixes installed.

Thank you!

Tom, are you still seeing high cpu on that one server?

If so, can you telnet into each of the servers and run: 'stats'? Wait 10 seconds and then run it again to give us an idea of the rate of change.

Thanks!

Perry

I've restarted the NorthScale service and the cpu utilization returned to normal. I fear that this will probably happen again, because it has already happened twice. If/when this happens again I'll get the stats before restarting the service.

I went ahead and got the stats anyway.

I suddenly realize that I've restarted the service, and then got the logs (which I posted on rapidshare). It means that things might seem normal at the end of the log (06-18-2010) and the problem would only appear earlier in the log (06-07-2010). Sorry about this, I know the log is extremely long.

Thanks!

Tom.

The stats:
First Sample:

STAT pid 46544
STAT uptime 793469
STAT time 1277559099
STAT version 1.4.4_203_g7bf58ee
STAT pointer_size 64
STAT daemon_connections 10
STAT curr_connections 886
STAT total_connections 1123660
STAT connection_structures 1158
STAT cmd_get 186864875
STAT cmd_set 609764
STAT cmd_flush 0
STAT auth_cmds 0
STAT get_hits 186259101
STAT get_misses 605774
STAT delete_misses 0
STAT delete_hits 0
STAT incr_misses 0
STAT incr_hits 0
STAT decr_misses 0
STAT decr_hits 0
STAT cas_misses 0
STAT cas_hits 0
STAT cas_badval 0
STAT bytes_read 11273554534
STAT bytes_written 121797510702
STAT limit_maxbytes 67108864
STAT rejected_conns 0
STAT threads 4
STAT conn_yields 0
STAT evictions 0
STAT curr_items 540
STAT total_items 609764
STAT bytes 812098
STAT engine_maxbytes 104857600
STAT bucket_conns 1399
END

Second Sample:

STAT pid 46544
STAT uptime 793517
STAT time 1277559147
STAT version 1.4.4_203_g7bf58ee
STAT pointer_size 64
STAT daemon_connections 10
STAT curr_connections 865
STAT total_connections 1123722
STAT connection_structures 1158
STAT cmd_get 186886963
STAT cmd_set 609862
STAT cmd_flush 0
STAT auth_cmds 0
STAT get_hits 186281091
STAT get_misses 605872
STAT delete_misses 0
STAT delete_hits 0
STAT incr_misses 0
STAT incr_hits 0
STAT decr_misses 0
STAT decr_hits 0
STAT cas_misses 0
STAT cas_hits 0
STAT cas_badval 0
STAT bytes_read 11274880296
STAT bytes_written 121811717992
STAT limit_maxbytes 67108864
STAT rejected_conns 0
STAT threads 4
STAT conn_yields 0
STAT evictions 0
STAT curr_items 551
STAT total_items 609862
STAT bytes 809148
STAT engine_maxbytes 104857600
STAT bucket_conns 1378
END

Tom, can you package up the stats from the other servers as well? I'm looking for any differences that may explain why this particular server has experienced such a high cpu load. It will be helpful to get that as soon as possible, and even more helpful if we can get it while the server is experiencing a high cpu load.

Thanks!

Perry

The cpu is currently high, so I was able to bring you the relevant data.

I've pulled the logs with dump_logs.bat:
[url]http://rapidshare.com/files/403744203/nslogs.zip.html[/url]

And attached the results of the "stats" command to this message.

I'll wait a while before restarting the service, please let me know soon if there's anymore data you'd like me to collect, since I'd like to restart the service as soon as possible.

Thank you very much!

Tom.

Thanks Tom!

Go ahead and restart the service when you need to. We'll analyze this information and let you know.

Thanks again.

Perry

Thanks Tom!

Go ahead and restart the service when you need to. We'll analyze this information and let you know.

Thanks again.

Perry

Great! Thanks Perry!

Tom, I haven't forgotten you, we're just swamped with a bunch of other things right now and this one is particularly hard to diagnose and reproduce.

Let me know if you're experiencing any problems...

Thanks

Perry

Thank you Perry.

The only problem I'm experiencing it the recurrence of the high cpu consumption (about once a week). :\

Tom.

Tom, I'm still trying to reproduce this and have as yet been unsuccessful. The next time that you need to restart the service, would it be possible to substitute a different machine and see if the problem recurs there? That will help determine whether there is anything specific to one system causing this.

Thanks

Perry

Perry, I'm not sure I understand the meaning of "substitute a different machine". I'm using amazon servers, so it is possible for me to tear down this server and replace it with a new one, while leaving the other the other two servers just as they are. I don't have to wait for the next occurrence of the problem, so I just might do that.

Thank you!

Tom.

Thanks Tom, it wasn't totally clear for me whether you were running on Amazon or not. That obviously makes things even easier...

If at all possible, when the "bad" server comes under high CPU load, you could leave it running but essentially disable it from your app and replace it with a new one. This would allow us to attach a debugger and/or profiler to it without affecting your site's performance.

I must apologize that this is taking so long, but being unable to reproduce this issue essentially means we're working blind. I really do appreciate your patience and I'm committed to getting this resolved.

Thanks!

Perry

I've recently replaced the problematic server with a new one. It seems to work just fine, so as far as I'm concerned the problem is solved. Thank you Perry!

Glad to hear Tom, let me know if there's anything else we can do for you.

Perry