Couchbase Server

High CPU usage after resuming from sleep/hibernate

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Blocker Blocker
  • Resolution: Fixed
  • Affects Version/s: 1.6.4.1, 1.8.1, 2.0
  • Fix Version/s: 2.0.2
  • Component/s: ns_server
  • Security Level: Public
  • Labels:
  • Environment:
    Windows 7 Professional x64
    MacOS 10.6
    Any

Description

On my test notebook, erl.exe spikes to 100% cpu usage (25%-50 actually, since it's a 2-core+HT CPU) after the system resumes from standby/hibernate.
No load is generated (clients are idle), the cache contains around 50K items using about 8MiB.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
onyxmaster added a comment -
I forgot to add that the CPU spike goes down in several minutes, and this happened around four or five times, all the time after it resumes from standby/hibernate (it's easy to spot on a notebook, since the CPU fan noise goes up when there's any noticeable CPU utilization).
Show
onyxmaster added a comment - I forgot to add that the CPU spike goes down in several minutes, and this happened around four or five times, all the time after it resumes from standby/hibernate (it's easy to spot on a notebook, since the CPU fan noise goes up when there's any noticeable CPU utilization).
Hide
Permalink
Sean Lynch added a comment -
This happens to me on OS X as well. I haven't bothered to look too deeply into it yet since I seemed to be the only person leaving it running on my laptop. I'll see if I can figure out what's causing this.
Show
Sean Lynch added a comment - This happens to me on OS X as well. I haven't bothered to look too deeply into it yet since I seemed to be the only person leaving it running on my laptop. I'll see if I can figure out what's causing this.
Hide
Permalink
Bill Kehoe added a comment -
I realize I'm way "late to the game" here but just in case it helps, I've observed the same behavior running v1.6.5.3 on Windows 7 64-bit. I will upgrade to 1.7.1 and post again if I still see the problem.
Show
Bill Kehoe added a comment - I realize I'm way "late to the game" here but just in case it helps, I've observed the same behavior running v1.6.5.3 on Windows 7 64-bit. I will upgrade to 1.7.1 and post again if I still see the problem.
Hide
Permalink
Aleksey Kondratenko added a comment -
doesn't seem to happen on GNU/Linux BTW
Show
Aleksey Kondratenko added a comment - doesn't seem to happen on GNU/Linux BTW
Hide
Permalink
Farshid Ghods added a comment -
seeing the same issue in my mac after resuming the proces.

attached the diags here.
Show
Farshid Ghods added a comment - seeing the same issue in my mac after resuming the proces. attached the diags here.
Hide
Permalink
Paul Carroll added a comment -
i'm also seeing this very regularly (nearly 80-90% of times coming out of hibernation)... is there anything I can do/upload/check to help someone diagnose?
Show
Paul Carroll added a comment - i'm also seeing this very regularly (nearly 80-90% of times coming out of hibernation)... is there anything I can do/upload/check to help someone diagnose?
Hide
Permalink
Matt Ingenthron added a comment -
Assigning to Dipti for scheduling.

This one has been around for quite a while and is very, very annoying from a developer perspective.
Show
Matt Ingenthron added a comment - Assigning to Dipti for scheduling. This one has been around for quite a while and is very, very annoying from a developer perspective.
Hide
Permalink
Matt Ingenthron added a comment - - edited
See also MB-4492
Show
Matt Ingenthron added a comment - - edited See also MB-4492
Hide
Permalink
Farshid Ghods added a comment -
comments from Aliaksey on the other bug the other bug :
"""
I'm seeing in logs that apparently erlang delivers all timer ticks that were 'skipped' while machine was suspended.
If we want to fix that it needs to be fixed in Erlang and it's highly OS-specific. Thus assigning to our OSX Guru.
"""
Show
Farshid Ghods added a comment - comments from Aliaksey on the other bug the other bug : """ I'm seeing in logs that apparently erlang delivers all timer ticks that were 'skipped' while machine was suspended. If we want to fix that it needs to be fixed in Erlang and it's highly OS-specific. Thus assigning to our OSX Guru. """
Hide
Permalink
Matt Ingenthron added a comment -
Note though that it affects multiple platforms. It's not just OS X.
Show
Matt Ingenthron added a comment - Note though that it affects multiple platforms. It's not just OS X.
Hide
Permalink
Aleksey Kondratenko added a comment -
As noted above at least on OSX this is about erlang/OS combination using real time clock and having erlang tick all missing ticks during hibernate/suspend whatever. So this seems to require fixing erlang VM.
Show
Aleksey Kondratenko added a comment - As noted above at least on OSX this is about erlang/OS combination using real time clock and having erlang tick all missing ticks during hibernate/suspend whatever. So this seems to require fixing erlang VM.
Hide
Permalink
Aleksey Kondratenko added a comment -
Changed component to ns_server. Logs indicate however this is fundamentally Erlang VM issue. We're not going to workaround it. If we fix this, then we fix erlang.
Show
Aleksey Kondratenko added a comment - Changed component to ns_server. Logs indicate however this is fundamentally Erlang VM issue. We're not going to workaround it. If we fix this, then we fix erlang.
Hide
Permalink
Aleksey Kondratenko added a comment -
Assigned back to PM to decide fate of this.
Show
Aleksey Kondratenko added a comment - Assigned back to PM to decide fate of this.
Hide
Permalink
Farshid Ghods added a comment -
Damien mentioned we can pass a new flag to erl while starting erlang process.
need to change ns_serrver and couchdbx-app
Show
Farshid Ghods added a comment - Damien mentioned we can pass a new flag to erl while starting erlang process. need to change ns_serrver and couchdbx-app
Hide
Permalink
Damien Katz added a comment - - edited
This is caused by Erlang attempting to replay lost time ticks, so that timers and receive timeouts have a chance to run. Erlang documentation say this can be disabled by using +c param when starting Erlang. The impact is that timers may fail and processes may timeout.

See the documentation for the http://www.erlang.org/doc/man/erl.html

Also see a similar Apache CouchDB bug, where we decided against using the +c option as it didn't seem to help:
https://issues.apache.org/jira/browse/COUCHDB-204
Show
Damien Katz added a comment - - edited This is caused by Erlang attempting to replay lost time ticks, so that timers and receive timeouts have a chance to run. Erlang documentation say this can be disabled by using +c param when starting Erlang. The impact is that timers may fail and processes may timeout. See the documentation for the http://www.erlang.org/doc/man/erl.html Also see a similar Apache CouchDB bug, where we decided against using the +c option as it didn't seem to help: https://issues.apache.org/jira/browse/COUCHDB-204
Hide
Permalink
Damien Katz added a comment -
Aleksey, it might be that just have to live with this issue, as I am not sure +c fix will fix this problem, and it might cause new problems.
Show
Damien Katz added a comment - Aleksey, it might be that just have to live with this issue, as I am not sure +c fix will fix this problem, and it might cause new problems.
Hide
Permalink
Aleksey Kondratenko added a comment -
You want to me to try it? My understanding of docs is that +c only affects clock not timers. But perhaps their docs is not clear.
Show
Aleksey Kondratenko added a comment - You want to me to try it? My understanding of docs is that +c only affects clock not timers. But perhaps their docs is not clear.
Hide
Permalink
Matt Ingenthron added a comment -
I don't think we can "live with this issue", as it leads to a pretty crappy developer experience. If +c doesn't work, can we escalate to someone to request a fix? Given erlang's desire to expand deployments and use cases, I'd think there'd be desire to provide some kind of fix.
Show
Matt Ingenthron added a comment - I don't think we can "live with this issue", as it leads to a pretty crappy developer experience. If +c doesn't work, can we escalate to someone to request a fix? Given erlang's desire to expand deployments and use cases, I'd think there'd be desire to provide some kind of fix.
Hide
Permalink
Farshid Ghods added a comment -
Aliaksey,

can you please update the ticket ?
Show
Farshid Ghods added a comment - Aliaksey, can you please update the ticket ?
Hide
Permalink
Aleksey Kondratenko added a comment -
update?

That's something we don't think we can fix very easily. That's all update I have.
Show
Aleksey Kondratenko added a comment - update? That's something we don't think we can fix very easily. That's all update I have.
Hide
Permalink
Aleksey Kondratenko added a comment -
Ah. In private discussions with Dipti and Damien I've mentioned alternative approach.

We can have something else babysit memcached & erlang VM. So that we could freely restart erlang VM after suspend. That also have some potential good effect on windows where it may allow node renaming at last. We have a ticket 'ported' from pivotal story for that
Show
Aleksey Kondratenko added a comment - Ah. In private discussions with Dipti and Damien I've mentioned alternative approach. We can have something else babysit memcached & erlang VM. So that we could freely restart erlang VM after suspend. That also have some potential good effect on windows where it may allow node renaming at last. We have a ticket 'ported' from pivotal story for that
Hide
Permalink
Dipti Borkar added a comment -
How major would the changes be and who needs to be involved in the design discussions?
Show
Dipti Borkar added a comment - How major would the changes be and who needs to be involved in the design discussions?
Hide
Permalink
Aleksey Kondratenko added a comment -
I don't feel it's major. But it's more than 1 day of labor. Most likely around 5 work days. Because it will involve keeping cluster_run and windows working in addition to 'most production' GNU/Linux initscripts. The piece that'll run stuff will likely be simple and mostly extracted from current ns_server code. I think it will be erlang. Just without any significant use of timeouts.
Show
Aleksey Kondratenko added a comment - I don't feel it's major. But it's more than 1 day of labor. Most likely around 5 work days. Because it will involve keeping cluster_run and windows working in addition to 'most production' GNU/Linux initscripts. The piece that'll run stuff will likely be simple and mostly extracted from current ns_server code. I think it will be erlang. Just without any significant use of timeouts.
Hide
Permalink
Aleksey Kondratenko added a comment -
And we still need to figure out how we'll detect wakeup from suspend and deciding respawning of ns_server. That needs a bit more thinking or even a bit of brainstorming.

Alternative o fcourse is to try to look at erlang vm source. Initial dive should not take more then 2-3 hours. We also have option of asking help of Erlang Solution folks.
Show
Aleksey Kondratenko added a comment - And we still need to figure out how we'll detect wakeup from suspend and deciding respawning of ns_server. That needs a bit more thinking or even a bit of brainstorming. Alternative o fcourse is to try to look at erlang vm source. Initial dive should not take more then 2-3 hours. We also have option of asking help of Erlang Solution folks.
Hide
Permalink
Dipti Borkar added a comment -
Acc. to Aliaksey, this will also help fix IP address issues on windows. Given both these are major adoption issues, I'm making this bug a blocker.
Show
Dipti Borkar added a comment - Acc. to Aliaksey, this will also help fix IP address issues on windows. Given both these are major adoption issues, I'm making this bug a blocker.
Hide
Permalink
Steve Yen added a comment -
moved to 2.0.2 per bug-scrub. potential VM-separation thing?
Show
Steve Yen added a comment - moved to 2.0.2 per bug-scrub. potential VM-separation thing?
Hide
Permalink
Farshid Ghods added a comment -
dupe of 6737 ?
Show
Farshid Ghods added a comment - dupe of 6737 ?
Hide
Permalink
Aleksey Kondratenko added a comment -
no. 6737 is about perf/tuning aspect of separating couch out of us
Show
Aleksey Kondratenko added a comment - no. 6737 is about perf/tuning aspect of separating couch out of us
Hide
Permalink
Maria McDuff added a comment -
per bug scrub: alk will try to get this in.
Show
Maria McDuff added a comment - per bug scrub: alk will try to get this in.
Hide
Permalink
Aleksey Kondratenko added a comment -
Current understanding is it's caused by timer's behavior of ticking all ticks that need to occur between previous tick time and now. So if "now" suddenly jumps we'll have tons of ticks happening at about same time.

Current fix is to fix timer module to skip those "accumulated" ticks if necessary.
Show
Aleksey Kondratenko added a comment - Current understanding is it's caused by timer's behavior of ticking all ticks that need to occur between previous tick time and now. So if "now" suddenly jumps we'll have tons of ticks happening at about same time. Current fix is to fix timer module to skip those "accumulated" ticks if necessary.
Hide
Permalink
Aleksey Kondratenko added a comment -
Fix merged http://review.couchbase.org/#/c/25551/ for 2.0.2 release
Show
Aleksey Kondratenko added a comment - Fix merged http://review.couchbase.org/#/c/25551/ for 2.0.2 release
Hide
Permalink
Dipti Borkar added a comment -
Maria, this is one , that we should verify on all platforms.
Show
Dipti Borkar added a comment - Maria, this is one , that we should verify on all platforms.
Hide
Permalink
Maria McDuff added a comment -
deep, pls verify (2.0.2) on all platforms -- very important. thanks.
Show
Maria McDuff added a comment - deep, pls verify (2.0.2) on all platforms -- very important. thanks.
Hide
Permalink
Maria McDuff added a comment -
QE:
-Deep to verify on all platforms, except MAC.
-Tony to verify on MAC only.

Show
Maria McDuff added a comment - QE: -Deep to verify on all platforms, except MAC. -Tony to verify on MAC only.

People

  • Assignee:
    Deepkaran Salooja
    Reporter:
    onyxmaster
Vote (2)
Watch (11)

Dates

  • Created:
    Updated:
    Resolved: