Couchbase Server

mb_mnesia,ensure_schema timeout when restarting ns_server

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Critical Critical
  • Resolution: Incomplete
  • Affects Version/s: 1.7 GA, 1.7.2
  • Fix Version/s: None
  • Component/s: ns_server
  • Security Level: Public
  • Environment:
    1.7 GA

Description

users are run into this issue too often which i suspect has sth to do with the uptime or maybe an unclean shutdown of the server can result in this behavior.

asking customers to delete their statistics is not a very pleasent thing to do

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Farshid Ghods added a comment -
When a process is killed or dies, mnesia files might be left in a poor state that prevents restart.
Show
Farshid Ghods added a comment - When a process is killed or dies, mnesia files might be left in a poor state that prevents restart.
Hide
Permalink
Farshid Ghods added a comment -
the workaround is to remove all files under /opt/membase/var/lib/membase/mnesia/*.*
Show
Farshid Ghods added a comment - the workaround is to remove all files under /opt/membase/var/lib/membase/mnesia/*.*
Hide
Permalink
Aleksey Kondratenko added a comment -
fix merged
Show
Aleksey Kondratenko added a comment - fix merged
Hide
Permalink
Farshid Ghods added a comment -
reported by a user on 1.7.2
diags here : http://www63.zippyshare.com/v/19241756/file.html
Show
Farshid Ghods added a comment - reported by a user on 1.7.2 diags here : http://www63.zippyshare.com/v/19241756/file.html
Hide
Permalink
Aleksey Kondratenko added a comment -
thanks for raising this again.

Looks like our workaround is not working all the time. For now manual workaround of deleting mnesia files and starting ns_server again will work.

For future we'll get rid of mnesia completely.
Show
Aleksey Kondratenko added a comment - thanks for raising this again. Looks like our workaround is not working all the time. For now manual workaround of deleting mnesia files and starting ns_server again will work. For future we'll get rid of mnesia completely.
Hide
Permalink
Aleksey Kondratenko added a comment -
Dipti, maybe I'm missing something and stats-in-mnesia is not "unfixable crap". But my plan since before 1.7.0 was to replace saving stats in mnesia with much simpler periodic snapshotting into plain file. Because stats are not really precious thing.

I estimate this work to be around 1-2 days including thorough testing.

We also had some plans for storing stats in couch, but given KISS principle I think my simplified approach is what we need.

Please, decide when we want this work to be done.
Show
Aleksey Kondratenko added a comment - Dipti, maybe I'm missing something and stats-in-mnesia is not "unfixable crap". But my plan since before 1.7.0 was to replace saving stats in mnesia with much simpler periodic snapshotting into plain file. Because stats are not really precious thing. I estimate this work to be around 1-2 days including thorough testing. We also had some plans for storing stats in couch, but given KISS principle I think my simplified approach is what we need. Please, decide when we want this work to be done.
Hide
Permalink
Aleksey Kondratenko added a comment -
Let me explain "not really precious thing" better. Apparently it's too easily to misunderstand my original phrase as "we shouldn't save stats at all".

Mnesia does it's own journalling and crash recovery and durable commits and stuff like that. Or supposedly does.

My point was we don't need _that_ level of durability. That means that instead of trying to commit supposedly durable transaction via mnesia after each stats sample we'll accumulate changes and save them once or twice per minute. Saving itself (because it's not much data) can be made durable and cheap.

Show
Aleksey Kondratenko added a comment - Let me explain "not really precious thing" better. Apparently it's too easily to misunderstand my original phrase as "we shouldn't save stats at all". Mnesia does it's own journalling and crash recovery and durable commits and stuff like that. Or supposedly does. My point was we don't need _that_ level of durability. That means that instead of trying to commit supposedly durable transaction via mnesia after each stats sample we'll accumulate changes and save them once or twice per minute. Saving itself (because it's not much data) can be made durable and cheap.
Hide
Permalink
Daniel added a comment -
Attaching the relevant logs.
Show
Daniel added a comment - Attaching the relevant logs.
Hide
Permalink
Aleksey Kondratenko added a comment -
Invalidated by removal of mnesia
Show
Aleksey Kondratenko added a comment - Invalidated by removal of mnesia
Hide
Permalink
Dipti Borkar added a comment -
have you already removed mnesia?

If so is there any impact in the upgrade process ?
Show
Dipti Borkar added a comment - have you already removed mnesia? If so is there any impact in the upgrade process ?
Hide
Permalink
Aleksey Kondratenko added a comment -
The code was actually ready for 2.0.1 but sadly too late. So 2.0.2 got it early.

Impact on upgrade process is simple. People will lose their stats archive. I.e. we don't bother converting stats from mnesia to new format. But note that rolling upgrade already causes them to lose historical stats.
Show
Aleksey Kondratenko added a comment - The code was actually ready for 2.0.1 but sadly too late. So 2.0.2 got it early. Impact on upgrade process is simple. People will lose their stats archive. I.e. we don't bother converting stats from mnesia to new format. But note that rolling upgrade already causes them to lose historical stats.

People

  • Assignee:
    Aleksey Kondratenko
    Reporter:
    Farshid Ghods
Vote (1)
Watch (4)

Dates

  • Created:
    Updated:
    Resolved: