Couchbase Server

lack of fsyncs in view engine may lead to silent index corruption

Details

  • Type: Bug Bug
  • Status: Open Open
  • Priority: Critical Critical
  • Resolution: Unresolved
  • Affects Version/s: 2.0
  • Fix Version/s: .next
  • Component/s: view-engine
  • Security Level: Public
  • Labels:
    None

Description

SUBJ. Found out about this in discussion with Filipe about how views work.

If I understood correctly it doesn't fsync at all silently assuming that if there's valid header, then preceding data is valid as well. Which is clearly not true.

IMHO that's a massive blocker that needs to be fixed sooner rather than later.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Steve Yen added a comment -
bug-scrub -- assigned to yaseen
Show
Steve Yen added a comment - bug-scrub -- assigned to yaseen
Hide
Permalink
Aleksey Kondratenko added a comment -
Comment was made that this cannot be silent index corruption due to CRC-ing of all btree nodes. But my point still holds, we if there's data corruption we'll know at query time and people we'll have to experience down time to manually rebuild index.
Show
Aleksey Kondratenko added a comment - Comment was made that this cannot be silent index corruption due to CRC-ing of all btree nodes. But my point still holds, we if there's data corruption we'll know at query time and people we'll have to experience down time to manually rebuild index.
Hide
Permalink
Steve Yen added a comment -
per bug scrub
Show
Steve Yen added a comment - per bug scrub
Hide
Permalink
Farshid Ghods added a comment -
Deep and Iryna have tried a scenario where they rebooted the system and did not hit this issue.
Show
Farshid Ghods added a comment - Deep and Iryna have tried a scenario where they rebooted the system and did not hit this issue.
Hide
Permalink
Steve Yen added a comment -
to .next per bug-scrub.

QE reports that deep & iryna tried to reproduce this and couldn't yet.
Show
Steve Yen added a comment - to .next per bug-scrub. QE reports that deep & iryna tried to reproduce this and couldn't yet.
Hide
Permalink
Aleksey Kondratenko added a comment -
It appears that move to .next was based on same old "we cannot reproduce" logic. It appears that we continue to under-prioritize IMHO important bugs merely because it's hard to reproduce them.

Because with that logic we'll I'm sure will forever move it to next release. If we think we don't need to that, IMHO it would be better to just close it.
Show
Aleksey Kondratenko added a comment - It appears that move to .next was based on same old "we cannot reproduce" logic. It appears that we continue to under-prioritize IMHO important bugs merely because it's hard to reproduce them. Because with that logic we'll I'm sure will forever move it to next release. If we think we don't need to that, IMHO it would be better to just close it.
Hide
Permalink
Filipe Manana added a comment -
Due to crc checks for every object written to a file (btree nodes), it won't certainly be silent.
Show
Filipe Manana added a comment - Due to crc checks for every object written to a file (btree nodes), it won't certainly be silent.
Hide
Permalink
Aleksey Kondratenko added a comment - - edited
I agree. My earlier comment above (based on your's or Damien's verbal comment) has same information.

But not being silent doesn't mean we can simply close it (or IMHO downgrade or forget it). Do we know what exactly will happen if querying or updating view will suddenly detect corrupted index file ?
Show
Aleksey Kondratenko added a comment - - edited I agree. My earlier comment above (based on your's or Damien's verbal comment) has same information. But not being silent doesn't mean we can simply close it (or IMHO downgrade or forget it). Do we know what exactly will happen if querying or updating view will suddenly detect corrupted index file ?
Hide
Permalink
Andrew DePue added a comment -
We just ran into this, or something like it. We have a development cluster and lost power to the entire cluster at once (it was a dev cluster so we didn't have backup power). The Couchbase cluster _seemed_ to start OK, but accessing certain views would result in strange behavior... mostly timeouts without any error or any indication as to what the problem could be.
Show
Andrew DePue added a comment - We just ran into this, or something like it. We have a development cluster and lost power to the entire cluster at once (it was a dev cluster so we didn't have backup power). The Couchbase cluster _seemed_ to start OK, but accessing certain views would result in strange behavior... mostly timeouts without any error or any indication as to what the problem could be.
Hide
Permalink
Filipe Manana added a comment - - edited
If there's a corruption issue with a file (either view or database), view queries will return an explicit file_corruption error if the index file is corrupted. If the corruption is in a database file, the error is only returned in a query response if the query is of type stale=false. For all cases, the error (and a stack trace) are logged.

Did you saw such error in your case? Example:
http://www.couchbase.com/forums/thread/filecorruption-error-executing-view
Show
Filipe Manana added a comment - - edited If there's a corruption issue with a file (either view or database), view queries will return an explicit file_corruption error if the index file is corrupted. If the corruption is in a database file, the error is only returned in a query response if the query is of type stale=false. For all cases, the error (and a stack trace) are logged. Did you saw such error in your case? Example: http://www.couchbase.com/forums/thread/filecorruption-error-executing-view

People

  • Assignee:
    Rahim Yaseen
    Reporter:
    Aleksey Kondratenko
Vote (1)
Watch (6)

Dates

  • Created:
    Updated: