Persistence breakdown while writing stuff

Aleksey Kondratenko added a comment - 07/Jun/12 12:18 PM What is much much worse is that ep_commit_failed remains 0! There's no way ns_server can alert user about this condition.

Aleksey Kondratenko added a comment - 07/Jun/12 12:49 PM After killing everything and restarting I eventually get this: [ns_server:warn] [2012-06-07 10:46:58] [ n_3@10.17.2.163 :<0.18504.0>:compaction_daemon:do_chain_compactors:453] Compactor for vbucket `default/199` (pid [{type,vbucket},                                           {name,<<"default/199">>},                                           {important,false},                                           {fa,                                            {#Fun<compaction_daemon.17.86034732>,                                             [<<"default/199">>]}},                                           {on_success,                                            {#Fun<compaction_daemon.18.31588185>,                                             [73,129]}}]) terminated unexpectedly (ignoring this): {file_corruption,                                                                                                    <<"file corruption">>} [error_logger:error] [2012-06-07 10:46:58] [ n_3@10.17.2.163 :error_logger:ale_error_logger_handler:log_report:72] =========================CRASH REPORT=========================   crasher:     initial call: compaction_daemon:-spawn_vbucket_compactor/1-fun-0-/0     pid: <0.19674.0>     registered_name: []     exception exit: {file_corruption,<<"file corruption">>}       in function compaction_daemon:'-spawn_vbucket_compactor/1-fun-0-'/2     ancestors: [<0.18504.0>,<0.18502.0>,<0.18501.0>,compaction_daemon,                   <0.564.0>,ns_server_sup,ns_server_cluster_sup,<0.59.0>]     messages: []     links: [<0.18504.0>]     dictionary: []     trap_exit: true     status: running     heap_size: 2584     stack_size: 24     reductions: 1371   neighbours:

Aleksey Kondratenko added a comment - 07/Jun/12 12:53 PM Could be related to 32-bitness of my environment and to 199.couch.3 being slightly larger than 4g. irb(main):001:0> File.stat("couch/3/default/199.couch.3") => #<File::Stat dev=0x802, ino=25085693, mode=0100644, nlink=1, uid=0, gid=0, rdev=0x0, size=4368449627, blksize=4096, blocks=8532144, atime=2012-06-07 09:45:18 -0700, mtime=2012-06-07 10:38:57 -0700, ctime=2012-06-07 10:38:57 -0700> irb(main):002:0> _.size.to_s(16) => "10461405b"

Aleksey Kondratenko added a comment - 07/Jun/12 1:00 PM Note original data file size is 4g but with proper compression it's below 16 megs !

Jin Lim added a comment - 07/Jun/12 3:02 PM I see there are two issues here. 1. EP Engine not reporting a commit failure to ns_server and not update ep_commit_failed stat after encountering such error. I will open a separate bug to track how we may go about addressing this (design change). However, the observed behavior here is expected behavior under the current design. EP Engine's CouchKVStore could not save mutated docs because underlying storage, couchstore, returned a failure while accessing the file, 199.couch.3. (COUCHSTORE_ERROR_READ, -4). In this case, EP Engine would continue to retry the failed commit. This behavior applies regardless backend types, sqlite or couchstore.    2. Couchstore, encountered a file corruption (199.couch.3) during or after compaction. After that it kept returning the COUCHSTORE_ERROR_READ return to CouchKVStore for any access to the corrupted file. Couchstore developers should first take a look at the issue whether this is related to 32-bit system environment or potential file corruption issue during compaction. I suggest that this bug tracks the issue #2 above. Thanks.

Jin Lim added a comment - 07/Jun/12 3:03 PM Assign this to Aaron for his quick triage on this from couchstore perspective. Thanks.

Aaron Miller added a comment - 07/Jun/12 3:07 PM If there's corruption it's very possible it could have something to do with the issue i fixed that was tripping up XDCR http://review.couchbase.org/#change,16958 I can diagnose corruption issues a lot more readily if someone can get me a sample of a corrupted file.

Aleksey Kondratenko added a comment - 07/Jun/12 3:45 PM File _is_ attached already

Aaron Miller added a comment - 07/Jun/12 3:46 PM Oh, didn't catch that. Thanks :)

Aaron Miller added a comment - 07/Jun/12 5:56 PM Well I can tell you that the attached file has a corrupt b-tree node in the sequence index. (Bad checksum reading the node, btree_read.c line 22), 3 levels deep. Not sure how it happened, but that sort of indicates a bit of the file that was already laid down got written over. The corrupted chunk lives at pos 7689100 in the file, which does at least look like a file chunk starts there. The node is incidentally relatively old, (there are a lot of nodes and successful commits after it in the file), so it's hard to say when it was corrupted. Definitely happened -after- that latest compaction (as the compactor would've barfed if there was a corrupt node in the seq tree).

Aaron Miller added a comment - 15/Jun/12 2:28 PM That's as much as I can tell from the data collected here. If it's still happening have we identified steps to repeat it?

Aleksey Kondratenko added a comment - 15/Jun/12 2:40 PM I'll retry and post steps. I definitely didn't have any deletions or empty docs uploaded.

Aleksey Kondratenko added a comment - 15/Jun/12 3:22 PM I'm unable to reproduce this problem anymore