Rebalance exited with reason {mover_failed,killed}

Steve Yen added a comment - 12/Nov/12 1:36 PM bug-scrub / alk says: feels like duplicate of similar bugs

Aleksey Kondratenko added a comment - 14/Nov/12 11:22 AM I'm going to add a bit more diagnostics to try to pin point potential issue specifically with mb_master. Because somehow timeouts seemingly affect only mb_master in this case.

Steve Yen added a comment - 15/Nov/12 1:17 PM more diagnostics merged

Steve Yen added a comment - 15/Nov/12 1:18 PM Ronnie, Can you help re-run this with alk's extra diagnostics?

Ronnie Sun added a comment - 15/Nov/12 1:57 PM alk, can you point me to the build number or the change you added for extra diags. I want to make sure picking up the right build.

Aleksey Kondratenko added a comment - 15/Nov/12 3:20 PM I'm not tracking specific build numbers but I'm adamant latest build has those changes

Ronnie Sun added a comment - 15/Nov/12 3:25 PM picked 1952

Steve Yen added a comment - 16/Nov/12 1:35 PM Ronnie - any update on this? Thx, steve

Ronnie Sun added a comment - 16/Nov/12 2:17 PM https://s3.amazonaws.com/bugdb/jira/MB-7152/diag-nov-15.gz

Aleksey Kondratenko added a comment - 16/Nov/12 5:30 PM Latest diags actually have evidence of running out of disk space: 2012-11-16 05:16:49.399 menelaus_web_alerts_srv:1:info:message( ns_1@10.2.1.65 ) - Approaching full disk warning. Usage of disk "/data2" on node "10.2.1.65" is around 91%. 2012-11-16 07:19:24.699 compaction_daemon:0:critical:message( ns_1@10.2.1.65 ) - Cannot compact view default/_design/A/main: the estimated necessary disk space is about 6904004384 bytes but the currently available disk space is 0 bytes.

Aleksey Kondratenko added a comment - 16/Nov/12 5:35 PM But other signs is what we wanted: [ns_server:debug,2012-11-15T18:35:39.655, ns_1@10.2.1.66 :compaction_daemon<0.620.0>:compaction_daemon:handle_info:309]Finished compaction iteration. [ns_server:debug,2012-11-15T18:35:39.655, ns_1@10.2.1.66 :compaction_daemon<0.620.0>:compaction_daemon:schedule_next_compaction:1199]Finished compaction too soon. Next run will be in 30s [stats:warn,2012-11-15T18:36:08.735, ns_1@10.2.1.66 :system_stats_collector<0.611.0>:system_stats_collector:handle_info:133]lost 2 ticks [ns_server:warn,2012-11-15T18:36:08.736, ns_1@10.2.1.66 :mb_master<0.555.0>:mb_master:handle_info:204]Skipped 12 heartbeats [user:info,2012-11-15T18:36:08.741, ns_1@10.2.1.66 :mb_master<0.555.0>:mb_master:handle_info:219]Haven't heard from a higher priority node or a master, so I'm taking over. So looks like CPU was taken away from mb_master for about 24 seconds. Or there's some issue with erlang timers. It's still a bit mystery why we're not seeing this issue elsewhere. But leading guess is master is both latency sensitive _and_ is constantly exercised. Perhaps a bit more often than say ns_config or nodes_disco

Aleksey Kondratenko added a comment - 16/Nov/12 5:36 PM Pinging Aliaksey

Pavel Paulau added a comment - 18/Nov/12 9:37 PM - edited Size of "data2" partition is only 250GB (SSD, though) and it's used for index files. During rebalance we easily eat all space. For further tests we should use smaller data sets.

Steve Yen added a comment - 19/Nov/12 1:32 PM bug-scrub - no extra experiment ideas at the moment.

Steve Yen added a comment - 20/Nov/12 1:37 PM bug-scrub - to 2.0.1

Aleksey Kondratenko added a comment - 02/Jan/13 5:03 PM MB-7182