rebalance hanging in latest 2.0

Tommie McAfee added a comment - 11/Jun/12 4:25 PM I believe rebalance is hung, which may be a separate issue. However, this is the first time I've seen indexing affected by this type of behavior.

Filipe Manana added a comment - 11/Jun/12 5:57 PM Where's the screenshot?

Tommie McAfee added a comment - 11/Jun/12 7:49 PM in diags of current run of this test I see: [ns_server:error] [2012-06-11 20:25:14] [ ns_1@10.2.2.60 :<0.28654.2>:ns_orchestrator:rebalance_progress:151] Couldn't talk to orchestrator: {exit,                                 {timeout,                                  {gen_fsm,sync_send_event,                                   [{global,ns_orchestrator},                                    rebalance_progress,2000]}}} [ns_server:error] [2012-06-11 20:25:14] [ ns_1@10.2.2.60 :<0.11010.3>:ns_orchestrator:rebalance_progress:151] Couldn't talk to orchestrator: {exit,                                 {timeout,                                  {gen_fsm,sync_send_event,                                   [{global,ns_orchestrator},                                    rebalance_progress,2000]}}} [rebalance:error] [2012-06-11 20:25:37] [ ns_1@10.2.2.60 :<0.12914.3>:ns_rebalancer:do_wait_buckets_shutdown:171] Failed to wait deletion of some buckets on some nodes: [{' ns_1@10.2.2.60 ',                                                          {'EXIT',                                                           {old_buckets_shutdown_wait_failed,                                                            ["default"]}}}]

Andrei Baranouski added a comment - 11/Jun/12 9:23 PM additional screenshot that can be useful: there no any data, no bucket, but indexing view hangs

Andrei Baranouski added a comment - 11/Jun/12 9:24 PM screenshot after cleaning cluster in teardoun

Aliaksey Artamonau added a comment - 12/Jun/12 10:59 AM Tommie, the errors you mentioned in you comment are unrelated. Please file another ticket.

Aliaksey Artamonau added a comment - 12/Jun/12 11:57 AM Filipe, I see couch_set_view:set_partition_states call stuck on 10.2.2.65: [views:info] [2012-06-11 17:02:05] [ ns_1@10.2.2.65 :'capi_set_view_manager-default':capi_set_view_manager:apply_ddoc_map:418] Calling couch_set_view:set_partition_states([<<"default">>,                                              <<"_design/dev_test_view-6b38a71">>,                                              "\\]^_`abcdefghijkl",[],[]]) [couchdb:info] [2012-06-11 17:02:05] [ ns_1@10.2.2.65 :<0.3508.1>:couch_log:info:39] Stopping updater for set view `default`, main group `_design/dev_test_view-6b38a71` And I can't find a similar message saying that updater was started again. Here's the stacktrace of the <0.3508.1>:  {<0.3508.1>,                       [{registered_name,[]},                        {status,waiting},                        {initial_call,{proc_lib,init_p,5}},                        {backtrace,                         [<<"Program counter: 0x00002aaab3695e50 (couch_set_view_group:stop_updater/1 + 712)">>,                          <<"CP: 0x0000000000000000 (invalid)">>,                          <<"arity = 0">>,<<>>,                          <<"x00002aaab1a16b78 Return addr 0x00002aaab368ee18 (couch_set_view_group:update_partition_st">>,                          <<"y(0) []">>,<<"y(1) []">>,                          <<"y(2) []">>,<<"y(3) []">>,                          <<"y(4) []">>,<<"y(5) []">>,                          <<"y(6) []">>,<<"y(7) []">>,                          <<"y(8) []">>,                          <<"(9) {state,{\"/opt/couchbase/var/lib/couchbase/data/\",<<7 bytes>>,{set_view_group,<<16 ">>,                          <<"y(10) <0.5285.1>">>,                          <<"(11) {set_view_group,<<16 bytes>>,<0.3511.1>,<<7 bytes>>,<<29 bytes>>,[],[{set_view,0,[">>,                          <<"(12) {\"/opt/couchbase/var/lib/couchbase/data/\",<<7 bytes>>,{set_view_group,<<16 bytes>>">>,                          <<>>,                          <<"x00002aaab1a16be8 Return addr 0x00002aaab3684b60 (couch_set_view_group:handle_call/3 + 254">>,                          <<"y(0) []">>,<<"y(1) []">>,                          <<"y(2) []">>,                          <<"(3) {state,{\"/opt/couchbase/var/lib/couchbase/data/\",<<7 bytes>>,{set_view_group,<<16 ">>,                          <<"y(4) []">>,<<"y(5) []">>,                          <<"y(6) \"\\]^_`abcdefghijkl\"">>,<<>>,                          <<"0x00002aaab1a16c28 Return addr 0x00002b30edb7af30 (gen_server:handle_msg/5 + 272)">>,                          <<"(0) {state,{\"/opt/couchbase/var/lib/couchbase/data/\",<<7 bytes>>,{set_view_group,<<16 ">>,                          <<"y(1) Catch 0x00002aaab3684bd0 (couch_set_view_group:handle_call/3 + 25584)">>,                          <<>>,                          <<"0x00002aaab1a16c40 Return addr 0x00002b30edb277d0 (proc_lib:init_p_do_apply/3 + 56)">>,                          <<"y(0) couch_set_view_group">>,                          <<"(1) {state,{\"/opt/couchbase/var/lib/couchbase/data/\",<<7 bytes>>,{set_view_group,<<16 ">>,                          <<"y(2) <0.3508.1>">>,<<"y(3) <0.3507.1>">>,                          <<"y(4) {set_state,\"\\]^_`abcdefghijkl\",[],[]}">>,                          <<"y(5) {<0.3305.1>,#Ref<0.0.7.235019>}">>,                          <<"y(6) Catch 0x00002b30edb7af30 (gen_server:handle_msg/5 + 272)">>,                          <<>>,                          <<"0x00002aaab1a16c80 Return addr 0x00000000008715b8 (<terminate process normally>)">>,                          <<"y(0) Catch 0x00002b30edb277f0 (proc_lib:init_p_do_apply/3 + 88)">>,                          <<>>]},                        {error_handler,error_handler},                        {garbage_collection,                         [{min_bin_vheap_size,46368},                          {min_heap_size,233},                          {fullsweep_after,65535},                          {minor_gcs,11}]},                        {heap_size,6765},                        {total_heap_size,17711},                        {links,[<0.3516.1>,<0.5285.1>,<0.211.0>]},                        {memory,145240},                        {message_queue_len,35},                        {reductions,153159},                        {trap_exit,true}]}, Approximately at the same time compaction daemon fetches data size for different groups but gets restarted because of config change (this is actually kind of a bug). Can there be some interference?

Filipe Manana added a comment - 12/Jun/12 6:06 PM Thanks Aliaksey. Fixed the deadlock on my side: http://review.couchbase.org/#change,17116 (merged). Can I close this ticket or do you want to reuse it for the issues you mentioned?

Aliaksey Artamonau added a comment - 12/Jun/12 6:08 PM I'll file other tickets were appropriate.

Thuan Nguyen added a comment - 12/Jun/12 8:08 PM Integrated in github-couchdb-preview #422 (See [ http://qa.hq.northscale.net/job/github-couchdb-preview/422/ ])      MB-5511 Avoid deadlock between updater and view group (Revision 48a45c6ca25ab58f9323a313f6cca5ebe21d6c4a)      Result = SUCCESS Filipe David Borba Manana : Files : * src/couch_set_view/src/couch_set_view_updater.erl * src/couch_set_view/src/couch_set_view_group.erl