rebalance is hung at 100% when rebalancing 6 nodes

Farshid Ghods added a comment - 23/Jan/12 7:59 PM the cluster is running here now in case you want to have a look in the next couple of hours http://ec2-23-20-19-84.compute-1.amazonaws.com:8091/index.html#sec=servers&dev_subset=subset_dev

Aliaksey Artamonau added a comment - 23/Jan/12 8:18 PM capi_set_view_manager sleeps on couch_set_view call on 10.34.78.23: {backtrace,  [<<"Program counter: 0x00002aaaabed77a8 (gen:do_call/4 + 576)">>,   <<"CP: 0x0000000000000000 (invalid)">>,   <<"arity = 0">>,<<>>,   <<"0x00002aaaad42aec0 Return addr 0x00002aaaaabc91e8 (gen_server:call/3 + 128)">>,   <<"y(0) #Ref<0.0.19.22148>">>,   <<"y(1) ' ns_1@10.34.78.23 '">>,<<"y(2) []">>,   <<"y(3) infinity">>,   <<"(4) {get_group_server,<<7 bytes>>,{set_view_group,<<16 bytes>>,nil,<<7 bytes>>,<<15 by">>,   <<"y(5) '$gen_call'">>,<<"y(6) <0.658.0>">>,   <<>>,   <<"x00002aaaad42af00 Return addr 0x00002aaaad6ba858 (couch_set_view:get_group_server/2 + 128)">>,   <<"y(0) infinity">>,   <<"(1) {get_group_server,<<7 bytes>>,{set_view_group,<<16 bytes>>,nil,<<7 bytes>>,<<15 by">>,   <<"y(2) couch_set_view">>,   <<"y(3) Catch 0x00002aaaaabc91e8 (gen_server:call/3 + 128)">>,   <<>>,   <<"0x00002aaaad42af28 Return addr 0x00002aaaad6ba490 (couch_set_view:define_group/3 + 136)">>,   <<>>,   <<"x00002aaaad42af30 Return addr 0x00002aaab12929f0 (capi_set_view_manager:define_group/4 + 3">>,   <<"y(0) {set_view_params,256,[],[],true}">>,<<>>,   <<"x00002aaaad42af40 Return addr 0x00002aaab1292b28 (capi_set_view_manager:maybe_define_group">>,   <<"y(0) Catch 0x00002aaab1292a28 (capi_set_view_manager:define_group/4 + 408)">>,   <<"y(1) []">>,   <<"(2) [{active,[129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,">>,   <<"y(3) <<15 bytes>>">>,   <<"y(4) \"default\"">>,<<>>,   <<"0x00002aaaad42af70 Return addr 0x00002aaaacc691e8 (sets:fold_bucket/3 + 120)">>,   <<"y(0) Catch 0x00002aaab1292b48 (capi_set_view_manager:maybe_define_group/4 + 96)">>,   <<>>,   <<"0x00002aaaad42af80 Return addr 0x00002aaaacc690d0 (sets:fold_seg/4 + 184)">>,   <<"y(0) #Fun<capi_set_view_manager.17.110924455>">>,   <<"y(1) []">>,<<>>,   <<"0x00002aaaad42af98 Return addr 0x00002aaaacc68f78 (sets:fold_segs/4 + 200)">>,   <<"y(0) 14">>,   <<"y(1) {[],[],[],[],[],[],[],[<<15 bytes>>],[],[],[],[],[],[<<15 bytes>>],[],[]}">>,   <<"y(2) #Fun<capi_set_view_manager.17.110924455>">>,   <<>>,   <<"x00002aaaad42afb8 Return addr 0x00002aaab1293d70 (capi_set_view_manager:apply_current_map/">>,   <<"y(0) 1">>,   <<"y(1) {{[],[],[],[],[],[],[],[<<15 bytes>>],[],[],[],[],[],[<<15 bytes>>],[],[]}}">>,   <<"y(2) #Fun<capi_set_view_manager.17.110924455>">>,   <<>>,   <<"0x00002aaaad42afd8 Return addr 0x00002aaab1290448 (capi_set_view_manager:init/1 + 800)">>,   <<"(0) [{active,[129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,">>,   <<"(1) {dict,256,52,64,32,260,156,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[[0">>,   <<"(2) [{sasl_password,[]},{num_replicas,1},{ram_quota,2865758208},{auth_type,sasl},{type">>,   <<"(3) {set,2,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],">>,   <<"(4) {dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[]">>,   <<"y(5) <0.16122.2>">>,   <<"y(6) \"default\"">>,<<>>,   <<"0x00002aaaad42b018 Return addr 0x00002aaaaabca008 (gen_server:init_it/6 + 376)">>,   <<>>,   <<"0x00002aaaad42b020 Return addr 0x00002aaaabef2758 (proc_lib:init_p_do_apply/3 + 56)">>,   <<"y(0) Catch 0x00002aaaaabca008 (gen_server:init_it/6 + 376)">>,   <<"y(1) 'capi_set_view_manager-default'">>,   <<"y(2) []">>,   <<"y(3) capi_set_view_manager">>,   <<"y(4) {local,'capi_set_view_manager-default'}">>,   <<"y(5) <0.10682.0>">>,   <<"y(6) <0.10682.0>">>,<<>>,   <<"0x00002aaaad42b060 Return addr 0x000000000088e318 (<terminate process normally>)">>,   <<"y(0) Catch 0x00002aaaabef2778 (proc_lib:init_p_do_apply/3 + 88)">>,   <<>>]}

Filipe Manana added a comment - 24/Jan/12 4:51 PM Some changes already merged might have fixed this, like http://review.couchbase.org/#change,12600 for example. I wasn't able to reproduce this manually yet.

Aliaksey Artamonau added a comment - 24/Jan/12 7:43 PM Does not happen to me anymore.

Filipe Manana added a comment - 26/Jan/12 12:21 PM Closing this. Reopen if someone is able to reproduce this with latest code/builds.

Farshid Ghods added a comment - 26/Jan/12 5:54 PM happened while i was running similar tests using cluster_run modified the view functionl twice when rebalance was in progress and there was a client getting vieews using stale=false

Farshid Ghods added a comment - 26/Jan/12 5:56 PM uploaded reopened-the-bug-diags.zip

Aliaksey Artamonau added a comment - 26/Jan/12 6:17 PM And again capi_set_view_manager stuck on a call to couch_set_view:  {<0.1670.13>,                       [{registered_name,'capi_set_view_manager-default'},                        {status,waiting},                        {initial_call,{proc_lib,init_p,5}},                        {backtrace,                         [<<"Program counter: 0x00000000110c23a8 (gen:do_call/4 + 576)">>,                          <<"CP: 0x0000000000000000 (invalid)">>,                          <<"arity = 0">>,<<>>,                          <<"0x000000001b6ad9d0 Return addr 0x0000000011250b58 (gen_server:call/3 + 128)">>,                          <<"y(0) #Ref<0.0.86.53547>">>,                          <<"y(1) ' n_0@10.17.1.175 '">>,<<"y(2) []">>,                          <<"y(3) infinity">>,                          <<"(4) {get_group_server,<<7 bytes>>,{set_view_group,<<16 bytes>>,nil,<<7 bytes>>,<<14 by">>,                          <<"y(5) '$gen_call'">>,<<"y(6) <0.194.0>">>,                          <<>>,                          <<"x000000001b6ada10 Return addr 0x0000000014dcd938 (couch_set_view:get_group_server/2 + 128)">>,                          <<"y(0) infinity">>,                          <<"(1) {get_group_server,<<7 bytes>>,{set_view_group,<<16 bytes>>,nil,<<7 bytes>>,<<14 by">>,                          <<"y(2) couch_set_view">>,                          <<"y(3) Catch 0x0000000011250b58 (gen_server:call/3 + 128)">>,                          <<>>,                          <<"0x000000001b6ada38 Return addr 0x0000000014dcd570 (couch_set_view:define_group/3 + 136)">>,                          <<>>,                          <<"0x000000001b6ada40 Return addr 0x00000000138cde18 (timer:tc/3 + 176)">>,                          <<"y(0) {set_view_params,256,[],[],true}">>,<<>>,                          <<"x000000001b6ada50 Return addr 0x00000000154614f8 (capi_set_view_manager:'-define_group/4-f">>,                          <<"y(0) []">>,                          <<"y(1) {1327,621819,590648}">>,<<>>,                          <<"x000000001b6ada68 Return addr 0x000000001545d870 (capi_set_view_manager:define_group/4 + 3">>,                          <<"y(0) []">>,                          <<"y(1) Catch 0x0000000015461670 (capi_set_view_manager:'-define_group/4-fun-0-'/3 + 768)">>,                          <<"y(2) <<14 bytes>>">>,                          <<"y(3) <<7 bytes>>">>,                          <<"y(4) {set_view_params,256,[],[],true}">>,<<>>,                          <<"x000000001b6ada98 Return addr 0x000000001545d9a8 (capi_set_view_manager:maybe_define_group">>,                          <<"y(0) Catch 0x000000001545d8a8 (capi_set_view_manager:define_group/4 + 432)">>,                          <<"y(1) []">>,                          <<"(2) [{active,[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,33,34,35,36,37,38,3">>,                          <<"y(3) <<14 bytes>>">>,                          <<"y(4) \"default\"">>,<<>>,                          <<"0x000000001b6adac8 Return addr 0x00000000121f65f0 (sets:fold_bucket/3 + 120)">>,                          <<"y(0) Catch 0x000000001545d9c8 (capi_set_view_manager:maybe_define_group/4 + 96)">>,                          <<>>,                          <<"0x000000001b6adad8 Return addr 0x00000000121f64d8 (sets:fold_seg/4 + 184)">>,                          <<"y(0) #Fun<capi_set_view_manager.21.65566525>">>,                          <<"y(1) []">>,<<>>,                          <<"0x000000001b6adaf0 Return addr 0x00000000121f6380 (sets:fold_segs/4 + 200)">>,                          <<"y(0) 12">>,                          <<"y(1) {[],[],[],[],[],[],[],[],[],[],[],[<<14 bytes>>],[],[],[],[]}">>,                          <<"y(2) #Fun<capi_set_view_manager.21.65566525>">>,                          <<>>,                          <<"x000000001b6adb10 Return addr 0x000000001545ec38 (capi_set_view_manager:apply_current_map/">>,                          <<"y(0) 1">>,                          <<"y(1) {{[],[],[],[],[],[],[],[],[],[],[],[<<14 bytes>>],[],[],[],[]}}">>,                          <<"y(2) #Fun<capi_set_view_manager.21.65566525>">>,                          <<>>,                          <<"0x000000001b6adb30 Return addr 0x000000001545b2b0 (capi_set_view_manager:init/1 + 800)">>,                          <<"(0) [{active,[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,33,34,35,36,37,38,3">>,                          <<"(1) {dict,256,52,64,32,260,156,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[[0">>,                          <<"(2) [{sasl_password,[]},{num_replicas,1},{ram_quota,2902458368},{auth_type,sasl},{type">>,                          <<"(3) {set,1,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],">>,                          <<"(4) {dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[]">>,                          <<"y(5) <0.1675.13>">>,                          <<"y(6) \"default\"">>,<<>>,                          <<"0x000000001b6adb70 Return addr 0x0000000011251978 (gen_server:init_it/6 + 376)">>,                          <<>>,                          <<"0x000000001b6adb78 Return addr 0x00000000110e0420 (proc_lib:init_p_do_apply/3 + 56)">>,                          <<"y(0) Catch 0x0000000011251978 (gen_server:init_it/6 + 376)">>,                          <<"y(1) 'capi_set_view_manager-default'">>,                          <<"y(2) []">>,                          <<"y(3) capi_set_view_manager">>,                          <<"y(4) {local,'capi_set_view_manager-default'}">>,                          <<"y(5) <0.1137.0>">>,<<"y(6) <0.1137.0>">>,                          <<>>,                          <<"0x000000001b6adbb8 Return addr 0x0000000010238388 (<terminate process normally>)">>,                          <<"y(0) Catch 0x00000000110e0440 (proc_lib:init_p_do_apply/3 + 88)">>,                          <<>>]}

Farshid Ghods added a comment - 29/Jan/12 11:59 AM seeing this again after doing a few stop/resume rebalance but in this case disk is not draining on one of the nodes so i guess rebalance is stuck because couchdb is waiting for some items tobe written to disk ?

Steve Yen added a comment - 30/Jan/12 6:25 PM Farshid believes this bug is dependent on MB-4693

Steve Yen added a comment - 03/Feb/12 6:41 PM Hi Filipe -- any news/progress on this one? Thx

Filipe Manana added a comment - 04/Feb/12 8:01 PM Steve, I think the QA team should retry reproducing this issue. Since this was reported, many code changes happened to help fixing this and other related issues.

Steve Yen added a comment - 06/Feb/12 7:28 PM (might be a dupe, too?)

Karan Kumar added a comment - 07/Feb/12 4:15 PM - edited This is Resolved. Rebalance is not handing anymore, but is considerable slow when views are indexing/compacting. Use case:- Loaded 2M items. 20 views, 20 view queries in parallel. Views start to compact/index when rebalance is in progress. Attaching the lastest diags, if anything interesting stands out.