Details
-
Type:
Bug
-
Status:
Closed
-
Priority:
Blocker
-
Resolution: Fixed
-
Affects Version/s: 2.0-developer-preview-3
-
Fix Version/s: 2.0-developer-preview-4
-
Component/s: ns_server
-
Security Level: Public
-
Labels:None
Description
rebalance failed initially because the reduce function was invalid.
i deleted the design doc with the invalid reduce function and then continued the rebalance.
rebalance made progress but hung at 100%
-
Hide
- 10.1.2.94-8091-diag.txt-latest.zip
- 07/Feb/12 4:15 PM
- 1.93 MB
- Karan Kumar
-
- 10.1.2.94-8091-diag.txt 45.70 MB
-
Hide
- 10.1.2.95-8091-diag.txt-latest.zip
- 07/Feb/12 4:15 PM
- 856 kB
- Karan Kumar
-
- 10.1.2.95-8091-diag.txt 23.03 MB
-
Hide
- 10.1.2.96-8091-diag.txt-latest.zip
- 07/Feb/12 4:15 PM
- 711 kB
- Karan Kumar
-
- 10.1.2.96-8091-diag.txt 18.65 MB
-
Hide
- 10.114.237.71-8091-diag.txt.zip
- 23/Jan/12 7:59 PM
- 1.06 MB
- Farshid Ghods
-
- 10.114.237.71-8091-diag.txt 27.46 MB
-
Hide
- 10.116.114.110-8091-diag.txt.zip
- 23/Jan/12 8:00 PM
- 846 kB
- Farshid Ghods
-
- 10.116.114.110-8091-diag.txt 18.09 MB
-
Hide
- 10.202.101.172-8091-diag.txt.zip
- 23/Jan/12 8:00 PM
- 814 kB
- Farshid Ghods
-
- 10.202.101.172-8091-diag.txt 17.73 MB
-
Hide
- 10.34.78.23-8091-diag.txt.zip
- 23/Jan/12 8:01 PM
- 851 kB
- Farshid Ghods
-
- 10.34.78.23-8091-diag.txt 18.10 MB
-
Hide
- 10.39.87.138-8091-diag.txt.zip
- 23/Jan/12 8:02 PM
- 863 kB
- Farshid Ghods
-
- 10.39.87.138-8091-diag.txt 18.38 MB
-
Hide
- 10.99.35.239-8091-diag.txt.zip
- 23/Jan/12 8:02 PM
- 861 kB
- Farshid Ghods
-
- 10.99.35.239-8091-diag.txt 18.21 MB
-
Hide
- reopened-the-bug-diags.zip
- 26/Jan/12 5:56 PM
- 10.75 MB
- Farshid Ghods
-
- 127.0.0.1-9000-diag.txt.zip 2.72 MB
- 127.0.0.1-9001-diag.txt.zip 5.36 MB
- 127.0.0.1-9002-diag.txt.zip 2.35 MB
- 127.0.0.1-9003-diag.txt.zip 2.49 MB
-
- Screen Shot 2012-01-29 at 9.59.18 AM.png
- 59 kB
- 29/Jan/12 12:00 PM
-
- Screen Shot 2012-01-29 at 9.59.50 AM.png
- 75 kB
- 29/Jan/12 12:00 PM
Issue Links
- depends on
-
MB-4693
EPERF - disk write stuck during large scale write load
-
Activity
- All
- Comments
- Work Log
- History
- Activity
- Gerrit Reviews
Hide
Aliaksey Artamonau
added a comment -
capi_set_view_manager sleeps on couch_set_view call on 10.34.78.23:
{backtrace,
[<<"Program counter: 0x00002aaaabed77a8 (gen:do_call/4 + 576)">>,
<<"CP: 0x0000000000000000 (invalid)">>,
<<"arity = 0">>,<<>>,
<<"0x00002aaaad42aec0 Return addr 0x00002aaaaabc91e8 (gen_server:call/3 + 128)">>,
<<"y(0) #Ref<0.0.19.22148>">>,
<<"y(1) 'ns_1@10.34.78.23'">>,<<"y(2) []">>,
<<"y(3) infinity">>,
<<"(4) {get_group_server,<<7 bytes>>,{set_view_group,<<16 bytes>>,nil,<<7 bytes>>,<<15 by">>,
<<"y(5) '$gen_call'">>,<<"y(6) <0.658.0>">>,
<<>>,
<<"x00002aaaad42af00 Return addr 0x00002aaaad6ba858 (couch_set_view:get_group_server/2 + 128)">>,
<<"y(0) infinity">>,
<<"(1) {get_group_server,<<7 bytes>>,{set_view_group,<<16 bytes>>,nil,<<7 bytes>>,<<15 by">>,
<<"y(2) couch_set_view">>,
<<"y(3) Catch 0x00002aaaaabc91e8 (gen_server:call/3 + 128)">>,
<<>>,
<<"0x00002aaaad42af28 Return addr 0x00002aaaad6ba490 (couch_set_view:define_group/3 + 136)">>,
<<>>,
<<"x00002aaaad42af30 Return addr 0x00002aaab12929f0 (capi_set_view_manager:define_group/4 + 3">>,
<<"y(0) {set_view_params,256,[],[],true}">>,<<>>,
<<"x00002aaaad42af40 Return addr 0x00002aaab1292b28 (capi_set_view_manager:maybe_define_group">>,
<<"y(0) Catch 0x00002aaab1292a28 (capi_set_view_manager:define_group/4 + 408)">>,
<<"y(1) []">>,
<<"(2) [{active,[129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,">>,
<<"y(3) <<15 bytes>>">>,
<<"y(4) \"default\"">>,<<>>,
<<"0x00002aaaad42af70 Return addr 0x00002aaaacc691e8 (sets:fold_bucket/3 + 120)">>,
<<"y(0) Catch 0x00002aaab1292b48 (capi_set_view_manager:maybe_define_group/4 + 96)">>,
<<>>,
<<"0x00002aaaad42af80 Return addr 0x00002aaaacc690d0 (sets:fold_seg/4 + 184)">>,
<<"y(0) #Fun<capi_set_view_manager.17.110924455>">>,
<<"y(1) []">>,<<>>,
<<"0x00002aaaad42af98 Return addr 0x00002aaaacc68f78 (sets:fold_segs/4 + 200)">>,
<<"y(0) 14">>,
<<"y(1) {[],[],[],[],[],[],[],[<<15 bytes>>],[],[],[],[],[],[<<15 bytes>>],[],[]}">>,
<<"y(2) #Fun<capi_set_view_manager.17.110924455>">>,
<<>>,
<<"x00002aaaad42afb8 Return addr 0x00002aaab1293d70 (capi_set_view_manager:apply_current_map/">>,
<<"y(0) 1">>,
<<"y(1) {{[],[],[],[],[],[],[],[<<15 bytes>>],[],[],[],[],[],[<<15 bytes>>],[],[]}}">>,
<<"y(2) #Fun<capi_set_view_manager.17.110924455>">>,
<<>>,
<<"0x00002aaaad42afd8 Return addr 0x00002aaab1290448 (capi_set_view_manager:init/1 + 800)">>,
<<"(0) [{active,[129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,">>,
<<"(1) {dict,256,52,64,32,260,156,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[[0">>,
<<"(2) [{sasl_password,[]},{num_replicas,1},{ram_quota,2865758208},{auth_type,sasl},{type">>,
<<"(3) {set,2,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],">>,
<<"(4) {dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[]">>,
<<"y(5) <0.16122.2>">>,
<<"y(6) \"default\"">>,<<>>,
<<"0x00002aaaad42b018 Return addr 0x00002aaaaabca008 (gen_server:init_it/6 + 376)">>,
<<>>,
<<"0x00002aaaad42b020 Return addr 0x00002aaaabef2758 (proc_lib:init_p_do_apply/3 + 56)">>,
<<"y(0) Catch 0x00002aaaaabca008 (gen_server:init_it/6 + 376)">>,
<<"y(1) 'capi_set_view_manager-default'">>,
<<"y(2) []">>,
<<"y(3) capi_set_view_manager">>,
<<"y(4) {local,'capi_set_view_manager-default'}">>,
<<"y(5) <0.10682.0>">>,
<<"y(6) <0.10682.0>">>,<<>>,
<<"0x00002aaaad42b060 Return addr 0x000000000088e318 (<terminate process normally>)">>,
<<"y(0) Catch 0x00002aaaabef2778 (proc_lib:init_p_do_apply/3 + 88)">>,
<<>>]}
{backtrace,
[<<"Program counter: 0x00002aaaabed77a8 (gen:do_call/4 + 576)">>,
<<"CP: 0x0000000000000000 (invalid)">>,
<<"arity = 0">>,<<>>,
<<"0x00002aaaad42aec0 Return addr 0x00002aaaaabc91e8 (gen_server:call/3 + 128)">>,
<<"y(0) #Ref<0.0.19.22148>">>,
<<"y(1) 'ns_1@10.34.78.23'">>,<<"y(2) []">>,
<<"y(3) infinity">>,
<<"(4) {get_group_server,<<7 bytes>>,{set_view_group,<<16 bytes>>,nil,<<7 bytes>>,<<15 by">>,
<<"y(5) '$gen_call'">>,<<"y(6) <0.658.0>">>,
<<>>,
<<"x00002aaaad42af00 Return addr 0x00002aaaad6ba858 (couch_set_view:get_group_server/2 + 128)">>,
<<"y(0) infinity">>,
<<"(1) {get_group_server,<<7 bytes>>,{set_view_group,<<16 bytes>>,nil,<<7 bytes>>,<<15 by">>,
<<"y(2) couch_set_view">>,
<<"y(3) Catch 0x00002aaaaabc91e8 (gen_server:call/3 + 128)">>,
<<>>,
<<"0x00002aaaad42af28 Return addr 0x00002aaaad6ba490 (couch_set_view:define_group/3 + 136)">>,
<<>>,
<<"x00002aaaad42af30 Return addr 0x00002aaab12929f0 (capi_set_view_manager:define_group/4 + 3">>,
<<"y(0) {set_view_params,256,[],[],true}">>,<<>>,
<<"x00002aaaad42af40 Return addr 0x00002aaab1292b28 (capi_set_view_manager:maybe_define_group">>,
<<"y(0) Catch 0x00002aaab1292a28 (capi_set_view_manager:define_group/4 + 408)">>,
<<"y(1) []">>,
<<"(2) [{active,[129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,">>,
<<"y(3) <<15 bytes>>">>,
<<"y(4) \"default\"">>,<<>>,
<<"0x00002aaaad42af70 Return addr 0x00002aaaacc691e8 (sets:fold_bucket/3 + 120)">>,
<<"y(0) Catch 0x00002aaab1292b48 (capi_set_view_manager:maybe_define_group/4 + 96)">>,
<<>>,
<<"0x00002aaaad42af80 Return addr 0x00002aaaacc690d0 (sets:fold_seg/4 + 184)">>,
<<"y(0) #Fun<capi_set_view_manager.17.110924455>">>,
<<"y(1) []">>,<<>>,
<<"0x00002aaaad42af98 Return addr 0x00002aaaacc68f78 (sets:fold_segs/4 + 200)">>,
<<"y(0) 14">>,
<<"y(1) {[],[],[],[],[],[],[],[<<15 bytes>>],[],[],[],[],[],[<<15 bytes>>],[],[]}">>,
<<"y(2) #Fun<capi_set_view_manager.17.110924455>">>,
<<>>,
<<"x00002aaaad42afb8 Return addr 0x00002aaab1293d70 (capi_set_view_manager:apply_current_map/">>,
<<"y(0) 1">>,
<<"y(1) {{[],[],[],[],[],[],[],[<<15 bytes>>],[],[],[],[],[],[<<15 bytes>>],[],[]}}">>,
<<"y(2) #Fun<capi_set_view_manager.17.110924455>">>,
<<>>,
<<"0x00002aaaad42afd8 Return addr 0x00002aaab1290448 (capi_set_view_manager:init/1 + 800)">>,
<<"(0) [{active,[129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,">>,
<<"(1) {dict,256,52,64,32,260,156,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[[0">>,
<<"(2) [{sasl_password,[]},{num_replicas,1},{ram_quota,2865758208},{auth_type,sasl},{type">>,
<<"(3) {set,2,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],">>,
<<"(4) {dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[]">>,
<<"y(5) <0.16122.2>">>,
<<"y(6) \"default\"">>,<<>>,
<<"0x00002aaaad42b018 Return addr 0x00002aaaaabca008 (gen_server:init_it/6 + 376)">>,
<<>>,
<<"0x00002aaaad42b020 Return addr 0x00002aaaabef2758 (proc_lib:init_p_do_apply/3 + 56)">>,
<<"y(0) Catch 0x00002aaaaabca008 (gen_server:init_it/6 + 376)">>,
<<"y(1) 'capi_set_view_manager-default'">>,
<<"y(2) []">>,
<<"y(3) capi_set_view_manager">>,
<<"y(4) {local,'capi_set_view_manager-default'}">>,
<<"y(5) <0.10682.0>">>,
<<"y(6) <0.10682.0>">>,<<>>,
<<"0x00002aaaad42b060 Return addr 0x000000000088e318 (<terminate process normally>)">>,
<<"y(0) Catch 0x00002aaaabef2778 (proc_lib:init_p_do_apply/3 + 88)">>,
<<>>]}
Show
Aliaksey Artamonau
added a comment - capi_set_view_manager sleeps on couch_set_view call on 10.34.78.23:
{backtrace,
[<<"Program counter: 0x00002aaaabed77a8 (gen:do_call/4 + 576)">>,
<<"CP: 0x0000000000000000 (invalid)">>,
<<"arity = 0">>,<<>>,
<<"0x00002aaaad42aec0 Return addr 0x00002aaaaabc91e8 (gen_server:call/3 + 128)">>,
<<"y(0) #Ref<0.0.19.22148>">>,
<<"y(1) ' ns_1@10.34.78.23 '">>,<<"y(2) []">>,
<<"y(3) infinity">>,
<<"(4) {get_group_server,<<7 bytes>>,{set_view_group,<<16 bytes>>,nil,<<7 bytes>>,<<15 by">>,
<<"y(5) '$gen_call'">>,<<"y(6) <0.658.0>">>,
<<>>,
<<"x00002aaaad42af00 Return addr 0x00002aaaad6ba858 (couch_set_view:get_group_server/2 + 128)">>,
<<"y(0) infinity">>,
<<"(1) {get_group_server,<<7 bytes>>,{set_view_group,<<16 bytes>>,nil,<<7 bytes>>,<<15 by">>,
<<"y(2) couch_set_view">>,
<<"y(3) Catch 0x00002aaaaabc91e8 (gen_server:call/3 + 128)">>,
<<>>,
<<"0x00002aaaad42af28 Return addr 0x00002aaaad6ba490 (couch_set_view:define_group/3 + 136)">>,
<<>>,
<<"x00002aaaad42af30 Return addr 0x00002aaab12929f0 (capi_set_view_manager:define_group/4 + 3">>,
<<"y(0) {set_view_params,256,[],[],true}">>,<<>>,
<<"x00002aaaad42af40 Return addr 0x00002aaab1292b28 (capi_set_view_manager:maybe_define_group">>,
<<"y(0) Catch 0x00002aaab1292a28 (capi_set_view_manager:define_group/4 + 408)">>,
<<"y(1) []">>,
<<"(2) [{active,[129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,">>,
<<"y(3) <<15 bytes>>">>,
<<"y(4) \"default\"">>,<<>>,
<<"0x00002aaaad42af70 Return addr 0x00002aaaacc691e8 (sets:fold_bucket/3 + 120)">>,
<<"y(0) Catch 0x00002aaab1292b48 (capi_set_view_manager:maybe_define_group/4 + 96)">>,
<<>>,
<<"0x00002aaaad42af80 Return addr 0x00002aaaacc690d0 (sets:fold_seg/4 + 184)">>,
<<"y(0) #Fun<capi_set_view_manager.17.110924455>">>,
<<"y(1) []">>,<<>>,
<<"0x00002aaaad42af98 Return addr 0x00002aaaacc68f78 (sets:fold_segs/4 + 200)">>,
<<"y(0) 14">>,
<<"y(1) {[],[],[],[],[],[],[],[<<15 bytes>>],[],[],[],[],[],[<<15 bytes>>],[],[]}">>,
<<"y(2) #Fun<capi_set_view_manager.17.110924455>">>,
<<>>,
<<"x00002aaaad42afb8 Return addr 0x00002aaab1293d70 (capi_set_view_manager:apply_current_map/">>,
<<"y(0) 1">>,
<<"y(1) {{[],[],[],[],[],[],[],[<<15 bytes>>],[],[],[],[],[],[<<15 bytes>>],[],[]}}">>,
<<"y(2) #Fun<capi_set_view_manager.17.110924455>">>,
<<>>,
<<"0x00002aaaad42afd8 Return addr 0x00002aaab1290448 (capi_set_view_manager:init/1 + 800)">>,
<<"(0) [{active,[129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,">>,
<<"(1) {dict,256,52,64,32,260,156,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[[0">>,
<<"(2) [{sasl_password,[]},{num_replicas,1},{ram_quota,2865758208},{auth_type,sasl},{type">>,
<<"(3) {set,2,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],">>,
<<"(4) {dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[]">>,
<<"y(5) <0.16122.2>">>,
<<"y(6) \"default\"">>,<<>>,
<<"0x00002aaaad42b018 Return addr 0x00002aaaaabca008 (gen_server:init_it/6 + 376)">>,
<<>>,
<<"0x00002aaaad42b020 Return addr 0x00002aaaabef2758 (proc_lib:init_p_do_apply/3 + 56)">>,
<<"y(0) Catch 0x00002aaaaabca008 (gen_server:init_it/6 + 376)">>,
<<"y(1) 'capi_set_view_manager-default'">>,
<<"y(2) []">>,
<<"y(3) capi_set_view_manager">>,
<<"y(4) {local,'capi_set_view_manager-default'}">>,
<<"y(5) <0.10682.0>">>,
<<"y(6) <0.10682.0>">>,<<>>,
<<"0x00002aaaad42b060 Return addr 0x000000000088e318 (<terminate process normally>)">>,
<<"y(0) Catch 0x00002aaaabef2778 (proc_lib:init_p_do_apply/3 + 88)">>,
<<>>]}
Hide
Filipe Manana
added a comment -
Some changes already merged might have fixed this, like
http://review.couchbase.org/#change,12600
for example. I wasn't able to reproduce this manually yet.
http://review.couchbase.org/#change,12600
for example. I wasn't able to reproduce this manually yet.
Show
Filipe Manana
added a comment - Some changes already merged might have fixed this, like
http://review.couchbase.org/#change,12600
for example. I wasn't able to reproduce this manually yet.
Show
Aliaksey Artamonau
added a comment - Does not happen to me anymore.
Hide
Filipe Manana
added a comment -
Closing this. Reopen if someone is able to reproduce this with latest code/builds.
Show
Filipe Manana
added a comment - Closing this. Reopen if someone is able to reproduce this with latest code/builds.
Hide
Farshid Ghods
added a comment -
happened while i was running similar tests using cluster_run
modified the view functionl twice when rebalance was in progress and there was a client getting vieews using stale=false
modified the view functionl twice when rebalance was in progress and there was a client getting vieews using stale=false
Show
Farshid Ghods
added a comment - happened while i was running similar tests using cluster_run
modified the view functionl twice when rebalance was in progress and there was a client getting vieews using stale=false
Show
Farshid Ghods
added a comment - uploaded reopened-the-bug-diags.zip
Hide
Aliaksey Artamonau
added a comment -
And again capi_set_view_manager stuck on a call to couch_set_view:
{<0.1670.13>,
[{registered_name,'capi_set_view_manager-default'},
{status,waiting},
{initial_call,{proc_lib,init_p,5}},
{backtrace,
[<<"Program counter: 0x00000000110c23a8 (gen:do_call/4 + 576)">>,
<<"CP: 0x0000000000000000 (invalid)">>,
<<"arity = 0">>,<<>>,
<<"0x000000001b6ad9d0 Return addr 0x0000000011250b58 (gen_server:call/3 + 128)">>,
<<"y(0) #Ref<0.0.86.53547>">>,
<<"y(1) 'n_0@10.17.1.175'">>,<<"y(2) []">>,
<<"y(3) infinity">>,
<<"(4) {get_group_server,<<7 bytes>>,{set_view_group,<<16 bytes>>,nil,<<7 bytes>>,<<14 by">>,
<<"y(5) '$gen_call'">>,<<"y(6) <0.194.0>">>,
<<>>,
<<"x000000001b6ada10 Return addr 0x0000000014dcd938 (couch_set_view:get_group_server/2 + 128)">>,
<<"y(0) infinity">>,
<<"(1) {get_group_server,<<7 bytes>>,{set_view_group,<<16 bytes>>,nil,<<7 bytes>>,<<14 by">>,
<<"y(2) couch_set_view">>,
<<"y(3) Catch 0x0000000011250b58 (gen_server:call/3 + 128)">>,
<<>>,
<<"0x000000001b6ada38 Return addr 0x0000000014dcd570 (couch_set_view:define_group/3 + 136)">>,
<<>>,
<<"0x000000001b6ada40 Return addr 0x00000000138cde18 (timer:tc/3 + 176)">>,
<<"y(0) {set_view_params,256,[],[],true}">>,<<>>,
<<"x000000001b6ada50 Return addr 0x00000000154614f8 (capi_set_view_manager:'-define_group/4-f">>,
<<"y(0) []">>,
<<"y(1) {1327,621819,590648}">>,<<>>,
<<"x000000001b6ada68 Return addr 0x000000001545d870 (capi_set_view_manager:define_group/4 + 3">>,
<<"y(0) []">>,
<<"y(1) Catch 0x0000000015461670 (capi_set_view_manager:'-define_group/4-fun-0-'/3 + 768)">>,
<<"y(2) <<14 bytes>>">>,
<<"y(3) <<7 bytes>>">>,
<<"y(4) {set_view_params,256,[],[],true}">>,<<>>,
<<"x000000001b6ada98 Return addr 0x000000001545d9a8 (capi_set_view_manager:maybe_define_group">>,
<<"y(0) Catch 0x000000001545d8a8 (capi_set_view_manager:define_group/4 + 432)">>,
<<"y(1) []">>,
<<"(2) [{active,[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,33,34,35,36,37,38,3">>,
<<"y(3) <<14 bytes>>">>,
<<"y(4) \"default\"">>,<<>>,
<<"0x000000001b6adac8 Return addr 0x00000000121f65f0 (sets:fold_bucket/3 + 120)">>,
<<"y(0) Catch 0x000000001545d9c8 (capi_set_view_manager:maybe_define_group/4 + 96)">>,
<<>>,
<<"0x000000001b6adad8 Return addr 0x00000000121f64d8 (sets:fold_seg/4 + 184)">>,
<<"y(0) #Fun<capi_set_view_manager.21.65566525>">>,
<<"y(1) []">>,<<>>,
<<"0x000000001b6adaf0 Return addr 0x00000000121f6380 (sets:fold_segs/4 + 200)">>,
<<"y(0) 12">>,
<<"y(1) {[],[],[],[],[],[],[],[],[],[],[],[<<14 bytes>>],[],[],[],[]}">>,
<<"y(2) #Fun<capi_set_view_manager.21.65566525>">>,
<<>>,
<<"x000000001b6adb10 Return addr 0x000000001545ec38 (capi_set_view_manager:apply_current_map/">>,
<<"y(0) 1">>,
<<"y(1) {{[],[],[],[],[],[],[],[],[],[],[],[<<14 bytes>>],[],[],[],[]}}">>,
<<"y(2) #Fun<capi_set_view_manager.21.65566525>">>,
<<>>,
<<"0x000000001b6adb30 Return addr 0x000000001545b2b0 (capi_set_view_manager:init/1 + 800)">>,
<<"(0) [{active,[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,33,34,35,36,37,38,3">>,
<<"(1) {dict,256,52,64,32,260,156,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[[0">>,
<<"(2) [{sasl_password,[]},{num_replicas,1},{ram_quota,2902458368},{auth_type,sasl},{type">>,
<<"(3) {set,1,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],">>,
<<"(4) {dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[]">>,
<<"y(5) <0.1675.13>">>,
<<"y(6) \"default\"">>,<<>>,
<<"0x000000001b6adb70 Return addr 0x0000000011251978 (gen_server:init_it/6 + 376)">>,
<<>>,
<<"0x000000001b6adb78 Return addr 0x00000000110e0420 (proc_lib:init_p_do_apply/3 + 56)">>,
<<"y(0) Catch 0x0000000011251978 (gen_server:init_it/6 + 376)">>,
<<"y(1) 'capi_set_view_manager-default'">>,
<<"y(2) []">>,
<<"y(3) capi_set_view_manager">>,
<<"y(4) {local,'capi_set_view_manager-default'}">>,
<<"y(5) <0.1137.0>">>,<<"y(6) <0.1137.0>">>,
<<>>,
<<"0x000000001b6adbb8 Return addr 0x0000000010238388 (<terminate process normally>)">>,
<<"y(0) Catch 0x00000000110e0440 (proc_lib:init_p_do_apply/3 + 88)">>,
<<>>]}
{<0.1670.13>,
[{registered_name,'capi_set_view_manager-default'},
{status,waiting},
{initial_call,{proc_lib,init_p,5}},
{backtrace,
[<<"Program counter: 0x00000000110c23a8 (gen:do_call/4 + 576)">>,
<<"CP: 0x0000000000000000 (invalid)">>,
<<"arity = 0">>,<<>>,
<<"0x000000001b6ad9d0 Return addr 0x0000000011250b58 (gen_server:call/3 + 128)">>,
<<"y(0) #Ref<0.0.86.53547>">>,
<<"y(1) 'n_0@10.17.1.175'">>,<<"y(2) []">>,
<<"y(3) infinity">>,
<<"(4) {get_group_server,<<7 bytes>>,{set_view_group,<<16 bytes>>,nil,<<7 bytes>>,<<14 by">>,
<<"y(5) '$gen_call'">>,<<"y(6) <0.194.0>">>,
<<>>,
<<"x000000001b6ada10 Return addr 0x0000000014dcd938 (couch_set_view:get_group_server/2 + 128)">>,
<<"y(0) infinity">>,
<<"(1) {get_group_server,<<7 bytes>>,{set_view_group,<<16 bytes>>,nil,<<7 bytes>>,<<14 by">>,
<<"y(2) couch_set_view">>,
<<"y(3) Catch 0x0000000011250b58 (gen_server:call/3 + 128)">>,
<<>>,
<<"0x000000001b6ada38 Return addr 0x0000000014dcd570 (couch_set_view:define_group/3 + 136)">>,
<<>>,
<<"0x000000001b6ada40 Return addr 0x00000000138cde18 (timer:tc/3 + 176)">>,
<<"y(0) {set_view_params,256,[],[],true}">>,<<>>,
<<"x000000001b6ada50 Return addr 0x00000000154614f8 (capi_set_view_manager:'-define_group/4-f">>,
<<"y(0) []">>,
<<"y(1) {1327,621819,590648}">>,<<>>,
<<"x000000001b6ada68 Return addr 0x000000001545d870 (capi_set_view_manager:define_group/4 + 3">>,
<<"y(0) []">>,
<<"y(1) Catch 0x0000000015461670 (capi_set_view_manager:'-define_group/4-fun-0-'/3 + 768)">>,
<<"y(2) <<14 bytes>>">>,
<<"y(3) <<7 bytes>>">>,
<<"y(4) {set_view_params,256,[],[],true}">>,<<>>,
<<"x000000001b6ada98 Return addr 0x000000001545d9a8 (capi_set_view_manager:maybe_define_group">>,
<<"y(0) Catch 0x000000001545d8a8 (capi_set_view_manager:define_group/4 + 432)">>,
<<"y(1) []">>,
<<"(2) [{active,[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,33,34,35,36,37,38,3">>,
<<"y(3) <<14 bytes>>">>,
<<"y(4) \"default\"">>,<<>>,
<<"0x000000001b6adac8 Return addr 0x00000000121f65f0 (sets:fold_bucket/3 + 120)">>,
<<"y(0) Catch 0x000000001545d9c8 (capi_set_view_manager:maybe_define_group/4 + 96)">>,
<<>>,
<<"0x000000001b6adad8 Return addr 0x00000000121f64d8 (sets:fold_seg/4 + 184)">>,
<<"y(0) #Fun<capi_set_view_manager.21.65566525>">>,
<<"y(1) []">>,<<>>,
<<"0x000000001b6adaf0 Return addr 0x00000000121f6380 (sets:fold_segs/4 + 200)">>,
<<"y(0) 12">>,
<<"y(1) {[],[],[],[],[],[],[],[],[],[],[],[<<14 bytes>>],[],[],[],[]}">>,
<<"y(2) #Fun<capi_set_view_manager.21.65566525>">>,
<<>>,
<<"x000000001b6adb10 Return addr 0x000000001545ec38 (capi_set_view_manager:apply_current_map/">>,
<<"y(0) 1">>,
<<"y(1) {{[],[],[],[],[],[],[],[],[],[],[],[<<14 bytes>>],[],[],[],[]}}">>,
<<"y(2) #Fun<capi_set_view_manager.21.65566525>">>,
<<>>,
<<"0x000000001b6adb30 Return addr 0x000000001545b2b0 (capi_set_view_manager:init/1 + 800)">>,
<<"(0) [{active,[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,33,34,35,36,37,38,3">>,
<<"(1) {dict,256,52,64,32,260,156,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[[0">>,
<<"(2) [{sasl_password,[]},{num_replicas,1},{ram_quota,2902458368},{auth_type,sasl},{type">>,
<<"(3) {set,1,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],">>,
<<"(4) {dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[]">>,
<<"y(5) <0.1675.13>">>,
<<"y(6) \"default\"">>,<<>>,
<<"0x000000001b6adb70 Return addr 0x0000000011251978 (gen_server:init_it/6 + 376)">>,
<<>>,
<<"0x000000001b6adb78 Return addr 0x00000000110e0420 (proc_lib:init_p_do_apply/3 + 56)">>,
<<"y(0) Catch 0x0000000011251978 (gen_server:init_it/6 + 376)">>,
<<"y(1) 'capi_set_view_manager-default'">>,
<<"y(2) []">>,
<<"y(3) capi_set_view_manager">>,
<<"y(4) {local,'capi_set_view_manager-default'}">>,
<<"y(5) <0.1137.0>">>,<<"y(6) <0.1137.0>">>,
<<>>,
<<"0x000000001b6adbb8 Return addr 0x0000000010238388 (<terminate process normally>)">>,
<<"y(0) Catch 0x00000000110e0440 (proc_lib:init_p_do_apply/3 + 88)">>,
<<>>]}
Show
Aliaksey Artamonau
added a comment - And again capi_set_view_manager stuck on a call to couch_set_view:
{<0.1670.13>,
[{registered_name,'capi_set_view_manager-default'},
{status,waiting},
{initial_call,{proc_lib,init_p,5}},
{backtrace,
[<<"Program counter: 0x00000000110c23a8 (gen:do_call/4 + 576)">>,
<<"CP: 0x0000000000000000 (invalid)">>,
<<"arity = 0">>,<<>>,
<<"0x000000001b6ad9d0 Return addr 0x0000000011250b58 (gen_server:call/3 + 128)">>,
<<"y(0) #Ref<0.0.86.53547>">>,
<<"y(1) ' n_0@10.17.1.175 '">>,<<"y(2) []">>,
<<"y(3) infinity">>,
<<"(4) {get_group_server,<<7 bytes>>,{set_view_group,<<16 bytes>>,nil,<<7 bytes>>,<<14 by">>,
<<"y(5) '$gen_call'">>,<<"y(6) <0.194.0>">>,
<<>>,
<<"x000000001b6ada10 Return addr 0x0000000014dcd938 (couch_set_view:get_group_server/2 + 128)">>,
<<"y(0) infinity">>,
<<"(1) {get_group_server,<<7 bytes>>,{set_view_group,<<16 bytes>>,nil,<<7 bytes>>,<<14 by">>,
<<"y(2) couch_set_view">>,
<<"y(3) Catch 0x0000000011250b58 (gen_server:call/3 + 128)">>,
<<>>,
<<"0x000000001b6ada38 Return addr 0x0000000014dcd570 (couch_set_view:define_group/3 + 136)">>,
<<>>,
<<"0x000000001b6ada40 Return addr 0x00000000138cde18 (timer:tc/3 + 176)">>,
<<"y(0) {set_view_params,256,[],[],true}">>,<<>>,
<<"x000000001b6ada50 Return addr 0x00000000154614f8 (capi_set_view_manager:'-define_group/4-f">>,
<<"y(0) []">>,
<<"y(1) {1327,621819,590648}">>,<<>>,
<<"x000000001b6ada68 Return addr 0x000000001545d870 (capi_set_view_manager:define_group/4 + 3">>,
<<"y(0) []">>,
<<"y(1) Catch 0x0000000015461670 (capi_set_view_manager:'-define_group/4-fun-0-'/3 + 768)">>,
<<"y(2) <<14 bytes>>">>,
<<"y(3) <<7 bytes>>">>,
<<"y(4) {set_view_params,256,[],[],true}">>,<<>>,
<<"x000000001b6ada98 Return addr 0x000000001545d9a8 (capi_set_view_manager:maybe_define_group">>,
<<"y(0) Catch 0x000000001545d8a8 (capi_set_view_manager:define_group/4 + 432)">>,
<<"y(1) []">>,
<<"(2) [{active,[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,33,34,35,36,37,38,3">>,
<<"y(3) <<14 bytes>>">>,
<<"y(4) \"default\"">>,<<>>,
<<"0x000000001b6adac8 Return addr 0x00000000121f65f0 (sets:fold_bucket/3 + 120)">>,
<<"y(0) Catch 0x000000001545d9c8 (capi_set_view_manager:maybe_define_group/4 + 96)">>,
<<>>,
<<"0x000000001b6adad8 Return addr 0x00000000121f64d8 (sets:fold_seg/4 + 184)">>,
<<"y(0) #Fun<capi_set_view_manager.21.65566525>">>,
<<"y(1) []">>,<<>>,
<<"0x000000001b6adaf0 Return addr 0x00000000121f6380 (sets:fold_segs/4 + 200)">>,
<<"y(0) 12">>,
<<"y(1) {[],[],[],[],[],[],[],[],[],[],[],[<<14 bytes>>],[],[],[],[]}">>,
<<"y(2) #Fun<capi_set_view_manager.21.65566525>">>,
<<>>,
<<"x000000001b6adb10 Return addr 0x000000001545ec38 (capi_set_view_manager:apply_current_map/">>,
<<"y(0) 1">>,
<<"y(1) {{[],[],[],[],[],[],[],[],[],[],[],[<<14 bytes>>],[],[],[],[]}}">>,
<<"y(2) #Fun<capi_set_view_manager.21.65566525>">>,
<<>>,
<<"0x000000001b6adb30 Return addr 0x000000001545b2b0 (capi_set_view_manager:init/1 + 800)">>,
<<"(0) [{active,[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,33,34,35,36,37,38,3">>,
<<"(1) {dict,256,52,64,32,260,156,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[[0">>,
<<"(2) [{sasl_password,[]},{num_replicas,1},{ram_quota,2902458368},{auth_type,sasl},{type">>,
<<"(3) {set,1,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[],">>,
<<"(4) {dict,0,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[],[]">>,
<<"y(5) <0.1675.13>">>,
<<"y(6) \"default\"">>,<<>>,
<<"0x000000001b6adb70 Return addr 0x0000000011251978 (gen_server:init_it/6 + 376)">>,
<<>>,
<<"0x000000001b6adb78 Return addr 0x00000000110e0420 (proc_lib:init_p_do_apply/3 + 56)">>,
<<"y(0) Catch 0x0000000011251978 (gen_server:init_it/6 + 376)">>,
<<"y(1) 'capi_set_view_manager-default'">>,
<<"y(2) []">>,
<<"y(3) capi_set_view_manager">>,
<<"y(4) {local,'capi_set_view_manager-default'}">>,
<<"y(5) <0.1137.0>">>,<<"y(6) <0.1137.0>">>,
<<>>,
<<"0x000000001b6adbb8 Return addr 0x0000000010238388 (<terminate process normally>)">>,
<<"y(0) Catch 0x00000000110e0440 (proc_lib:init_p_do_apply/3 + 88)">>,
<<>>]}
Hide
Farshid Ghods
added a comment -
seeing this again after doing a few stop/resume rebalance but in this case disk is not draining on one of the nodes so i guess rebalance is stuck because couchdb is waiting for some items tobe written to disk ?
Show
Farshid Ghods
added a comment - seeing this again after doing a few stop/resume rebalance but in this case disk is not draining on one of the nodes so i guess rebalance is stuck because couchdb is waiting for some items tobe written to disk ?
Hide
Filipe Manana
added a comment -
Steve, I think the QA team should retry reproducing this issue.
Since this was reported, many code changes happened to help fixing this and other related issues.
Since this was reported, many code changes happened to help fixing this and other related issues.
Show
Filipe Manana
added a comment - Steve, I think the QA team should retry reproducing this issue.
Since this was reported, many code changes happened to help fixing this and other related issues.
Hide
This is Resolved. Rebalance is not handing anymore, but is considerable slow when views are indexing/compacting.
Use case:-
Loaded 2M items. 20 views, 20 view queries in parallel. Views start to compact/index when rebalance is in progress.
Attaching the lastest diags, if anything interesting stands out.
Use case:-
Loaded 2M items. 20 views, 20 view queries in parallel. Views start to compact/index when rebalance is in progress.
Attaching the lastest diags, if anything interesting stands out.
Show
Karan Kumar
added a comment - - edited This is Resolved. Rebalance is not handing anymore, but is considerable slow when views are indexing/compacting.
Use case:-
Loaded 2M items. 20 views, 20 view queries in parallel. Views start to compact/index when rebalance is in progress.
Attaching the lastest diags, if anything interesting stands out.
in case you want to have a look in the next couple of hours
http://ec2-23-20-19-84.compute-1.amazonaws.com:8091/index.html#sec=servers&dev_subset=subset_dev