Details
-
Type:
Bug
-
Status:
Resolved
-
Priority:
Blocker
-
Resolution: Fixed
-
Affects Version/s: 2.0-developer-preview-4
-
Fix Version/s: 2.0-developer-preview-4
-
Component/s: None
-
Security Level: Public
-
Labels:None
Description
Created 10 nodes cluster. Created a view: {"reduce":{"map":"function (doc) {\n emit(doc._id, null);\n}","reduce":"_count"}. Uploaded 100k json documents using mcsoda. Queried a view with stale=false. Removed one node from the cluster. Queried the view with stale=false again while rebalance was in progress. View returned partial result:
{"rows":[
{"key":null,"value":88693}
],
"errors":[
{"from":"http://10.17.1.200:9503/_view_merge/?stale=false","reason":"{<<\"noproc\">>,<<\"{gen_server,call,[<0.2544.0>,{add,<0.6822.0>},infinity]}\">>}"}
]}
This state is permanent: the view keeps returning partial result with this error. There are plenty of compaction_loop_died errors in the log on the failed node:
=========================CRASH REPORT=========================
crasher:
initial call: couchbase_compaction_daemon:init/1
pid: <0.12968.0>
registered_name: couchbase_compaction_daemon
exception exit: {compaction_loop_died,
{noproc,
{gen_server,call,
[<0.2544.0>,{add,<0.12969.0>},infinity]}}}
in function gen_server:terminate/6
ancestors: [<0.12967.0>,ns_server_sup,ns_server_cluster_sup,<0.42.0>]
messages: []
links: [<0.12967.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 233
stack_size: 24
reductions: 1074
neighbours:
Attaching logs and diag from the node where the error happens.
{"rows":[
{"key":null,"value":88693}
],
"errors":[
{"from":"http://10.17.1.200:9503/_view_merge/?stale=false","reason":"{<<\"noproc\">>,<<\"{gen_server,call,[<0.2544.0>,{add,<0.6822.0>},infinity]}\">>}"}
]}
This state is permanent: the view keeps returning partial result with this error. There are plenty of compaction_loop_died errors in the log on the failed node:
=========================CRASH REPORT=========================
crasher:
initial call: couchbase_compaction_daemon:init/1
pid: <0.12968.0>
registered_name: couchbase_compaction_daemon
exception exit: {compaction_loop_died,
{noproc,
{gen_server,call,
[<0.2544.0>,{add,<0.12969.0>},infinity]}}}
in function gen_server:terminate/6
ancestors: [<0.12967.0>,ns_server_sup,ns_server_cluster_sup,<0.42.0>]
messages: []
links: [<0.12967.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 233
stack_size: 24
reductions: 1074
neighbours:
Attaching logs and diag from the node where the error happens.
-
- logs.tar.bz2
- 24/Jan/12 2:14 PM
- 106 kB
- Aliaksey Artamonau
-
- ns-diag-20120124121218.txt.bz2
- 24/Jan/12 2:14 PM
- 514 kB
- Aliaksey Artamonau
-
- ns-diag-20120125183105.txt.bz2
- 25/Jan/12 8:35 PM
- 347 kB
- Aliaksey Artamonau
-
- reopened_diag2.bz2
- 26/Jan/12 8:53 PM
- 521 kB
- Aliaksey Artamonau
-
- reopened_log1.bz2
- 26/Jan/12 8:46 PM
- 188 kB
- Aliaksey Artamonau
-
- reopened_log2.bz2
- 26/Jan/12 8:46 PM
- 187 kB
- Aliaksey Artamonau
-
- verbose_log.tar.bz2
- 28/Jan/12 7:16 PM
- 2.84 MB
- Aliaksey Artamonau
Activity
- All
- Comments
- Work Log
- History
- Activity
- Gerrit Reviews
Hide
Aliaksey Artamonau
added a comment -
Happened to me again with the latest couchdb sources. It seems that new diag won't help a lot. But I'm attaching it for completeness.
Show
Aliaksey Artamonau
added a comment - Happened to me again with the latest couchdb sources. It seems that new diag won't help a lot. But I'm attaching it for completeness.
Hide
Filipe Manana
added a comment -
The possible fix was merged today.
Fix regarding the {noproc,
{gen_server,call,
[<0.2544.0>,{add,<0.12969.0>},infinity]}}}
Fix regarding the {noproc,
{gen_server,call,
[<0.2544.0>,{add,<0.12969.0>},infinity]}}}
Show
Filipe Manana
added a comment - The possible fix was merged today.
Fix regarding the {noproc,
{gen_server,call,
[<0.2544.0>,{add,<0.12969.0>},infinity]}}}
Hide
Steve Yen
added a comment -
Resolving this on your behalf (feel free to hit Resolve yourself next time to move these forward). Thanks!
http://review.couchbase.org/#change,12638
http://review.couchbase.org/#change,12638
Hide
Aliaksey Artamonau
added a comment -
Encountered very similar error.
[couchdb:info] [2012-01-26 17:45:19] [n_1@10.17.2.9:<0.2400.0>:couch_log:info:39] Set view `default`, main group `_design/dev_test`, terminating with reason: {noproc,
{gen_server,
call,
[<0.4008.0>,
{add,
<0.6714.0>},
infinity]}}
[couchdb:info] [2012-01-26 17:45:19] [n_1@10.17.2.9:<0.6165.0>:couch_log:error:42] Uncaught error in HTTP request: {exit,
{normal,
{gen_server,
call,
[<0.2400.0>,
{request_group,
ok},
infinity]}}}
[couchdb:info] [2012-01-26 17:45:19] [n_1@10.17.2.9:<0.6165.0>:couch_log:info:39] Stacktrace: [{couch_index_merger,
do_query_index,
4},
{couch_index_merger,
query_index_loop,
5},
{couch_httpd,
handle_request_int,
6},
{mochiweb_http,
headers,
5},
{proc_lib,
init_p_do_apply,
3}]
And correspondingly it was reported to the client as:
{"from":"http://10.17.2.9:9501/_view_merge/","reason":"{<<\"normal\">>,\n <<\"{gen_server,call,[<0.2400.0>,{request_group,ok},infinity]}\">>}"}
That's not obvious to me though why couch_set_view_group terminates with reason different from normal but the caller's error mentions 'normal'.
[couchdb:info] [2012-01-26 17:45:19] [n_1@10.17.2.9:<0.2400.0>:couch_log:info:39] Set view `default`, main group `_design/dev_test`, terminating with reason: {noproc,
{gen_server,
call,
[<0.4008.0>,
{add,
<0.6714.0>},
infinity]}}
[couchdb:info] [2012-01-26 17:45:19] [n_1@10.17.2.9:<0.6165.0>:couch_log:error:42] Uncaught error in HTTP request: {exit,
{normal,
{gen_server,
call,
[<0.2400.0>,
{request_group,
ok},
infinity]}}}
[couchdb:info] [2012-01-26 17:45:19] [n_1@10.17.2.9:<0.6165.0>:couch_log:info:39] Stacktrace: [{couch_index_merger,
do_query_index,
4},
{couch_index_merger,
query_index_loop,
5},
{couch_httpd,
handle_request_int,
6},
{mochiweb_http,
headers,
5},
{proc_lib,
init_p_do_apply,
3}]
And correspondingly it was reported to the client as:
{"from":"http://10.17.2.9:9501/_view_merge/","reason":"{<<\"normal\">>,\n <<\"{gen_server,call,[<0.2400.0>,{request_group,ok},infinity]}\">>}"}
That's not obvious to me though why couch_set_view_group terminates with reason different from normal but the caller's error mentions 'normal'.
Show
Aliaksey Artamonau
added a comment - Encountered very similar error.
[couchdb:info] [2012-01-26 17:45:19] [ n_1@10.17.2.9 :<0.2400.0>:couch_log:info:39] Set view `default`, main group `_design/dev_test`, terminating with reason: {noproc,
{gen_server,
call,
[<0.4008.0>,
{add,
<0.6714.0>},
infinity]}}
[couchdb:info] [2012-01-26 17:45:19] [ n_1@10.17.2.9 :<0.6165.0>:couch_log:error:42] Uncaught error in HTTP request: {exit,
{normal,
{gen_server,
call,
[<0.2400.0>,
{request_group,
ok},
infinity]}}}
[couchdb:info] [2012-01-26 17:45:19] [ n_1@10.17.2.9 :<0.6165.0>:couch_log:info:39] Stacktrace: [{couch_index_merger,
do_query_index,
4},
{couch_index_merger,
query_index_loop,
5},
{couch_httpd,
handle_request_int,
6},
{mochiweb_http,
headers,
5},
{proc_lib,
init_p_do_apply,
3}]
And correspondingly it was reported to the client as:
{"from":" http://10.17.2.9:9501/_view_merge/ ","reason":"{<<\"normal\">>,\n <<\"{gen_server,call,[<0.2400.0>,{request_group,ok},infinity]}\">>}"}
That's not obvious to me though why couch_set_view_group terminates with reason different from normal but the caller's error mentions 'normal'.
Hide
Aliaksey Artamonau
added a comment -
More of this:
[couchdb:info] [2012-01-26 18:36:25] [n_4@10.17.2.9:<0.2364.0>:couch_log:info:39] Set view `default`, replica group `_design/dev_test`, terminating with reason: {noproc,
{gen_server,
call,
[<0.6551.0>,
{add,
<0.28961.0>},
infinity]}}
[couchdb:info] [2012-01-26 18:36:25] [n_4@10.17.2.9:<0.2353.0>:couch_log:info:39] Set view `default`, main group `_design/dev_test`, terminating with reason: {function_clause,
[{couch_set_view_group,
handle_info,
[{ack,
<0.2364.0>,
{noproc,
{gen_server,
call,
[<0.6551.0>,
{add,
<0.28961.0>},
infinity]}}}, ....
[couchdb:info] [2012-01-26 18:36:25] [n_4@10.17.2.9:<0.28318.0>:couch_log:error:42] Uncaught error in HTTP request: {exit,
{normal,
{gen_server,
call,
[<0.2364.0>,
{request_group,
ok},
infinity]}}}
[couchdb:info] [2012-01-26 18:36:25] [n_4@10.17.2.9:<0.28318.0>:couch_log:info:39] Stacktrace: [{couch_index_merger,
do_query_index,
4},
{couch_index_merger,
query_index_loop,
5},
{couch_httpd,
handle_request_int,
6},
{mochiweb_http,
headers,
5},
{proc_lib,
init_p_do_apply,
3}]
[couchdb:info] [2012-01-26 18:36:25] [n_4@10.17.2.9:<0.2364.0>:couch_log:info:39] Set view `default`, replica group `_design/dev_test`, terminating with reason: {noproc,
{gen_server,
call,
[<0.6551.0>,
{add,
<0.28961.0>},
infinity]}}
[couchdb:info] [2012-01-26 18:36:25] [n_4@10.17.2.9:<0.2353.0>:couch_log:info:39] Set view `default`, main group `_design/dev_test`, terminating with reason: {function_clause,
[{couch_set_view_group,
handle_info,
[{ack,
<0.2364.0>,
{noproc,
{gen_server,
call,
[<0.6551.0>,
{add,
<0.28961.0>},
infinity]}}}, ....
[couchdb:info] [2012-01-26 18:36:25] [n_4@10.17.2.9:<0.28318.0>:couch_log:error:42] Uncaught error in HTTP request: {exit,
{normal,
{gen_server,
call,
[<0.2364.0>,
{request_group,
ok},
infinity]}}}
[couchdb:info] [2012-01-26 18:36:25] [n_4@10.17.2.9:<0.28318.0>:couch_log:info:39] Stacktrace: [{couch_index_merger,
do_query_index,
4},
{couch_index_merger,
query_index_loop,
5},
{couch_httpd,
handle_request_int,
6},
{mochiweb_http,
headers,
5},
{proc_lib,
init_p_do_apply,
3}]
Show
Aliaksey Artamonau
added a comment - More of this:
[couchdb:info] [2012-01-26 18:36:25] [ n_4@10.17.2.9 :<0.2364.0>:couch_log:info:39] Set view `default`, replica group `_design/dev_test`, terminating with reason: {noproc,
{gen_server,
call,
[<0.6551.0>,
{add,
<0.28961.0>},
infinity]}}
[couchdb:info] [2012-01-26 18:36:25] [ n_4@10.17.2.9 :<0.2353.0>:couch_log:info:39] Set view `default`, main group `_design/dev_test`, terminating with reason: {function_clause,
[{couch_set_view_group,
handle_info,
[{ack,
<0.2364.0>,
{noproc,
{gen_server,
call,
[<0.6551.0>,
{add,
<0.28961.0>},
infinity]}}}, ....
[couchdb:info] [2012-01-26 18:36:25] [ n_4@10.17.2.9 :<0.28318.0>:couch_log:error:42] Uncaught error in HTTP request: {exit,
{normal,
{gen_server,
call,
[<0.2364.0>,
{request_group,
ok},
infinity]}}}
[couchdb:info] [2012-01-26 18:36:25] [ n_4@10.17.2.9 :<0.28318.0>:couch_log:info:39] Stacktrace: [{couch_index_merger,
do_query_index,
4},
{couch_index_merger,
query_index_loop,
5},
{couch_httpd,
handle_request_int,
6},
{mochiweb_http,
headers,
5},
{proc_lib,
init_p_do_apply,
3}]
Show
Aliaksey Artamonau
added a comment - Attached full logs for these two failures.
Hide
Aliaksey Artamonau
added a comment -
And by the way. In the second case views are non-functional on this node anymore. Index folder always hangs on some couch_set_view_group call:
Program counter: 0x00007fc3212c6e08 (gen:do_call/4 + 576)
CP: 0x0000000000000000 (invalid)
arity = 0
0x00007fc2cd3e3288 Return addr 0x00007fc32136d9a8 (gen_server:call/3 + 128)
y(0) #Ref<0.0.5.155655>
y(1) 'n_4@10.17.2.9'
y(2) []
y(3) infinity
y(4) {get_group_server,<<7 bytes>>,{set_view_group,<<16 bytes>>,nil,<<7 bytes>>,<<12 bytes>>,<<10 bytes>>,[],[{set_view,0,[],[],[],<<41 bytes>>,nil,[{<<6 bytes>>,<<6 bytes>>}],[]}],{[]},nil,nil,nil,nil,nil,nil,undefined,nil,nil}}
y(5) '$gen_call'
y(6) <0.195.0>
0x00007fc2cd3e32c8 Return addr 0x00007fc31de4d118 (couch_set_view:get_group_server/2 + 128)
y(0) infinity
y(1) {get_group_server,<<7 bytes>>,{set_view_group,<<16 bytes>>,nil,<<7 bytes>>,<<12 bytes>>,<<10 bytes>>,[],[{set_view,0,[],[],[],<<41 bytes>>,nil,[{<<6 bytes>>,<<6 bytes>>}],[]}],{[]},nil,nil,nil,nil,nil,nil,undefined,nil,nil}}
y(2) couch_set_view
y(3) Catch 0x00007fc32136d9a8 (gen_server:call/3 + 128)
0x00007fc2cd3e32f0 Return addr 0x00007fc31de4cb28 (couch_set_view:get_group/3 + 56)
0x00007fc2cd3e32f8 Return addr 0x00007fc31de4f390 (couch_set_view:get_reduce_view/5 + 88)
y(0) ok
0x00007fc2cd3e3308 Return addr 0x00007fc2cc7731a8 (couch_view_merger:get_set_view/6 + 208)
y(0) [99,100,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,125,126,128,129,148,149,166,167,168,175,177,178,191,192,193,200,201,218,219,243,244,255]
y(1) <<6 bytes>>
0x00007fc2cd3e3320 Return addr 0x00007fc2cc773490 (couch_view_merger:prepare_set_view/4 + 352)
y(0) [99,100,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,125,126,128,129,148,149,166,167,168,175,177,178,191,192,193,200,201,218,219,243,244,255]
y(1) update_after
y(2) <<6 bytes>>
y(3) <<12 bytes>>
y(4) <<7 bytes>>
y(5) #Fun<couch_set_view.get_reduce_view.5>
0x00007fc2cd3e3358 Return addr 0x00007fc2cc7774c0 (couch_view_merger:reduce_set_view_folder/4 + 720)
y(0) []
y(1) []
y(2) <0.31777.0>
y(3) <<12 bytes>>
y(4) <<7 bytes>>
y(5) Catch 0x00007fc2cc773750 (couch_view_merger:prepare_set_view/4 + 1056)
0x00007fc2cd3e3390 Return addr 0x00000000008a4cc8 (<terminate process normally>)
y(0) []
y(1) []
y(2) []
y(3) []
y(4) []
y(5) <<14 bytes>>
y(6) <0.31777.0>
y(7) {doc,<<12 bytes>>,{1,<<4 bytes>>},{[{<<5 bytes>>,{[{<<6 bytes>>,{[{<<3 bytes>>,<<41 bytes>>},{<<6 bytes>>,<<6 bytes>>}]}}]}}]},nil,false,[]}
y(8) {index_merge,[{set_view_spec,<<7 bytes>>,<<12 bytes>>,<<6 bytes>>,[99,100,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,125,126,128,129,148,149,166,167,168,175,177,178,191,192,193,200,201,218,219,243,244,255],nil,nil}],#Fun<couch_httpd_view_merger.0.27295029>,{sender_acc,{httpd,{mochiweb_request,#Port<0.29092>,'POST',"/_view_merge/",{1,1},{4,{"accept",{'Accept',"application/json"},nil,{"content-type",{'Content-Type',"application/json"},{"content-length",{'Content-Length',"272"},nil,nil},{"host",{'Host',"10.17.2.9:9504"},nil,nil}}}}},"10.17.2.9",'POST',[<<11 bytes>>],[<<11 bytes>>],capi_frontend,{dict,8,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[[<<7 bytes>>|#Fun<couch_httpd.8.26678597>]],[[<<9 bytes>>|#Fun<couch_httpd.8.26678597>]],[[<<13 bytes>>|#Fun<couch_httpd.8.26678597>]],[],[],[[<<7 bytes>>|#Fun<couch_httpd.8.26678597>]],[[<<16 bytes>>|#Fun<couch_httpd.8.26678597>]],[],[[<<8 bytes>>|#Fun<couch_httpd.8.26678597>]],[],[],[[<<10 bytes>>|#Fun<couch_httpd.8.26678597>]],[[<<8 bytes>>|#Fun<couch_httpd.8.26678597>]],[],[],[]}}},{user_ctx,null,[<<6 bytes>>],<<50 bytes>>},undefined,{dict,11,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[[<<8 bytes>>|#Fun<couch_httpd.10.12509357>],[<<5 bytes>>|#Fun<couch_httpd.10.12509357>]],[[<<5 bytes>>|#Fun<couch_httpd.10.12509357>],[<<5 bytes>>|#Fun<couch_httpd.10.12509357>]],[[<<7 bytes>>|#Fun<couch_httpd.10.12509357>]],[],[[<<14 bytes>>|#Fun<couch_httpd.10.12509357>],[<<14 bytes>>|#Fun<couch_httpd.10.12509357>]],[],[],[],[[<<8 bytes>>|#Fun<couch_httpd.10.12509357>]],[],[],[],[[<<5 bytes>>|#Fun<couch_httpd.10.12509357>]],[],[[<<17 bytes>>|#Fun<couch_httpd.10.12509357>],[<<12 bytes>>|#Fun<couch_httpd.10.12509357>]]}}},undefined,#Fun<couch_httpd.6.96738937>,{dict,18,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[[<<8 bytes>>|#Fun<couch_httpd.6.96738937>],[<<10 bytes>>|#Fun<couch_httpd.6.96738937>]],[[<<13 bytes>>|#Fun<couch_httpd.6.96738937>]],[],[[<<14 bytes>>|#Fun<couch_httpd.6.96738937>]],[[<<6 bytes>>|#Fun<couch_httpd.5.32618959>],[<<1 byte>>|#Fun<couch_httpd.5.32618959>],[<<7 bytes>>|#Fun<couch_httpd.6.96738937>]],[[<<4 bytes>>|#Fun<couch_httpd.6.96738937>]],[[<<9 bytes>>|#Fun<couch_httpd.6.96738937>]],[],[[<<8 bytes>>|#Fun<couch_httpd.6.96738937>],[<<14 bytes>>|#Fun<couch_httpd.6.96738937>],[<<14 bytes>>|#Fun<couch_httpd.6.96738937>]],[],[[<<6 bytes>>|#Fun<couch_httpd.6.96738937>],[<<11 bytes>>|#Fun<couch_httpd.5.32618959>]],[[<<8 bytes>>|#Fun<couch_httpd.6.96738937>],[<<6 bytes>>|#Fun<couch_httpd.6.96738937>]],[[<<11 bytes>>|#Fun<couch_httpd.6.96738937>]],[],[[<<6 bytes>>|#Fun<couch_httpd.6.96738937>]],[]}}}},nil,continue,<<0 bytes>>,[]},60000,continue,{1,<<4 bytes>>},{view_query_args,undefined,undefined,<<0 bytes>>,<<1 byte>>,fwd,true,10000000000,0,0,reduce,false,false,update_after,false,nil,nil,true,nil,<<6 bytes>>},{user_ctx,null,[<<6 bytes>>],<<50 bytes>>},{view_merge,nil,nil,<<10 bytes>>,nil}}
y(9) {view_query_args,undefined,undefined,<<0 bytes>>,<<1 byte>>,fwd,true,10000000000,0,0,reduce,false,false,update_after,false,nil,nil,true,nil,<<6 bytes>>}
y(10) <<12 bytes>>
y(11) <<7 bytes>>
Program counter: 0x00007fc3212c6e08 (gen:do_call/4 + 576)
CP: 0x0000000000000000 (invalid)
arity = 0
0x00007fc2cd3e3288 Return addr 0x00007fc32136d9a8 (gen_server:call/3 + 128)
y(0) #Ref<0.0.5.155655>
y(1) 'n_4@10.17.2.9'
y(2) []
y(3) infinity
y(4) {get_group_server,<<7 bytes>>,{set_view_group,<<16 bytes>>,nil,<<7 bytes>>,<<12 bytes>>,<<10 bytes>>,[],[{set_view,0,[],[],[],<<41 bytes>>,nil,[{<<6 bytes>>,<<6 bytes>>}],[]}],{[]},nil,nil,nil,nil,nil,nil,undefined,nil,nil}}
y(5) '$gen_call'
y(6) <0.195.0>
0x00007fc2cd3e32c8 Return addr 0x00007fc31de4d118 (couch_set_view:get_group_server/2 + 128)
y(0) infinity
y(1) {get_group_server,<<7 bytes>>,{set_view_group,<<16 bytes>>,nil,<<7 bytes>>,<<12 bytes>>,<<10 bytes>>,[],[{set_view,0,[],[],[],<<41 bytes>>,nil,[{<<6 bytes>>,<<6 bytes>>}],[]}],{[]},nil,nil,nil,nil,nil,nil,undefined,nil,nil}}
y(2) couch_set_view
y(3) Catch 0x00007fc32136d9a8 (gen_server:call/3 + 128)
0x00007fc2cd3e32f0 Return addr 0x00007fc31de4cb28 (couch_set_view:get_group/3 + 56)
0x00007fc2cd3e32f8 Return addr 0x00007fc31de4f390 (couch_set_view:get_reduce_view/5 + 88)
y(0) ok
0x00007fc2cd3e3308 Return addr 0x00007fc2cc7731a8 (couch_view_merger:get_set_view/6 + 208)
y(0) [99,100,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,125,126,128,129,148,149,166,167,168,175,177,178,191,192,193,200,201,218,219,243,244,255]
y(1) <<6 bytes>>
0x00007fc2cd3e3320 Return addr 0x00007fc2cc773490 (couch_view_merger:prepare_set_view/4 + 352)
y(0) [99,100,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,125,126,128,129,148,149,166,167,168,175,177,178,191,192,193,200,201,218,219,243,244,255]
y(1) update_after
y(2) <<6 bytes>>
y(3) <<12 bytes>>
y(4) <<7 bytes>>
y(5) #Fun<couch_set_view.get_reduce_view.5>
0x00007fc2cd3e3358 Return addr 0x00007fc2cc7774c0 (couch_view_merger:reduce_set_view_folder/4 + 720)
y(0) []
y(1) []
y(2) <0.31777.0>
y(3) <<12 bytes>>
y(4) <<7 bytes>>
y(5) Catch 0x00007fc2cc773750 (couch_view_merger:prepare_set_view/4 + 1056)
0x00007fc2cd3e3390 Return addr 0x00000000008a4cc8 (<terminate process normally>)
y(0) []
y(1) []
y(2) []
y(3) []
y(4) []
y(5) <<14 bytes>>
y(6) <0.31777.0>
y(7) {doc,<<12 bytes>>,{1,<<4 bytes>>},{[{<<5 bytes>>,{[{<<6 bytes>>,{[{<<3 bytes>>,<<41 bytes>>},{<<6 bytes>>,<<6 bytes>>}]}}]}}]},nil,false,[]}
y(8) {index_merge,[{set_view_spec,<<7 bytes>>,<<12 bytes>>,<<6 bytes>>,[99,100,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,125,126,128,129,148,149,166,167,168,175,177,178,191,192,193,200,201,218,219,243,244,255],nil,nil}],#Fun<couch_httpd_view_merger.0.27295029>,{sender_acc,{httpd,{mochiweb_request,#Port<0.29092>,'POST',"/_view_merge/",{1,1},{4,{"accept",{'Accept',"application/json"},nil,{"content-type",{'Content-Type',"application/json"},{"content-length",{'Content-Length',"272"},nil,nil},{"host",{'Host',"10.17.2.9:9504"},nil,nil}}}}},"10.17.2.9",'POST',[<<11 bytes>>],[<<11 bytes>>],capi_frontend,{dict,8,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[[<<7 bytes>>|#Fun<couch_httpd.8.26678597>]],[[<<9 bytes>>|#Fun<couch_httpd.8.26678597>]],[[<<13 bytes>>|#Fun<couch_httpd.8.26678597>]],[],[],[[<<7 bytes>>|#Fun<couch_httpd.8.26678597>]],[[<<16 bytes>>|#Fun<couch_httpd.8.26678597>]],[],[[<<8 bytes>>|#Fun<couch_httpd.8.26678597>]],[],[],[[<<10 bytes>>|#Fun<couch_httpd.8.26678597>]],[[<<8 bytes>>|#Fun<couch_httpd.8.26678597>]],[],[],[]}}},{user_ctx,null,[<<6 bytes>>],<<50 bytes>>},undefined,{dict,11,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[[<<8 bytes>>|#Fun<couch_httpd.10.12509357>],[<<5 bytes>>|#Fun<couch_httpd.10.12509357>]],[[<<5 bytes>>|#Fun<couch_httpd.10.12509357>],[<<5 bytes>>|#Fun<couch_httpd.10.12509357>]],[[<<7 bytes>>|#Fun<couch_httpd.10.12509357>]],[],[[<<14 bytes>>|#Fun<couch_httpd.10.12509357>],[<<14 bytes>>|#Fun<couch_httpd.10.12509357>]],[],[],[],[[<<8 bytes>>|#Fun<couch_httpd.10.12509357>]],[],[],[],[[<<5 bytes>>|#Fun<couch_httpd.10.12509357>]],[],[[<<17 bytes>>|#Fun<couch_httpd.10.12509357>],[<<12 bytes>>|#Fun<couch_httpd.10.12509357>]]}}},undefined,#Fun<couch_httpd.6.96738937>,{dict,18,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[[<<8 bytes>>|#Fun<couch_httpd.6.96738937>],[<<10 bytes>>|#Fun<couch_httpd.6.96738937>]],[[<<13 bytes>>|#Fun<couch_httpd.6.96738937>]],[],[[<<14 bytes>>|#Fun<couch_httpd.6.96738937>]],[[<<6 bytes>>|#Fun<couch_httpd.5.32618959>],[<<1 byte>>|#Fun<couch_httpd.5.32618959>],[<<7 bytes>>|#Fun<couch_httpd.6.96738937>]],[[<<4 bytes>>|#Fun<couch_httpd.6.96738937>]],[[<<9 bytes>>|#Fun<couch_httpd.6.96738937>]],[],[[<<8 bytes>>|#Fun<couch_httpd.6.96738937>],[<<14 bytes>>|#Fun<couch_httpd.6.96738937>],[<<14 bytes>>|#Fun<couch_httpd.6.96738937>]],[],[[<<6 bytes>>|#Fun<couch_httpd.6.96738937>],[<<11 bytes>>|#Fun<couch_httpd.5.32618959>]],[[<<8 bytes>>|#Fun<couch_httpd.6.96738937>],[<<6 bytes>>|#Fun<couch_httpd.6.96738937>]],[[<<11 bytes>>|#Fun<couch_httpd.6.96738937>]],[],[[<<6 bytes>>|#Fun<couch_httpd.6.96738937>]],[]}}}},nil,continue,<<0 bytes>>,[]},60000,continue,{1,<<4 bytes>>},{view_query_args,undefined,undefined,<<0 bytes>>,<<1 byte>>,fwd,true,10000000000,0,0,reduce,false,false,update_after,false,nil,nil,true,nil,<<6 bytes>>},{user_ctx,null,[<<6 bytes>>],<<50 bytes>>},{view_merge,nil,nil,<<10 bytes>>,nil}}
y(9) {view_query_args,undefined,undefined,<<0 bytes>>,<<1 byte>>,fwd,true,10000000000,0,0,reduce,false,false,update_after,false,nil,nil,true,nil,<<6 bytes>>}
y(10) <<12 bytes>>
y(11) <<7 bytes>>
Show
Aliaksey Artamonau
added a comment - And by the way. In the second case views are non-functional on this node anymore. Index folder always hangs on some couch_set_view_group call:
Program counter: 0x00007fc3212c6e08 (gen:do_call/4 + 576)
CP: 0x0000000000000000 (invalid)
arity = 0
0x00007fc2cd3e3288 Return addr 0x00007fc32136d9a8 (gen_server:call/3 + 128)
y(0) #Ref<0.0.5.155655>
y(1) ' n_4@10.17.2.9 '
y(2) []
y(3) infinity
y(4) {get_group_server,<<7 bytes>>,{set_view_group,<<16 bytes>>,nil,<<7 bytes>>,<<12 bytes>>,<<10 bytes>>,[],[{set_view,0,[],[],[],<<41 bytes>>,nil,[{<<6 bytes>>,<<6 bytes>>}],[]}],{[]},nil,nil,nil,nil,nil,nil,undefined,nil,nil}}
y(5) '$gen_call'
y(6) <0.195.0>
0x00007fc2cd3e32c8 Return addr 0x00007fc31de4d118 (couch_set_view:get_group_server/2 + 128)
y(0) infinity
y(1) {get_group_server,<<7 bytes>>,{set_view_group,<<16 bytes>>,nil,<<7 bytes>>,<<12 bytes>>,<<10 bytes>>,[],[{set_view,0,[],[],[],<<41 bytes>>,nil,[{<<6 bytes>>,<<6 bytes>>}],[]}],{[]},nil,nil,nil,nil,nil,nil,undefined,nil,nil}}
y(2) couch_set_view
y(3) Catch 0x00007fc32136d9a8 (gen_server:call/3 + 128)
0x00007fc2cd3e32f0 Return addr 0x00007fc31de4cb28 (couch_set_view:get_group/3 + 56)
0x00007fc2cd3e32f8 Return addr 0x00007fc31de4f390 (couch_set_view:get_reduce_view/5 + 88)
y(0) ok
0x00007fc2cd3e3308 Return addr 0x00007fc2cc7731a8 (couch_view_merger:get_set_view/6 + 208)
y(0) [99,100,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,125,126,128,129,148,149,166,167,168,175,177,178,191,192,193,200,201,218,219,243,244,255]
y(1) <<6 bytes>>
0x00007fc2cd3e3320 Return addr 0x00007fc2cc773490 (couch_view_merger:prepare_set_view/4 + 352)
y(0) [99,100,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,125,126,128,129,148,149,166,167,168,175,177,178,191,192,193,200,201,218,219,243,244,255]
y(1) update_after
y(2) <<6 bytes>>
y(3) <<12 bytes>>
y(4) <<7 bytes>>
y(5) #Fun<couch_set_view.get_reduce_view.5>
0x00007fc2cd3e3358 Return addr 0x00007fc2cc7774c0 (couch_view_merger:reduce_set_view_folder/4 + 720)
y(0) []
y(1) []
y(2) <0.31777.0>
y(3) <<12 bytes>>
y(4) <<7 bytes>>
y(5) Catch 0x00007fc2cc773750 (couch_view_merger:prepare_set_view/4 + 1056)
0x00007fc2cd3e3390 Return addr 0x00000000008a4cc8 (<terminate process normally>)
y(0) []
y(1) []
y(2) []
y(3) []
y(4) []
y(5) <<14 bytes>>
y(6) <0.31777.0>
y(7) {doc,<<12 bytes>>,{1,<<4 bytes>>},{[{<<5 bytes>>,{[{<<6 bytes>>,{[{<<3 bytes>>,<<41 bytes>>},{<<6 bytes>>,<<6 bytes>>}]}}]}}]},nil,false,[]}
y(8) {index_merge,[{set_view_spec,<<7 bytes>>,<<12 bytes>>,<<6 bytes>>,[99,100,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,125,126,128,129,148,149,166,167,168,175,177,178,191,192,193,200,201,218,219,243,244,255],nil,nil}],#Fun<couch_httpd_view_merger.0.27295029>,{sender_acc,{httpd,{mochiweb_request,#Port<0.29092>,'POST',"/_view_merge/",{1,1},{4,{"accept",{'Accept',"application/json"},nil,{"content-type",{'Content-Type',"application/json"},{"content-length",{'Content-Length',"272"},nil,nil},{"host",{'Host',"10.17.2.9:9504"},nil,nil}}}}},"10.17.2.9",'POST',[<<11 bytes>>],[<<11 bytes>>],capi_frontend,{dict,8,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[[<<7 bytes>>|#Fun<couch_httpd.8.26678597>]],[[<<9 bytes>>|#Fun<couch_httpd.8.26678597>]],[[<<13 bytes>>|#Fun<couch_httpd.8.26678597>]],[],[],[[<<7 bytes>>|#Fun<couch_httpd.8.26678597>]],[[<<16 bytes>>|#Fun<couch_httpd.8.26678597>]],[],[[<<8 bytes>>|#Fun<couch_httpd.8.26678597>]],[],[],[[<<10 bytes>>|#Fun<couch_httpd.8.26678597>]],[[<<8 bytes>>|#Fun<couch_httpd.8.26678597>]],[],[],[]}}},{user_ctx,null,[<<6 bytes>>],<<50 bytes>>},undefined,{dict,11,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[[<<8 bytes>>|#Fun<couch_httpd.10.12509357>],[<<5 bytes>>|#Fun<couch_httpd.10.12509357>]],[[<<5 bytes>>|#Fun<couch_httpd.10.12509357>],[<<5 bytes>>|#Fun<couch_httpd.10.12509357>]],[[<<7 bytes>>|#Fun<couch_httpd.10.12509357>]],[],[[<<14 bytes>>|#Fun<couch_httpd.10.12509357>],[<<14 bytes>>|#Fun<couch_httpd.10.12509357>]],[],[],[],[[<<8 bytes>>|#Fun<couch_httpd.10.12509357>]],[],[],[],[[<<5 bytes>>|#Fun<couch_httpd.10.12509357>]],[],[[<<17 bytes>>|#Fun<couch_httpd.10.12509357>],[<<12 bytes>>|#Fun<couch_httpd.10.12509357>]]}}},undefined,#Fun<couch_httpd.6.96738937>,{dict,18,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[[<<8 bytes>>|#Fun<couch_httpd.6.96738937>],[<<10 bytes>>|#Fun<couch_httpd.6.96738937>]],[[<<13 bytes>>|#Fun<couch_httpd.6.96738937>]],[],[[<<14 bytes>>|#Fun<couch_httpd.6.96738937>]],[[<<6 bytes>>|#Fun<couch_httpd.5.32618959>],[<<1 byte>>|#Fun<couch_httpd.5.32618959>],[<<7 bytes>>|#Fun<couch_httpd.6.96738937>]],[[<<4 bytes>>|#Fun<couch_httpd.6.96738937>]],[[<<9 bytes>>|#Fun<couch_httpd.6.96738937>]],[],[[<<8 bytes>>|#Fun<couch_httpd.6.96738937>],[<<14 bytes>>|#Fun<couch_httpd.6.96738937>],[<<14 bytes>>|#Fun<couch_httpd.6.96738937>]],[],[[<<6 bytes>>|#Fun<couch_httpd.6.96738937>],[<<11 bytes>>|#Fun<couch_httpd.5.32618959>]],[[<<8 bytes>>|#Fun<couch_httpd.6.96738937>],[<<6 bytes>>|#Fun<couch_httpd.6.96738937>]],[[<<11 bytes>>|#Fun<couch_httpd.6.96738937>]],[],[[<<6 bytes>>|#Fun<couch_httpd.6.96738937>]],[]}}}},nil,continue,<<0 bytes>>,[]},60000,continue,{1,<<4 bytes>>},{view_query_args,undefined,undefined,<<0 bytes>>,<<1 byte>>,fwd,true,10000000000,0,0,reduce,false,false,update_after,false,nil,nil,true,nil,<<6 bytes>>},{user_ctx,null,[<<6 bytes>>],<<50 bytes>>},{view_merge,nil,nil,<<10 bytes>>,nil}}
y(9) {view_query_args,undefined,undefined,<<0 bytes>>,<<1 byte>>,fwd,true,10000000000,0,0,reduce,false,false,update_after,false,nil,nil,true,nil,<<6 bytes>>}
y(10) <<12 bytes>>
y(11) <<7 bytes>>
Show
Aliaksey Artamonau
added a comment - Attached also a full diag for the second case.
Show
Filipe Manana
added a comment - Try with http://review.couchbase.org/#change,12767
Hide
Aliaksey Artamonau
added a comment -
Was able to reproduce with all the latest fixes and disabled replica index.
Show
Aliaksey Artamonau
added a comment - Was able to reproduce with all the latest fixes and disabled replica index.
Show
Aliaksey Artamonau
added a comment - Attached verbose log for this issue.
Show
Farshid Ghods
added a comment - incremental rebalance out with reduce
Show
Aliaksey Artamonau
added a comment - It's still reproducible. Aliaksey K is looking into it.
Show
Filipe Manana
added a comment - Try querying now with ?debug=true. It will help diagnose.
Hide
Thuan Nguyen
added a comment -
Integrated in github-couchdb-preview #339 (See [http://qa.hq.northscale.net/job/github-couchdb-preview/339/])
made notify_db_updated consume exits only from parent.MB-4686 (Revision df1e48bea718fc9d6f57feb00aa1a06a1d8e124c)
Result = SUCCESS
Aliaksey Kandratsenka :
Files :
* src/couchdb/couch_db_updater.erl
made notify_db_updated consume exits only from parent.
Result = SUCCESS
Aliaksey Kandratsenka :
Files :
* src/couchdb/couch_db_updater.erl
Show
Thuan Nguyen
added a comment - Integrated in github-couchdb-preview #339 (See [ http://qa.hq.northscale.net/job/github-couchdb-preview/339/ ])
made notify_db_updated consume exits only from parent. MB-4686 (Revision df1e48bea718fc9d6f57feb00aa1a06a1d8e124c)
Result = SUCCESS
Aliaksey Kandratsenka :
Files :
* src/couchdb/couch_db_updater.erl
Show
Aleksey Kondratenko
added a comment - We believe it's fixed
There's a TODO in the view group regarding this that could be the cause.
I just uploaded a change to gerrit to do the todo:
http://review.couchbase.org/#change,12638
This can be a race condition, hard to reproduce.