Couchbase Server

ns_server stops responding on windows while running view tests which create 20 design docs ( :8091 shuts down but erlang process keeps running)

Details

Description

the test :

./testrunner -t viewtests.ViewBasicTests.test_load_data_get_view_x_mins_y_design_docs,num-design-docs=20,load-time=10,run-view-time=1,num-tries=100

couchbase-server runs fine for the first few minutes but then it stops responding and test is unable to connect to :8091 after that

[2012-06-29 17:26:25,883] - [viewtests:461] INFO - Params {'connection_timeout': 60000, 'full_set': 'true'}
[2012-06-29 17:26:25,883] - [rest_client:233] INFO - index query url: http://10.3.2.144:8091/couchBase/default/_design/dev_test_view_on_10000_docs-40b5013/_view/dev_test_view_on_10000_docs-40b5013?connection_timeout=60000&full_set=true
[2012-06-29 17:26:25,890] - [rest_client:497] ERROR - http://10.3.2.144:8091/couchBase/default/_design/dev_test_view_on_10000_docs-40b5013/_view/dev_test_view_on_10000_docs-40b5013?connection_timeout=60000&full_set=true error 500 reason: unknown ["Unexpected server error, request logged."]
[2012-06-29 17:26:25,891] - [viewtests:485] ERROR - view_results not ready yet , try again in 10 seconds... , error Error occured querying view dev_test_view_on_10000_docs-40b5013: ["Unexpected server error, request logged."]
[2012-06-29 17:26:26,300] - [viewtests:637] INFO - 581.945871115 seconds has passed ....
[2012-06-29 17:26:26,301] - [viewtests:461] INFO - Params {'connection_timeout': 60000, 'full_set': 'true'}
[2012-06-29 17:26:26,301] - [rest_client:233] INFO - index query url: http://10.3.2.144:8091/couchBase/default/_design/dev_test_view_on_10000_docs-603e0d1/_view/dev_test_view_on_10000_docs-603e0d1?connection_timeout=60000&full_set=true
[2012-06-29 17:26:26,302] - [rest_client:500] ERROR - socket error while connecting to 10.3.2.144:8091 error [Errno 111] Connection refused:
[2012-06-29 17:26:26,708] - [viewtests:637] INFO - 582.344176054 seconds has passed ....
[2012-06-29 17:26:26,709] - [viewtests:461] INFO - Params {'connection_timeout': 60000, 'full_set': 'true'}
[2012-06-29 17:26:26,709] - [rest_client:233] INFO - index query url: http://10.3.2.144:8091/couchBase/default/_design/dev_test_view_on_10000_docs-c7fe9a7/_view/dev_test_view_on_10000_docs-c7fe9a7?connection_timeout=60000&full_set=true
[2012-06-29 17:26:26,710] - [rest_client:500] ERROR - socket error while connecting to 10.3.2.144:8091 error [Errno 111] Connection refused:
[2012-06-29 17:26:26,727] - [viewtests:637] INFO - 582.409823895 seconds has passed ....
[2012-06-29 17:26:26,727] - [viewtests:461] INFO - Params {'connection_timeout': 60000, 'full_set': 'true'}
[2012-06-29 17:26:26,729] - [rest_client:233] INFO - index query url: http://10.3.2.144:8091/couchBase/default/_design/dev_test_view_on_10000_docs-cf2133b/_view/dev_test_view_on_10000_docs-cf2133b?connection_timeout=60000&full_set=true
[2012-06-29 17:26:26,731] - [rest_client:500] ERROR - socket error while connecting to 10.3.2.144:8091 error [Errno 111] Connection refused:

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Farshid Ghods added a comment -
attaching diags from this node.
i had to restart this node couple fo times but it would hang again right after running the view test for few miutes.

i have not tried other tests like rebalancing so im not sure if its related to views or database ( couchstore or ns-server)
Show
Farshid Ghods added a comment - attaching diags from this node. i had to restart this node couple fo times but it would hang again right after running the view test for few miutes. i have not tried other tests like rebalancing so im not sure if its related to views or database ( couchstore or ns-server)
Hide
Permalink
Aleksey Kondratenko added a comment -
[error_logger:error] [2012-06-29 17:26:27] [ns_1@10.3.2.144:error_logger:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
  crasher:
    initial call: couch_server:init/1
    pid: <0.20000.4>
    registered_name: []
    exception exit: {{badmatch,{error,eacces}},
                     [{couch_file,'-init_delete_dir/1-fun-0-',2},
                      {filelib,do_fold_files2,8},
                      {couch_server,init,1},
                      {gen_server,init_it,6},
                      {proc_lib,init_p_do_apply,3}]}
      in function gen_server:init_it/6
    ancestors: [couch_primary_services,couch_server_sup,cb_couch_sup,
                  ns_server_cluster_sup,<0.67.0>]
    messages: []
    links: [<0.14084.3>]
    dictionary: []
    trap_exit: false
    status: running
    heap_size: 1597
    stack_size: 24
    reductions: 1066
  neighbours:
Show
Aleksey Kondratenko added a comment - [error_logger:error] [2012-06-29 17:26:27] [ ns_1@10.3.2.144 :error_logger:ale_error_logger_handler:log_report:72] =========================CRASH REPORT=========================   crasher:     initial call: couch_server:init/1     pid: <0.20000.4>     registered_name: []     exception exit: {{badmatch,{error,eacces}},                      [{couch_file,'-init_delete_dir/1-fun-0-',2},                       {filelib,do_fold_files2,8},                       {couch_server,init,1},                       {gen_server,init_it,6},                       {proc_lib,init_p_do_apply,3}]}       in function gen_server:init_it/6     ancestors: [couch_primary_services,couch_server_sup,cb_couch_sup,                   ns_server_cluster_sup,<0.67.0>]     messages: []     links: [<0.14084.3>]     dictionary: []     trap_exit: false     status: running     heap_size: 1597     stack_size: 24     reductions: 1066   neighbours:
Hide
Permalink
Aleksey Kondratenko added a comment -
Could be related to that windows problems.

AFAIK we had something similar in DP4 where we 'fixed' by retrying
Show
Aleksey Kondratenko added a comment - Could be related to that windows problems. AFAIK we had something similar in DP4 where we 'fixed' by retrying
Hide
Permalink
Aleksey Kondratenko added a comment -
actually problems happen quite before that.


[couchdb:info] [2012-06-29 17:26:04] [ns_1@10.3.2.144:<0.18798.4>:couch_log:error:42] Set view `default`, main group `_design/dev_test_view_on_10000_docs-19fc1cb`, doc loader error
error: {badmatch,eof}
stacktrace: [{couch_db,open_doc_int,3},
             {couch_set_view_updater,load_doc,4},
             {couch_set_view_updater,'-load_changes/7-fun-0-',6},
             {couch_btree,stream_kv_node2,8},
             {couch_btree,stream_kp_node,7},
             {couch_btree,fold,4},
             {couch_db,enum_docs_since,5},
             {couch_db,fast_reads,2}]
Show
Aleksey Kondratenko added a comment - actually problems happen quite before that. [couchdb:info] [2012-06-29 17:26:04] [ ns_1@10.3.2.144 :<0.18798.4>:couch_log:error:42] Set view `default`, main group `_design/dev_test_view_on_10000_docs-19fc1cb`, doc loader error error: {badmatch,eof} stacktrace: [{couch_db,open_doc_int,3},              {couch_set_view_updater,load_doc,4},              {couch_set_view_updater,'-load_changes/7-fun-0-',6},              {couch_btree,stream_kv_node2,8},              {couch_btree,stream_kp_node,7},              {couch_btree,fold,4},              {couch_db,enum_docs_since,5},              {couch_db,fast_reads,2}]
Hide
Permalink
Peter Wansch added a comment -
If this is not Windows specific, can you update the title and also link duplicate bugs if you stumble across them?
Show
Peter Wansch added a comment - If this is not Windows specific, can you update the title and also link duplicate bugs if you stumble across them?
Hide
Permalink
Aaron Miller added a comment -
Fixed in couchstore commit 460646e3cf6581383a3db4771ce0043ebd8747b8

http://review.couchbase.org/#change,17999
Show
Aaron Miller added a comment - Fixed in couchstore commit 460646e3cf6581383a3db4771ce0043ebd8747b8 http://review.couchbase.org/#change,17999

People

  • Assignee:
    Aaron Miller
    Reporter:
    Farshid Ghods
Vote (0)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved: