[MB-6664] [windows 7] bucket creation on windows 7 sometimes timeout if bucket is created immediately after another bucket was deleted Created: 14/Sep/12  Updated: 10/Jan/13  Resolved: 03/Oct/12

Status: Closed
Project: Couchbase Server
Component/s: couchbase-bucket
Affects Version/s: None
Fix Version/s: 2.0
Security Level: Public

Type: Bug Priority: Major
Reporter: Iryna Mironava Assignee: Iryna Mironava
Resolution: Won't Fix Votes: 0
Labels: 2.0-beta-release-notes
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: windows 7 64-bit
build 1723
single node

Attachments: GZip Archive win7debuglogs.tar.gz     GZip Archive win7single.tar.gz    

 Description   
unable to create a bucket with replica=2
[error_logger:error,2012-09-13T16:31:17.647,ns_1@10.3.2.188:error_logger:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
  crasher:
    initial call: ns_memcached:init/1
    pid: <0.27705.2>
    registered_name: []
    exception exit: {ensure_bucket_failed,{error,function_clause}}
      in function gen_server:init_it/6
    ancestors: ['single_bucket_sup-2replica_90e83784-3382-4a59-8e49-586975bceb76',
                  <0.27670.2>]
    messages: []
    links: [<0.64.0>,<0.27671.2>,#Port<0.47362>]
    dictionary: []
    trap_exit: true
    status: running
    heap_size: 28657
    stack_size: 24
    reductions: 3751
  neighbours:

 Comments   
Comment by Aleksey Kondratenko [ 14/Sep/12 ]
I need full diags. Or at least all debug.* logs
Comment by Farshid Ghods [ 14/Sep/12 ]
Iryna,

please use the fixVersion as 2.0 beta or 2.0 and also set the priorities accordingly/
being unable to create a bukcet on windows 7 is a blocker bug
Comment by Iryna Mironava [ 14/Sep/12 ]
all debug logs attached
Comment by Iryna Mironava [ 14/Sep/12 ]
logs attached
Comment by Iryna Mironava [ 14/Sep/12 ]
ip of machine is 10.3.2.188
Jin, can you take a look?
Comment by Jin Lim [ 14/Sep/12 ]

1) NS Server reported TIMEOUT on deleting the bucket, "2replica_90e83784-3382-4a59-8e49-586975bceb76"

[ns_server:warn,2012-09-13T16:31:47.616,ns_1@10.3.2.188:<0.430.0>:ns_orchestrator:idle:418]Nodes ['ns_1@10.3.2.188'] failed to delete bucket "2replica_90e83784-3382-4a59-8e49-586975bceb76" within expected time.

2) Prior to the TIMEOUT there were any error reported on mecached(ep_engine) side. However, there were this repeating CRASH from ns_server side.

[error_logger:error,2012-09-13T16:31:17.678,ns_1@10.3.2.188:error_logger:ale_error_logger_handler:log_msg:76]Error in process <0.27670.2> on node 'ns_1@10.3.2.188' with exit value: {{badmatch,{error,shutdown}},[{single_bucket_sup,'-start_link/1-fun-0-',2}]}


[ns_server:debug,2012-09-13T16:31:18.553,ns_1@10.3.2.188:<0.27676.2>:janitor_agent:new_style_query_vbucket_states_loop:111]Exception from query_vbucket_states of "2replica_90e83784-3382-4a59-8e49-586975bceb76":'ns_1@10.3.2.188'
{'EXIT',
    {noproc,
        {gen_server,call,
            [{'janitor_agent-2replica_90e83784-3382-4a59-8e49-586975bceb76',
                 'ns_1@10.3.2.188'},
             query_vbucket_states,infinity]}}}
[ns_server:debug,2012-09-13T16:31:18.553,ns_1@10.3.2.188:<0.27676.2>:janitor_agent:new_style_query_vbucket_states_loop_next_step:116]Waiting for "2replica_90e83784-3382-4a59-8e49-586975bceb76" on 'ns_1@10.3.2.188'
[ns_server:debug,2012-09-13T16:31:19.569,ns_1@10.3.2.188:<0.27676.2>:janitor_agent:new_style_query_vbucket_states_loop:111]Exception from query_vbucket_states of "2replica_90e83784-3382-4a59-8e49-586975bceb76":'ns_1@10.3.2.188'
{'EXIT',
    {noproc,
        {gen_server,call,
            [{'janitor_agent-2replica_90e83784-3382-4a59-8e49-586975bceb76',
                 'ns_1@10.3.2.188'},
             query_vbucket_states,infinity]}}}
[ns_server:debug,2012-09-13T16:31:19.569,ns_1@10.3.2.188:<0.27676.2>:janitor_agent:new_style_query_vbucket_states_loop_next_step:116]Waiting for "2replica_90e83784-3382-4a59-8e49-586975bceb76" on 'ns_1@10.3.2.188'
[ns_server:debug,2012-09-13T16:31:20.585,ns_1@10.3.2.188:<0.27676.2>:janitor_agent:new_style_query_vbucket_states_loop:111]Exception from query_vbucket_states of "2replica_90e83784-3382-4a59-8e49-586975bceb76":'ns_1@10.3.2.188'
{'EXIT',
    {noproc,
        {gen_server,call,
            [{'janitor_agent-2replica_90e83784-3382-4a59-8e49-586975bceb76',
                 'ns_1@10.3.2.188'},
             query_vbucket_states,infinity]}}}
[ns_server:debug,2012-09-13T16:31:20.585,ns_1@10.3.2.188:<0.27676.2>:janitor_agent:new_style_query_vbucket_states_loop_next_step:116]Waiting for "2replica_90e83784-3382-4a59-8e49-586975bceb76" on 'ns_1@10.3.2.188'
[ns_server:debug,2012-09-13T16:31:21.600,ns_1@10.3.2.188:<0.27676.2>:janitor_agent:new_style_query_vbucket_states_loop:111]Exception from query_vbucket_states of "2replica_90e83784-3382-4a59-8e49-586975bceb76":'ns_1@10.3.2.188'
{'EXIT',
    {noproc,
        {gen_server,call,
            [{'janitor_agent-2replica_90e83784-3382-4a59-8e49-586975bceb76',
                 'ns_1@10.3.2.188'},
             query_vbucket_states,infinity]}}}
[ns_server:debug,2012-09-13T16:31:21.600,ns_1@10.3.2.188:<0.27676.2>:janitor_agent:new_style_query_vbucket_states_loop_next_step:116]Waiting for "2replica_90e83784-3382-4a59-8e49-586975bceb76" on 'ns_1@10.3.2.188'
[ns_doctor:debug,2012-09-13T16:31:38.788,ns_1@10.3.2.188:ns_doctor:ns_doctor:handle_info:136]Current node statuses:
[{'ns_1@10.3.2.188',

Can we rerun the test with increased memcached verbose level? wget -O- -q --user=Administrator --password=password --post-data='ns_config:update_key({node, node(), memcached}, fun (PList) -> lists:keyreplace(verbosity, 1, PList, {verbosity, "-vvv"}) end).' http://10.3.2.188:8091/diag/eval

Comment by Chiyoung Seo [ 14/Sep/12 ]
Assign it to Alk for the initial investigation.
Comment by Aleksey Kondratenko [ 14/Sep/12 ]
Logs were rotated past incident in ticket's description
Comment by Jin Lim [ 14/Sep/12 ]
Iryna & Alk,
The log snippets from my initial investigation above are directly from the node (ssh to the node and peeked at the logs under ~/logs) Give either Alk the direct access info or collect entire logs for him. Thanks.
  
Comment by Aleksey Kondratenko [ 14/Sep/12 ]
It may be easier to reproduce _and_ right after problem happens grab me _diags_.
Generated at Sun May 19 08:10:50 CDT 2013 using JIRA 5.2.4#845-sha1:c9f4cc41abe72fb236945343a1f485c2c844dac9.