Couchbase Server

server goes unavailable, protocol violations after a period of time

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Critical Critical
  • Resolution: Incomplete
  • Affects Version/s: 2.0-developer-preview-3
  • Fix Version/s: 2.0-developer-preview-4
  • Component/s: None
  • Security Level: Public
  • Labels:
    None
  • Environment:
    - We only have 1 node (4GB of RAM) and 1 bucket ('default' bucket)
    - We run Couchbase 2.0 Dev Preview 3
    - The server is homed in a large Amazon EC2 instance (2 virtuals cores)

Description

See full description at
http://www.couchbase.org/forums/thread/node-down-impossible-get-it-back

Summary is that after some time under regular workload, the end user sees a protocol violation via the Ruby Client and sees in the log that it the bucket was shutdown and did not restart.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Matt Ingenthron added a comment -
Perhaps this is another example of MB-4064 ? Not sure if it's a duplicate.
Show
Matt Ingenthron added a comment - Perhaps this is another example of MB-4064 ? Not sure if it's a duplicate.
Hide
Permalink
Farshid Ghods added a comment -
[stats:error] [2011-12-23 10:18:00] [ns_1@127.0.0.1:<0.18155.5>:stats_reader:log_bad_responses:191] Some nodes didn't respond: ['ns_1@127.0.0.1']
[ns_server:error] [2011-12-23 10:18:07] [ns_1@127.0.0.1:xdc_rep_manager:xdc_rep_manager:handle_info:263] database update notifier died. Reason: killed
[error_logger:error] [2011-12-23 10:18:07] [ns_1@127.0.0.1:error_logger:ale_error_logger_handler:log_msg:76] Error in process <0.16293.9> on node 'ns_1@127.0.0.1' with exit value: {{badmatch,{ok,<<295271 bytes>>}},[{couch_file,read_raw_iolist_int,3},{couch_file,maybe_read_more_iolist,4},{couch_file,reader_loop,1}]}
[stats:error] [2011-12-23 10:18:05] [ns_1@127.0.0.1:<0.7201.9>:stats_reader:log_bad_responses:191] Some nodes didn't respond: ['ns_1@127.0.0.1']
[ns_server:error] [2011-12-23 10:18:16] [ns_1@127.0.0.1:<0.16329.9>:ns_janitor:do_cleanup:68] Bucket "default" not yet ready on ['ns_1@127.0.0.1']
[stats:error] [2011-12-23 10:18:10] [ns_1@127.0.0.1:<0.10559.9>:stats_reader:log_bad_responses:191] Some nodes didn't respond: ['ns_1@127.0.0.1']
[error_logger:error] [2011-12-23 10:18:17] [ns_1@127.0.0.1:error_logger:ale_error_logger_handler:log_msg:76] ** Generic server xdc_rep_manager terminating
** Last message in was {'EXIT',<0.1625.9>,killed}
** When Server state == {rep_db_state,<0.1624.9>,<<"_replicator">>,<0.1625.9>}** Reason for termination ==
** {timeout,
       {gen_server,call,
           ['logger-ns_server',
            {log,
                {log_info,ns_server,error,xdc_rep_manager,handle_info,263,
                    {1324,635487,988595},
                    xdc_rep_manager,'ns_1@127.0.0.1',undefined},
                "database update notifier died. Reason: ~p",
                [killed]}]}}
Show
Farshid Ghods added a comment - [stats:error] [2011-12-23 10:18:00] [ ns_1@127.0.0.1 :<0.18155.5>:stats_reader:log_bad_responses:191] Some nodes didn't respond: [' ns_1@127.0.0.1 '] [ns_server:error] [2011-12-23 10:18:07] [ ns_1@127.0.0.1 :xdc_rep_manager:xdc_rep_manager:handle_info:263] database update notifier died. Reason: killed [error_logger:error] [2011-12-23 10:18:07] [ ns_1@127.0.0.1 :error_logger:ale_error_logger_handler:log_msg:76] Error in process <0.16293.9> on node ' ns_1@127.0.0.1 ' with exit value: {{badmatch,{ok,<<295271 bytes>>}},[{couch_file,read_raw_iolist_int,3},{couch_file,maybe_read_more_iolist,4},{couch_file,reader_loop,1}]} [stats:error] [2011-12-23 10:18:05] [ ns_1@127.0.0.1 :<0.7201.9>:stats_reader:log_bad_responses:191] Some nodes didn't respond: [' ns_1@127.0.0.1 '] [ns_server:error] [2011-12-23 10:18:16] [ ns_1@127.0.0.1 :<0.16329.9>:ns_janitor:do_cleanup:68] Bucket "default" not yet ready on [' ns_1@127.0.0.1 '] [stats:error] [2011-12-23 10:18:10] [ ns_1@127.0.0.1 :<0.10559.9>:stats_reader:log_bad_responses:191] Some nodes didn't respond: [' ns_1@127.0.0.1 '] [error_logger:error] [2011-12-23 10:18:17] [ ns_1@127.0.0.1 :error_logger:ale_error_logger_handler:log_msg:76] ** Generic server xdc_rep_manager terminating ** Last message in was {'EXIT',<0.1625.9>,killed} ** When Server state == {rep_db_state,<0.1624.9>,<<"_replicator">>,<0.1625.9>}** Reason for termination == ** {timeout,        {gen_server,call,            ['logger-ns_server',             {log,                 {log_info,ns_server,error,xdc_rep_manager,handle_info,263,                     {1324,635487,988595},                     xdc_rep_manager,' ns_1@127.0.0.1 ',undefined},                 "database update notifier died. Reason: ~p",                 [killed]}]}}
Hide
Permalink
Matt Ingenthron added a comment -
Was that comment intended to help me understand this issue? :)
Show
Matt Ingenthron added a comment - Was that comment intended to help me understand this issue? :)
Hide
Permalink
Farshid Ghods added a comment -
no :) didn't find diags link in the forum so just copied the error snippet i thought is more relevant here.

Show
Farshid Ghods added a comment - no :) didn't find diags link in the forum so just copied the error snippet i thought is more relevant here.
Hide
Permalink
Farshid Ghods added a comment -
Filipe,

can you also take a look at these couch errors?
Show
Farshid Ghods added a comment - Filipe, can you also take a look at these couch errors?
Hide
Permalink
Filipe Manana added a comment -
This seems to be related to the cross data center replication code:

{log_info,ns_server,error,xdc_rep_manager,handle_info,263,

Ask Srini, I'm not familiar with its implementation neither have the time to do it.
Show
Filipe Manana added a comment - This seems to be related to the cross data center replication code: {log_info,ns_server,error,xdc_rep_manager,handle_info,263, Ask Srini, I'm not familiar with its implementation neither have the time to do it.
Hide
Permalink
Aleksey Kondratenko added a comment -
xdc replication manager issues are unrelated. This think always crashes when we shutdown.

Here's what looks like real issue in couch:

[error_logger:error] [2011-12-23 10:18:07] [ns_1@127.0.0.1:error_logger:ale_error_logger_handler:log_msg:76] Error in process <0.16293.9> on node 'ns_1@127.0.0.1' with exit value: {{badmatch,{ok,<<295271 bytes>>}},[{couch_file,read_raw_iolist_int,3},{couch_file,maybe_read_more_iolist,4},{couch_file,reader_loop,1}]}
Show
Aleksey Kondratenko added a comment - xdc replication manager issues are unrelated. This think always crashes when we shutdown. Here's what looks like real issue in couch: [error_logger:error] [2011-12-23 10:18:07] [ ns_1@127.0.0.1 :error_logger:ale_error_logger_handler:log_msg:76] Error in process <0.16293.9> on node ' ns_1@127.0.0.1 ' with exit value: {{badmatch,{ok,<<295271 bytes>>}},[{couch_file,read_raw_iolist_int,3},{couch_file,maybe_read_more_iolist,4},{couch_file,reader_loop,1}]}
Hide
Permalink
Steve Yen added a comment -
Hi Aliaksey, does this need reassignment?
Show
Steve Yen added a comment - Hi Aliaksey, does this need reassignment?
Hide
Permalink
Steve Yen added a comment -
alk is impacted -- reassigning to aliaksey a. to help
Show
Steve Yen added a comment - alk is impacted -- reassigning to aliaksey a. to help
Hide
Permalink
Steve Yen added a comment -
need more diag from servers
Show
Steve Yen added a comment - need more diag from servers

People

  • Assignee:
    Aliaksey Artamonau
    Reporter:
    Matt Ingenthron
Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: