Couchbase Server

Erlang dump created on membase server restart

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: 1.7.2, 1.8.0
  • Fix Version/s: 1.8.1
  • Component/s: None
  • Security Level: Public
  • Environment:
    1.7.2

Description

After a recent OS update and reboot, the 2-server clusters did not reload from disk successfully, and a membase-server restart results in an erlang error.

When the cluster was warming up, both nodes are responding to a membase-server restart command with "{"init terminating in do_boot",{{badmatch,{error,{shutdown,{ns_server,start,[normal,[]]}}}},[{init,start_it,1},{init,start_em,1}]}}

Erlang has closed /opt/membase/lib/erlang/lib/os_mon-2.2.6/priv/bin/memsup: Erlang has closed.
Crash dump was written to: erl_crash.dump

Once the warmup finished the cluster was back in a working state.

Logs and crash file attached.
  1. File
    erl_crash.dump
    23/Jan/12 3:39 PM
    487 kB
    James Mauss
  2. Text File
    JDPCachePrimary_01112012a.log
    23/Jan/12 3:39 PM
    11.88 MB
    James Mauss

Activity

Ascending order - Click to sort in descending order
James Mauss made changes -
Field Original Value New Value
Assignee Aleksey Kondratenko [ alkondratenko ]
Hide
Permalink
James Mauss added a comment -
The customer would like to know why the erlang dump was being created.
Show
James Mauss added a comment - The customer would like to know why the erlang dump was being created.
Hide
Permalink
Aleksey Kondratenko added a comment -
another copy of erlang was still running.

We have known issue fixed in 2.0 that initscript stop action merely sends shutdown signal to ns_server without waiting for actual shutdown. Actual shutdown waits until memcached ends persisting it's data. So may take time. Thus initscript restart doesn't really work in most real world cases in 1.7 and current 1.8.
Show
Aleksey Kondratenko added a comment - another copy of erlang was still running. We have known issue fixed in 2.0 that initscript stop action merely sends shutdown signal to ns_server without waiting for actual shutdown. Actual shutdown waits until memcached ends persisting it's data. So may take time. Thus initscript restart doesn't really work in most real world cases in 1.7 and current 1.8.
Hide
Permalink
Aleksey Kondratenko added a comment -
added pivotal story to address that for 1.8.1 https://www.pivotaltracker.com/projects/212245
Show
Aleksey Kondratenko added a comment - added pivotal story to address that for 1.8.1 https://www.pivotaltracker.com/projects/212245
Dipti Borkar made changes -
Project Couchbase Support/Engineering Tasks [ 10061 ] Couchbase Server [ 10010 ]
Key CBSE-93 MB-4765
Security Public [ 10011 ]
Dipti Borkar made changes -
Fix Version/s 1.8.1 [ 10249 ]
Affects Version/s 1.8.0 [ 10248 ]
Affects Version/s 1.7.2 [ 10203 ]
Hide
Permalink
Aleksey Kondratenko added a comment -
Pivotal link is broken.

Anyway. This is done. 1.8.1 has reliable shutdown backported from 2.0
Show
Aleksey Kondratenko added a comment - Pivotal link is broken. Anyway. This is done. 1.8.1 has reliable shutdown backported from 2.0
Aleksey Kondratenko made changes -
Status Open [ 1 ] Resolved [ 5 ]
Fix Version/s 1.8.1 [ 10295 ]
Fix Version/s 1.8.2 [ 10249 ]
Resolution Fixed [ 1 ]
Farshid Ghods made changes -
Labels 1.8.1-release-notes

People

  • Assignee:
    Aleksey Kondratenko
    Reporter:
    James Mauss
Vote (0)
Watch (3)

Dates

  • Created:
    Updated:
    Resolved: