Couchbase Server

cluster failover/rebalance expected behavior when one or more nodes are out of disk space

Details

  • Flagged:
    Release Note

Description

On or more nodes running out of disk space cannot take down a cluster. Data already stored on a node should stay accessible, only writes to membase buckets should fail.

Nodes should still be able to be removed from cluster or failed over.

Attention also needs to be paid to behaviour of other areas that use disk space, such as logs

Issue Links

duplicates

Bug - A problem which impairs or prevents the functions of the product. MB-3401 Rebalance failed when one node is out of disk space.

  • Major - Major loss of function.
  • Resolved - A resolution has been taken, and it is awaiting verification by reporter. From here issues are either reopened, or are closed.

Activity

Ascending order - Click to sort in descending order
Sean Lynch made changes -
Field Original Value New Value
Assignee Sean Lynch [ sean ]
Sean Lynch made changes -
Fix Version/s 1.7 [ 10111 ]
Sean Lynch made changes -
Fix Version/s 1.7 [ 10111 ]
Hide
Permalink
Frank Weigel added a comment -
A Pivotal Tracker story has been created for this Issue: http://www.pivotaltracker.com/story/show/9305789
Show
Frank Weigel added a comment - A Pivotal Tracker story has been created for this Issue: http://www.pivotaltracker.com/story/show/9305789
Aleksey Kondratenko made changes -
Assignee Sean Lynch [ sean ]
Farshid Ghods made changes -
Assignee Farshid Ghods [ farshid ]
Hide
Permalink
Farshid Ghods added a comment -
try this scenario and update before RC
Show
Farshid Ghods added a comment - try this scenario and update before RC
Hide
Permalink
Farshid Ghods added a comment -
when one node runs out of disk space memcached goes into pending mode ) and the user can rebalance this node out from the cluster.


Shutting down bucket "default" on 'ns_1@172.16.75.128' for server shutdown ns_memcached002 ns_1@172.16.75.128 18:48:24 - Tue May 24, 2011
Usage of disk "/" on node "172.16.75.128" is over 100%

if you have two nodes running out of disk space you will not be able to failover those two nodes because failover will timeout.
the workaround is if you have two or more nodes running out of disk space you need to stop membase server on those two nodes and then you can fail over those nodes.
Show
Farshid Ghods added a comment - when one node runs out of disk space memcached goes into pending mode ) and the user can rebalance this node out from the cluster. Shutting down bucket "default" on ' ns_1@172.16.75.128 ' for server shutdown ns_memcached002 ns_1@172.16.75.128 18:48:24 - Tue May 24, 2011 Usage of disk "/" on node "172.16.75.128" is over 100% if you have two nodes running out of disk space you will not be able to failover those two nodes because failover will timeout. the workaround is if you have two or more nodes running out of disk space you need to stop membase server on those two nodes and then you can fail over those nodes.
Farshid Ghods made changes -
Summary Having a node running out of disk space cannot take down a cluster cluster failover/rebalance expected behavior when one or more nodes are out of disk space
Issue Type Story [ 6 ] Bug [ 1 ]
Assignee Farshid Ghods [ farshid ]
Fix Version/s 1.7.1 [ 10183 ]
Fix Version/s 1.7 GA [ 10111 ]
Priority Blocker [ 1 ] Major [ 3 ]
Component/s ns_server [ 10019 ]
Flagged [Release Note]
Hide
Permalink
Perry Krug added a comment -
Just so long as we understand this this is still a bug. Failover should NEVER time out
Show
Perry Krug added a comment - Just so long as we understand this this is still a bug. Failover should NEVER time out
Perry Krug made changes -
Link This issue duplicates MB-3401 [ MB-3401 ]
Farshid Ghods made changes -
Labels 1.7.0-release-notes
Farshid Ghods made changes -
Labels 1.7.0-release-notes 1.7.0-release-notes 1.7.1-release-notes
Farshid Ghods made changes -
Fix Version/s 2.0 Beta [ 10113 ]
Fix Version/s 1.7.1 [ 10183 ]
Farshid Ghods made changes -
Component/s couchbase-bucket [ 10173 ]
Component/s ep_engine [ 10013 ]
Hide
Permalink
Peter Wansch added a comment -
Farshid, this can be closed right?
Show
Peter Wansch added a comment - Farshid, this can be closed right?
Peter Wansch made changes -
Assignee Farshid Ghods [ farshid ]
Peter Wansch made changes -
Assignee Farshid Ghods [ farshid ] Dipti Borkar [ dipti ]
Fix Version/s 2.0 [ 10114 ]
Fix Version/s 2.0-beta [ 10113 ]
Dipti Borkar made changes -
Status Open [ 1 ] Resolved [ 5 ]
Resolution Won't Fix [ 2 ]

Error rendering 'com.meetme.plugins.jira.gerrit-plugin:subtask-reviews-tabpanel'. Please contact your JIRA administrators.

People

  • Assignee:
    Dipti Borkar
    Reporter:
    Frank Weigel
Vote (0)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved: