Details
-
Type:
Technical task
-
Status:
Resolved
-
Priority:
Blocker
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 2011-01-20
-
Component/s: couchbase-bucket
-
Security Level: Public
-
Labels:None
Description
We need to understand what happens to the cluster if a node runsout of disk space.
Key questions are:
1) will the node still service reads
2) Can the node be failed over
3) can the node be balanced out
4) do writes get accepted again once disk space if freed up
Primary focus should be on Membase nodes, but the experiment should be repeated with memcached buckets (where there is still logging traffic)
Key questions are:
1) will the node still service reads
2) Can the node be failed over
3) can the node be balanced out
4) do writes get accepted again once disk space if freed up
Primary focus should be on Membase nodes, but the experiment should be repeated with memcached buckets (where there is still logging traffic)
http://screencast.com/t/NMwGITI6bz
Then the death node will automatically reload to initial setup page
http://screencast.com/t/63vUmCwPxR
From node 55, we can fail over death node (52) out.
After fail over, RAM used info on node 55 is incorrect on cluster overview page.
http://screencast.com/t/m7OlgTNja
http://screencast.com/t/xQ69bJN9yH
Delete all data in bucket
Load data back to bucket (not full)
Verify data in bucket (ok)
See RAM quota info incorrectly in data bucket page. File bug
MB-3346http://screencast.com/t/WcjYDlKC9
For the death node (52), eventhough it reloads to innitial setup, we can not setup membase server due to no disk space available.
Stop membase-server service on node 52
Delete all database file
Restart membase-server service.
Go through initial setup
Load data to default bucket (not full).
Verify data (ok).
Add node 55 back to node 52. Rebalance (ok)
Verify data on both nodes (ok)