Couchbase Server

rebalance froze when node failed over and added back (observed mem used > high water mark for bucket)

Details

Description

- 2 node cluster
- 2 buckets
- Bucket 'bkt' had a very high percentage of sets in its front end load.
- Failed over ec2-54-252-25-132.ap-southeast-2.compute.amazonaws.com, added back, rebalance.
- Rebalance froze at around 98%.
- Stopped front end loads, disk write queue drained.
- Mem used for both nodes, greater than higher water mark.
- Restarted couchbase server, waited for warm up to complete, retried rebalance, rebalance remained frozen at 50%.
- Rebooted nodes, waited for warm up to complete, retried rebalance, rebalance remained frozen at 50%.

Cluster diags:
1 https://s3.amazonaws.com/bugdb/MB-7382/ec2-54-252-25-132.ap-southeast-2.compute.amazonaws.com-8091-diag.txt.gz

2 https://s3.amazonaws.com/bugdb/MB-7382/ec2-54-252-20-171.ap-southeast-2.compute.amazonaws.com-8091-diag.txt.gz

Attached the cbstats of all, raw memory for both nodes.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Chiyoung Seo added a comment -
The load rate from clients was too high, which caused the cluster to be highly overloaded during rebalance. There were lots of backlogs in the replication queues, which caused the bucket "bkt" to have memory usage more than 90% of bucket quota. If memory usage is above 90% of bucket quota, the replication or vbucket takeover would stop.

If we don't set up the cluster with the enough capacity, we could have the rebalance issues.

Please set up the cluster with the enough capacity.
Show
Chiyoung Seo added a comment - The load rate from clients was too high, which caused the cluster to be highly overloaded during rebalance. There were lots of backlogs in the replication queues, which caused the bucket "bkt" to have memory usage more than 90% of bucket quota. If memory usage is above 90% of bucket quota, the replication or vbucket takeover would stop. If we don't set up the cluster with the enough capacity, we could have the rebalance issues. Please set up the cluster with the enough capacity.
Hide
Permalink
Chiyoung Seo added a comment -
Rebalance tests with two nodes wouldn't be good for system tests. All of our customers use three node cluster at least.
Show
Chiyoung Seo added a comment - Rebalance tests with two nodes wouldn't be good for system tests. All of our customers use three node cluster at least.
Hide
Permalink
Abhinav Dangeti added a comment -
Not part of a system test, this was the cluster where I was checking the deleted items' status, I just tried failing over and adding back one of the nodes.
Show
Abhinav Dangeti added a comment - Not part of a system test, this was the cluster where I was checking the deleted items' status, I just tried failing over and adding back one of the nodes.

People

  • Assignee:
    Abhinav Dangeti
    Reporter:
    Abhinav Dangeti
Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: