rebalance froze when node failed over and added back (observed mem used > high water mark for bucket)

Details

Type: Bug
Status: Resolved
Priority: Major
Resolution: Won't Fix
Affects Version/s: 2.0
Fix Version/s: 2.0.1
Component/s: couchbase-bucket
Security Level: Public
Labels:
None
Environment:

Hide
http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1976-rel.deb.manifest.xml
12.04 Ubuntu LTS ec2

Show
http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1976-rel.deb.manifest.xml 12.04 Ubuntu LTS ec2

Description

- 2 node cluster
- 2 buckets
- Bucket 'bkt' had a very high percentage of sets in its front end load.
- Failed over ec2-54-252-25-132.ap-southeast-2.compute.amazonaws.com, added back, rebalance.
- Rebalance froze at around 98%.
- Stopped front end loads, disk write queue drained.
- Mem used for both nodes, greater than higher water mark.
- Restarted couchbase server, waited for warm up to complete, retried rebalance, rebalance remained frozen at 50%.
- Rebooted nodes, waited for warm up to complete, retried rebalance, rebalance remained frozen at 50%.

Cluster diags:
1 https://s3.amazonaws.com/bugdb/MB-7382/ec2-54-252-25-132.ap-southeast-2.compute.amazonaws.com-8091-diag.txt.gz

2 https://s3.amazonaws.com/bugdb/MB-7382/ec2-54-252-20-171.ap-southeast-2.compute.amazonaws.com-8091-diag.txt.gz

Attached the cbstats of all, raw memory for both nodes.

Options
- Sort By Name
- Sort By Date
- Ascending
- Descending
- Download All

Attachments

ec2-54-252-20-171.ap-southeast-2.compute.amazonaws.com.txt

09/Dec/12 1:55 AM

13 kB

Abhinav Dangeti
ec2-54-252-25-132.ap-southeast-2.compute.amazonaws.com.txt

09/Dec/12 1:55 AM

13 kB

Abhinav Dangeti

Activity

Ascending order - Click to sort in descending order

Abhinav Dangeti made changes - 09/Dec/12 1:56 AM

Field	Original Value	New Value
Description	- 2 node cluster - Bucket 'bkt' had a very high percentage of sets in its front end load. - Failed over ec2-54-252-25-132.ap-southeast-2.compute.amazonaws.com, added back, rebalance. - Rebalance froze at around 98%. - Stopped front end loads, disk write queue drained. - Mem used for both nodes, greater than higher water mark. - Restarted couchbase server, waited for warm up to complete, retried rebalance, rebalance remained frozen at 50%. - Rebooted nodes, waited for warm up to complete, retried rebalance, rebalance remained frozen at 50%. Live cluster: http://ec2-54-252-25-132.ap-southeast-2.compute.amazonaws.com:8091/ http://ec2-54-252-20-171.ap-southeast-2.compute.amazonaws.com:8091/ Attached the cbstats of all, raw memory for both nodes.	- 2 node cluster - 2 buckets - Bucket 'bkt' had a very high percentage of sets in its front end load. - Failed over ec2-54-252-25-132.ap-southeast-2.compute.amazonaws.com, added back, rebalance. - Rebalance froze at around 98%. - Stopped front end loads, disk write queue drained. - Mem used for both nodes, greater than higher water mark. - Restarted couchbase server, waited for warm up to complete, retried rebalance, rebalance remained frozen at 50%. - Rebooted nodes, waited for warm up to complete, retried rebalance, rebalance remained frozen at 50%. Live cluster: http://ec2-54-252-25-132.ap-southeast-2.compute.amazonaws.com:8091/ http://ec2-54-252-20-171.ap-southeast-2.compute.amazonaws.com:8091/ Attached the cbstats of all, raw memory for both nodes.

Trond Norbye made changes - 09/Dec/12 3:15 AM

Component/s		couchbase-bucket [ 10173 ]
Component/s	bucket-engine [ 10010 ]

Hide

Permalink

Chiyoung Seo added a comment - 09/Dec/12 3:17 AM

The load rate from clients was too high, which caused the cluster to be highly overloaded during rebalance. There were lots of backlogs in the replication queues, which caused the bucket "bkt" to have memory usage more than 90% of bucket quota. If memory usage is above 90% of bucket quota, the replication or vbucket takeover would stop.

If we don't set up the cluster with the enough capacity, we could have the rebalance issues.

Please set up the cluster with the enough capacity.

Show

Chiyoung Seo added a comment - 09/Dec/12 3:17 AM The load rate from clients was too high, which caused the cluster to be highly overloaded during rebalance. There were lots of backlogs in the replication queues, which caused the bucket "bkt" to have memory usage more than 90% of bucket quota. If memory usage is above 90% of bucket quota, the replication or vbucket takeover would stop. If we don't set up the cluster with the enough capacity, we could have the rebalance issues. Please set up the cluster with the enough capacity.

Chiyoung Seo made changes - 09/Dec/12 3:17 AM

Assignee

Chiyoung Seo [ chiyoung ]

Abhinav Dangeti [ abhinav ]

Hide

Permalink

Chiyoung Seo added a comment - 09/Dec/12 3:19 AM

Rebalance tests with two nodes wouldn't be good for system tests. All of our customers use three node cluster at least.

Show

Chiyoung Seo added a comment - 09/Dec/12 3:19 AM Rebalance tests with two nodes wouldn't be good for system tests. All of our customers use three node cluster at least.

Hide

Permalink

Abhinav Dangeti added a comment - 09/Dec/12 4:30 AM

Not part of a system test, this was the cluster where I was checking the deleted items' status, I just tried failing over and adding back one of the nodes.

Show

Abhinav Dangeti added a comment - 09/Dec/12 4:30 AM Not part of a system test, this was the cluster where I was checking the deleted items' status, I just tried failing over and adding back one of the nodes.

Abhinav Dangeti made changes - 09/Dec/12 4:43 AM

Description

- 2 node cluster
- 2 buckets
- Bucket 'bkt' had a very high percentage of sets in its front end load.
- Failed over ec2-54-252-25-132.ap-southeast-2.compute.amazonaws.com, added back, rebalance.
- Rebalance froze at around 98%.
- Stopped front end loads, disk write queue drained.
- Mem used for both nodes, greater than higher water mark.
- Restarted couchbase server, waited for warm up to complete, retried rebalance, rebalance remained frozen at 50%.
- Rebooted nodes, waited for warm up to complete, retried rebalance, rebalance remained frozen at 50%.

Live cluster:
http://ec2-54-252-25-132.ap-southeast-2.compute.amazonaws.com:8091/
http://ec2-54-252-20-171.ap-southeast-2.compute.amazonaws.com:8091/

Attached the cbstats of all, raw memory for both nodes.

- 2 node cluster
- 2 buckets
- Bucket 'bkt' had a very high percentage of sets in its front end load.
- Failed over ec2-54-252-25-132.ap-southeast-2.compute.amazonaws.com, added back, rebalance.
- Rebalance froze at around 98%.
- Stopped front end loads, disk write queue drained.
- Mem used for both nodes, greater than higher water mark.
- Restarted couchbase server, waited for warm up to complete, retried rebalance, rebalance remained frozen at 50%.
- Rebooted nodes, waited for warm up to complete, retried rebalance, rebalance remained frozen at 50%.

Cluster diags:
1 http://ec2-54-252-25-132.ap-southeast-2.compute.amazonaws.com:8091/
https://s3.amazonaws.com/bugdb/MB-7382/ec2-54-252-25-132.ap-southeast-2.compute.amazonaws.com-8091-diag.txt.gz

2 http://ec2-54-252-20-171.ap-southeast-2.compute.amazonaws.com:8091/
https://s3.amazonaws.com/bugdb/MB-7382/ec2-54-252-20-171.ap-southeast-2.compute.amazonaws.com-8091-diag.txt.gz

Attached the cbstats of all, raw memory for both nodes.

Abhinav Dangeti made changes - 09/Dec/12 4:43 AM

Description

- 2 node cluster
- 2 buckets
- Bucket 'bkt' had a very high percentage of sets in its front end load.
- Failed over ec2-54-252-25-132.ap-southeast-2.compute.amazonaws.com, added back, rebalance.
- Rebalance froze at around 98%.
- Stopped front end loads, disk write queue drained.
- Mem used for both nodes, greater than higher water mark.
- Restarted couchbase server, waited for warm up to complete, retried rebalance, rebalance remained frozen at 50%.
- Rebooted nodes, waited for warm up to complete, retried rebalance, rebalance remained frozen at 50%.

Cluster diags:
1 http://ec2-54-252-25-132.ap-southeast-2.compute.amazonaws.com:8091/
https://s3.amazonaws.com/bugdb/MB-7382/ec2-54-252-25-132.ap-southeast-2.compute.amazonaws.com-8091-diag.txt.gz

2 http://ec2-54-252-20-171.ap-southeast-2.compute.amazonaws.com:8091/
https://s3.amazonaws.com/bugdb/MB-7382/ec2-54-252-20-171.ap-southeast-2.compute.amazonaws.com-8091-diag.txt.gz

Attached the cbstats of all, raw memory for both nodes.