Couchbase Server

[system test]: swap rebalance hang in online upgrade from 1.8.1 to 2.0.1-168

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Incomplete
  • Affects Version/s: 2.0.1
  • Fix Version/s: 2.0.1
  • Component/s: ns_server
  • Security Level: Public
  • Labels:
  • Environment:
    ubuntu 11.04 64 bit

Description

Rebalance hang when doing swap rebalance by add node 2.0.1-168 to cluster and remove one 1.8.1 out of
cluster.

Environment:

1.8.1 nodes

1 = ec2-50-16-9-117.compute-1.amazonaws.com
# short public IP 50.16.9.117
# 10.149.28.28
2 = ec2-54-242-127-186.compute-1.amazonaws.com
# short public IP 54.242.127.186
# 10.149.19.204
3 = ec2-23-20-192-140.compute-1.amazonaws.com <== this node is remove to do 1st swap rebalance
# short public IP 23.20.192.140
# 10.149.27.66
4 = ec2-204-236-252-180.compute-1.amazonaws.com
# short public IP 204.236.252.180
# 10.149.21.76

Install couchbase server 2.0.1-168 in below node
5 = ec2-50-16-32-181.compute-1.amazonaws.com <== 2.0.1-167 node added to do 1st swap rebalance for online upgrade
# short public IP 50.16.32.181
# 10.149.8.5

Doing swap rebalance. Just in few minutes when I check vbuckets in node 66, I saw it is 0.

Diags will be posted soon.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Thuan Nguyen added a comment -
Diags files from all nodes https://s3.amazonaws.com/bugdb/jira/MB-7829/5ndoes-onupgrade-181_201-168_reb-hang-20130226-182430.tgz
Show
Thuan Nguyen added a comment - Diags files from all nodes https://s3.amazonaws.com/bugdb/jira/MB-7829/5ndoes-onupgrade-181_201-168_reb-hang-20130226-182430.tgz
Hide
Permalink
Jin Lim added a comment -
Test continued OK
Show
Jin Lim added a comment - Test continued OK
Hide
Permalink
Aliaksey Artamonau added a comment -
According to the logs the node that you were trying to add was already part of the cluster. It seems that you just reinstalled everything on that node. So the node itself believed that is was not part of any cluster. On the other hand, all the other nodes believed that it was part of their cluster. When a node gets added to the cluster, we just ensure that the node is not part of other cluster. Because of this you were able to add it second time to the same cluster. And this caused massive problems which (among other) resulted in stuck rebalance. I'm closing this as "won't fix" since this scenario is not something we really want to support. Please reopen if you can reproduce it using sane steps.
Show
Aliaksey Artamonau added a comment - According to the logs the node that you were trying to add was already part of the cluster. It seems that you just reinstalled everything on that node. So the node itself believed that is was not part of any cluster. On the other hand, all the other nodes believed that it was part of their cluster. When a node gets added to the cluster, we just ensure that the node is not part of other cluster. Because of this you were able to add it second time to the same cluster. And this caused massive problems which (among other) resulted in stuck rebalance. I'm closing this as "won't fix" since this scenario is not something we really want to support. Please reopen if you can reproduce it using sane steps.
Hide
Permalink
Thuan Nguyen added a comment -
I saw rebalance moving later on. So I will close this bug
Show
Thuan Nguyen added a comment - I saw rebalance moving later on. So I will close this bug

People

  • Assignee:
    Thuan Nguyen
    Reporter:
    Thuan Nguyen
Vote (0)
Watch (3)

Dates

  • Created:
    Updated:
    Resolved: