Details
-
Type:
Bug
-
Status:
Resolved
-
Priority:
Blocker
-
Resolution: Cannot Reproduce
-
Affects Version/s: 2.0
-
Fix Version/s: 2.0.1
-
Component/s: couchbase-bucket
-
Security Level: Public
-
Labels:
Description
scenario:
add 2x2.0 nodes to a 3 node cluster with 2 buckets ( default and sasl where active resident ration is 60% ) few minutes after upgrade process begins active item ration on 10.3.2.43 which is an existing 1.8.1 node , drops from 60% -> 58->48->38-15->5 and then we decided to stop rebalance
i grabbed diags from the node which resident ratio dropped from 63 percent to 5 percent.
https://s3.amazonaws.com/bugdb/jira/systemtest/resident-ratio-drop-2853bb89.zip
please open a bug asap and mention this there. assign the bug to chiyoung and hoping that he or mike can take a look at the cluster.
[stats:debug] [2012-12-08 17:43:52]
active item resident ratio is at 63% and everything looks normal
and then at
[stats:debug] [2012-12-08 17:45:32]
vb_active_perc_mem_resident 58
and at
[stats:debug] [2012-12-08 17:47:12]
vb_active_perc_mem_resident 49
and at
[stats:debug] [2012-12-08 18:03:51]
vb_active_perc_mem_resident 35
what has happened between 17:45 and 17:47 or between 17:56 and 18:03 that pushed the resident ratio this low.
whatever it is there is a combination of 1.8.1 and 2.0 that is causing the issue
collect info from other nodes:
add 2x2.0 nodes to a 3 node cluster with 2 buckets ( default and sasl where active resident ration is 60% ) few minutes after upgrade process begins active item ration on 10.3.2.43 which is an existing 1.8.1 node , drops from 60% -> 58->48->38-15->5 and then we decided to stop rebalance
i grabbed diags from the node which resident ratio dropped from 63 percent to 5 percent.
https://s3.amazonaws.com/bugdb/jira/systemtest/resident-ratio-drop-2853bb89.zip
please open a bug asap and mention this there. assign the bug to chiyoung and hoping that he or mike can take a look at the cluster.
[stats:debug] [2012-12-08 17:43:52]
active item resident ratio is at 63% and everything looks normal
and then at
[stats:debug] [2012-12-08 17:45:32]
vb_active_perc_mem_resident 58
and at
[stats:debug] [2012-12-08 17:47:12]
vb_active_perc_mem_resident 49
and at
[stats:debug] [2012-12-08 18:03:51]
vb_active_perc_mem_resident 35
what has happened between 17:45 and 17:47 or between 17:56 and 18:03 that pushed the resident ratio this low.
whatever it is there is a combination of 1.8.1 and 2.0 that is causing the issue
collect info from other nodes:
Activity
Farshid Ghods
made changes -
| Field | Original Value | New Value |
|---|---|---|
| Summary | active item resident ratio drop significantly when adding a 2.0 node to 1.8.1 cluster for upgrade | active item resident ratio drop significantly when adding a 2.0 node to 1.8.1 cluster for upgrade ( sasl bucket ) |
| Labels | 2.0.0-hotfix | |
| Fix Version/s | 2.0.1 [ 10399 ] | |
| Fix Version/s | 2.0 [ 10114 ] | |
| Priority | Major [ 3 ] | Blocker [ 1 ] |
| Component/s | couchbase-bucket [ 10173 ] |
Farshid Ghods
made changes -
| Assignee | Chiyoung Seo [ chiyoung ] |
Farshid Ghods
made changes -
| Assignee | Chiyoung Seo [ chiyoung ] | Chisheng Hong [ Chisheng ] |
Mike Wiederhold
made changes -
| Sprint Status | Current Sprint |
Chisheng Hong
made changes -
| Status | Open [ 1 ] | Resolved [ 5 ] |
| Resolution | Cannot Reproduce [ 5 ] |
Mike Wiederhold
made changes -
| Sprint Status | Current Sprint |
2.0 node : https://s3.amazonaws.com/bugdb/jira/MB-7383/10.3.2.41-1292012-161-diag.zip
existing 1.8.1 node : https://s3.amazonaws.com/bugdb/jira/MB-7383/10.3.2.43-1292012-1551-diag.zip
existing 1.8.1 node : https://s3.amazonaws.com/bugdb/jira/MB-7383/10.3.2.47-1292012-1558-diag.zip