[MB-7383] active item resident ratio drop significantly when adding a 2.0 node to 1.8.1 cluster for upgrade ( sasl bucket )

Wed, 2 Jan 2013 11:57:19 -0600

scenario:
add 2x2.0 nodes to a 3 node cluster with 2 buckets ( default and sasl where active resident ration is 60% ) few minutes after upgrade process begins active item ration on 10.3.2.43 which is an existing 1.8.1 node , drops from 60% -> 58->48->38-15->5 and then we decided to stop rebalance

i grabbed diags from the node which resident ratio dropped from 63 percent to 5 percent.
https://s3.amazonaws.com/bugdb/jira/systemtest/resident-ratio-drop-2853bb89.zip

please open a bug asap and mention this there. assign the bug to chiyoung and hoping that he or mike can take a look at the cluster.

[stats:debug] [2012-12-08 17:43:52]
active item resident ratio is at 63% and everything looks normal

and then at
[stats:debug] [2012-12-08 17:45:32]
vb_active_perc_mem_resident 58

and at
[stats:debug] [2012-12-08 17:47:12]
vb_active_perc_mem_resident 49

and at
[stats:debug] [2012-12-08 18:03:51]
vb_active_perc_mem_resident 35

what has happened between 17:45 and 17:47 or between 17:56 and 18:03 that pushed the resident ratio this low.
whatever it is there is a combination of 1.8.1 and 2.0 that is causing the issue

collect info from other nodes:

Couchbase

[MB-7383] active item resident ratio drop significantly when adding a 2.0 node to 1.8.1 cluster for upgrade ( sasl bucket )