[longevity] Rebalance hang after failover and remove node because of the memory leak on a couple of nodes

Cluster information:
- 11 centos 6.2 64bit server with 4 cores CPU
- Each server has 10 GB RAM and 150 GB disk.
- 8 GB RAM for couchbase server at each node (80% total system memmories)
- Disk format ext3 on both data and root
- Each server has its own drive, no disk sharing with other server.
- Load 9 million items to both buckets
- Cluster has 2 buckets, default (3GB) and saslbucket (3GB)
- Each bucket has one doc and 2 views for each doc (default d1 and saslbucket d11)
- Add one more doc d2 with 2 views to default bucket

* Start cluster with 10 nodes installed couchbase server 2.0.0-1663
10.3.121.13
10.3.121.14
10.3.121.15
10.3.121.16
10.3.121.17
10.3.121.20
10.3.121.22
10.3.121.24
10.3.121.25
10.3.121.23
* Data path /data
* View path /data

* The last run, I do swap rebalance remove node 13 and add node 26.
* Then node 26 failed due to physical failure. I failover node 26 and rebalance.
* Rebalance failed with known issue MB-6497 at the end of rebalance saslbucket
* Node 22 down due to run out of disk space. Failover node 22.
* Remove node 13. Start rebalance from 19:26:35 - Wed Sep 5, 2012

Bucket "default" rebalance does not seem to be swap rebalance ns_vbucket_mover000 ns_1@10.3.121.14 19:26:35 - Wed Sep 5, 2012

Rebalance hang until now Thu Sep 6 19:25:29 PDT 2012

CPU and beam stats

10.3.121.15
    Vm: 2796m Rm: 613m CPU: 13.7 beam.smp
    Vm: 6091m Rm: 4.2g CPU: 9.8 memcached
10.3.121.13
    Vm: 1845m Rm: 338m CPU: 9.9 beam.smp
    Vm: 1230m Rm: 1.0g CPU: 2.0 memcached
10.3.121.23
    Vm: 2443m Rm: 652m CPU: 9.8 beam.smp
    Vm: 4969m Rm: 3.4g CPU: 7.9 memcached
10.3.121.24
    Vm: 3304m Rm: 907m CPU: 19.4 beam.smp
    Vm: 5440m Rm: 4.0g CPU: 3.9 memcached
10.3.121.14
    Vm: 3462m Rm: 665m CPU: 30.7 beam.smp
    Vm: 6329m Rm: 4.1g CPU: 5.1 memcached
10.3.121.16
    Vm: 2702m Rm: 642m CPU: 13.2 beam.smp
    Vm: 4845m Rm: 3.5g CPU: 5.0 memcached
10.3.121.17
    Vm: 4498m Rm: 1.4g CPU: 91.2 beam.smp
    Vm: 5359m Rm: 3.6g CPU: 1.7 memcached
10.3.121.20
    Vm: 3793m Rm: 1.0g CPU: 11.7 beam.smp
    Vm: 5356m Rm: 3.7g CPU: 1.7 memcached

Swap stats in MB
                   Total Used Free
10.3.121.15
Swap: 5199 1815 3384
10.3.121.13
Swap: 5199 10 5189
10.3.121.22
Swap: 5199 15 5184
10.3.121.14
Swap: 5199 2503 2696
10.3.121.23
Swap: 5199 1037 4162
10.3.121.24
Swap: 5199 1543 3656
10.3.121.17
Swap: 5199 2156 3043
10.3.121.16
Swap: 5199 1156 4043
10.3.121.20
Swap: 5199 1949 3250


Link to diags of all nodes
https://s3.amazonaws.com/packages.couchbase/diag-logs/orange/201209/9nodes-1663-reb-hang-20120906.tgz