200% beam.smp usage during rebalance after failover ( cluster becomes unresponsive) - amazon ec2

this happened while running failover tests :

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 1184 membase 20 0 1192m 120m 2472 S 195.0 1.6 16:48.01 beam.smp
 1224 membase 20 0 2878m 2.0g 3048 S 3.9 28.1 10:49.18 memcached
  390 root 20 0 0 0 0 S 2.0 0.0 0:08.14 jbd2/xvda1-8
    1 root 20 0 19108 1388 1144 S 0.0 0.0 0:00.35 init
    2 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kthreadd
    3 root 20 0 0 0 0 S 0.0 0.0 0:00.53 ksoftirqd/0
    4 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/0
    5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/0
    6 root RT 0 0 0 0 S 0.0 0.0 0:00.00 migration/1
    7 root 20 0 0 0 0 S 0.0 0.0 0:01.36 ksoftirqd/1
    8 root RT 0 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
    9 root 20 0 0 0 0 S 0.0 0.0 0:00.78 events/0
   10 root 20 0 0 0 0 S 0.0 0.0 0:00.44 events/1
   11 root 20 0 0 0 0 S 0.0 0.0 0:00.00 cpuset
   12 root 20 0 0 0 0 S 0.0 0.0 0:00.01 khelper
   76 root 20 0 0 0 0 S 0.0 0.0 0:00.00 netns
   77 root 20 0 0 0 0 S 0.0 0.0 0:00.00 async/mgr

1- build 30 node cluster with
2- failover 3 nodes , 30->27
3- rebalance

loop through 1-3

this happened when rebalancing from i think 18->15


attached the diags from mbbrowselog and also the erlang crash dump file generated by running killall -SIGUSR on beam.smp