Thank you very much for you details responses.
I adopt your Idea and will try to reproduce the issues.
Having said that, I’m not convinced that 62G data and 15G metadata requires 100G of ram and especially over usage of the cluster quota.
i’d expected the cluster to use exactly what was allocate to it, if i hit the water mark than ejections occurs. the 50G should be used to handle rebalances and stress times. that’s how i see it.
my hunch is that something isn’t working well in terms of memory management during the rebalance which causes over usage of cluster quota. currently i’m sure that my cluster as is cannot rebalance without affecting the application, (just because i learned the hard way that when the cluster is over used its quota - that’s a fact) therefore I think I’ll do XDCR and will update here how the cluster continue to behave, I’m sure others will be interested in that too 