Home | Forums | Couchbase | Couchbase Server 1.8.x

Starvation of rebalancing after 'remove server'

2 replies [Last post]
Thu, 02/23/2012 - 05:56
Andreas JP
Offline
Joined: 02/23/2012
Groups: None

Hi,
I deployed a cluster with 7 nodes with a 2 or 3 replica bucket. I fill the cluster with 2 Mio keys (200b values) which allows roughly 50k insertions. I can read back 140khz probably client limited. When I remove one or several nodes and start the rebalancing procedure it works fine if the Cluster never went into TMP OOM.
If I overflow the cluster ( I artificially gave low memory on each machine) and I see TMP OOM rebalancing does not finish anymore.

Thanks for advice,
Andreas.

Top
Fri, 03/02/2012 - 20:51
mikew
Offline
Joined: 03/14/2011
Groups:

What is the memory quota when you change it to be a low value? Also I assume you still have 7 nodes and 2 million items in this scenario?

Top
Mon, 03/05/2012 - 07:48
Andreas JP
Offline
Joined: 02/23/2012
Groups: None

Hi, I have configured 7 x 128 MB = 896 MB.
The 2 million values take 400 MB + replication (2 copies) ==> exceeds the available memory and break the instance. Should be trivial to reproduce. In fact, I never have 2 Million, because I get TMP_OOM before and some of the insertions fail as expected.

Top