How to get active docs resident % back to 100% after rebalancing

Couchbase Server
1

Hello, we have a 4 node cluster running 2.0.1. After taking some nodes out of the cluster, rebalancing, and then adding them back in (again rebalancing) our resident % is floating around 80%. How can we get this back to 100%?

Our cache miss ratio is zero. Thank you.

2

Hey in order to help you I do need a little bit more information;

  1. What sized items are you storing
  2. Were they near high water mark on nodes before rebalance?

Thanks

Laura

3
  1. Docs data size: 125G (spread across 4 nodes)
  2. Before the re-balance, resident % was at 100%.

I will also attach a screenshot of the performance metrics. Thanks!

couchbase.png946×946 86.3 KB

4

Hi @Alexi102000,
First of the drop in RP can happen due to backup jobs that have been run previously. I’m not sure if this is the case for you.
Running a backup job will take up memory from the head memory (this is to respect the overall memory allocated to the cluster). Reclaiming the memory from the backup job is a bit involved.

I have chatted with support about ways to get the RP to 100% again and it seems that the best way to do this is by replacing all nodes with new servers. I know this is not an easy task and this has also been address in CB v 4 to make this a less resources intensive task.

I hope this helps.

5

Thank you–that is bad news, but is very helpful. I don’t recall ever running any kind of backup job. Is there a way to replace all nodes with no production downtime?

Our lead developer indicates the following: Couldn’t it be that docs (in our
case, really just objects since we’re using Couchbase as a distributed,
persistent key-value store only) that are no longer accessed were never
re-loaded after the re-balancing since they’re not being used/accessed? Because interestingly, our cache
miss ratio is essentially 0%, indicating that all of our read operations are
being satisfied by RAM.