Hello, we have a 4 node cluster running 2.0.1. After taking some nodes out of the cluster, rebalancing, and then adding them back in (again rebalancing) our resident % is floating around 80%. How can we get this back to 100%?
Our cache miss ratio is zero. Thank you.
Hey in order to help you I do need a little bit more information;
- What sized items are you storing
- Were they near high water mark on nodes before rebalance?
- Docs data size: 125G (spread across 4 nodes)
- Before the re-balance, resident % was at 100%.
I will also attach a screenshot of the performance metrics. Thanks!
First of the drop in RP can happen due to backup jobs that have been run previously. I’m not sure if this is the case for you.
Running a backup job will take up memory from the head memory (this is to respect the overall memory allocated to the cluster). Reclaiming the memory from the backup job is a bit involved.
I have chatted with support about ways to get the RP to 100% again and it seems that the best way to do this is by replacing all nodes with new servers. I know this is not an easy task and this has also been address in CB v 4 to make this a less resources intensive task.
I hope this helps.
Thank you–that is bad news, but is very helpful. I don’t recall ever running any kind of backup job. Is there a way to replace all nodes with no production downtime?
Our lead developer indicates the following: Couldn’t it be that docs (in our
case, really just objects since we’re using Couchbase as a distributed,
persistent key-value store only) that are no longer accessed were never
re-loaded after the re-balancing since they’re not being used/accessed? Because interestingly, our cache
miss ratio is essentially 0%, indicating that all of our read operations are
being satisfied by RAM.