Using Eviction Effectively to Manage Memory in Couchbase GSI

Eviction

In Couchbase server, the Plasma storage subsystem used by Global Secondary Index is initialized with a specific value to be used as its memory quota. The storage subsystem constantly tracks its memory usage and adheres to its memory quota by freeing up allocated memory that is not mandatory. The freeing up of memory is often referred to as eviction. In this article we will give a description of the eviction algorithms used and the configuration parameters that can be tuned to effectively control the eviction process.

The storage subsystem organizes the data it manages into variable size pages in memory. It tracks its memory usage and starts to evict pages to disk when the memory usage goes over the assigned memory quota. After evicting sufficient number of pages, to bring the memory used below the quota, eviction stops.  This process of triggering eviction when memory used is over the memory quota is referred to as “Burst Eviction”. Burst Eviction stops mutations and scans till the memory used is below the quota.  This impacts the CPU usage, latency and throughput of the system negatively.

To mitigate the negative impacts on latency and throughput caused by Burst Eviction, another type of eviction, “Periodic Eviction”, also runs along with it.  Periodic eviction is triggered at regular intervals, called sweep interval, to scan pages in memory and track actively used pages. The subset of pages actively used by mutations or scans between the current and the previous sweep intervals is referred to as the working set. Evicting the pages that are not in the working set will create headroom in case of a spike in memory usage and also keep the needed pages in memory. When the memory used is either over the quota or above a preset minimum threshold, the pages not in the working set are evicted by the swapper thread. The swapper threads also run at regular, much shorter, intervals.

Memory quota, which is applicable across all plasma instances on a node is set through GSI. Plasma ensures that all the instances adhere to this memory quota by attempting to free up allocations that are not mandatory. When the memory used is over the quota the pages not in the working set are found and evicted by the swapper thread to bring down the memory used. This burst eviction stops once the memory used is below the quota. When burst eviction runs again it must again find the victims to evict. To make eviction efficient the swapper thread uses the clock algorithm to mark and sweep pages to be evicted. It runs periodically to mark the victim pages to be evicted making it easy for burst eviction to find victim pages to evict.

Configuration Parameter

The following configuration parameters are used to control the effectiveness of the eviction process to keep memory usage under the allocated quota.

evictSweepInterval:  Time interval in seconds at which periodic eviction will sweep through index pages to build the working set. By default, sweep runs every 600 seconds. This can be increased by the customer if they want to keep more memory used. The below commands are example for increasing this interval to 1200 seconds.

Statistics

The following storage statistics are useful in analyzing the effectiveness of storage memory usage. Based on these statistics the above eviction configuration parameters can be tuned to obtain an effective memory usage setup.

cache_hit_ratio: This is the ratio of cache hits to total number of attempts to read a page based on the activity in the last 5 seconds. The higher the cache hit ratio the better as going to disk is expensive.

rcache_hit_ratio: Similar to the stat cache_hit_ratio but only for scans

resident_ratio: This stat is the fraction of page records that are already in memory. This will vary based on the working set of the index. If this value is small and if the cache_hit_ratio Is high, it means the working set is small.