Couchbase Server 2.0 provides improved performance for server warmup; this is the process a restarted server must undergo before it can serve data. During this process the server loads items persisted on disk into RAM. One approach to load data is to do sequential loading of items from disk into RAM; however it is not necessarily an effective process because the server does not take into account whether the items are frequently used. In Couchbase Server 2.0, we provide additional optimizations during the warmup process to make data more rapidly available, and to prioritize frequently-used items in an access log. The server pre-fetches a list of most-frequently accessed keys and fetches these documents before it fetches any other items from disk.
The server also runs a configurable scanner process which will determine which keys are most frequently-used. You can use Couchbase Server command-line tools to change the initial time and the interval for the process. You may want to do this for instance, if you have a peak time for your application when you want the keys used during this time to be quickly available after server restart. For more information, see Section 7.6.3, “Changing Access Log Settings”.
The server can also switch into a ready mode before it has actually retrieved all documents for keys into RAM, and therefore can begin serving data before it has loaded all stored items. This is also a setting you can configure so that server warmup is faster.
The following describes the new initial warmup phases for the Couchbase Server 2.0. In these first phase, the server begins fetch all keys and metadata from disk. Then the server gets access log information it needs to retrieve the most-used keys:
Initialize. At this phase, the server does not have any data that it can serve yet. The server starts populating a list of all vBuckets stored on disk by loading the recorded, initial state of each vBucket.
Key Dump. In this next phase, the server begins pre-fetching all keys and metadata from disk based on items in the vBucket list.
Check Access Logs. The server then reads a single cached access log which indicates which keys are frequently accessed. The server generates and maintains this log on a periodic basis and it can be configured. If this log exists, the server will first load items based on this log before it loads other items from disk.
Once Couchbase Server has information about keys and has read in any access log information, it is ready to load documents:
Loading based on Access Logs Couchbase Server loads documents into memory based on the frequently-used items identified in the access log.
Loading Data. If the access log is empty or is disabled, the server will sequentially load documents for each key based on the vBucket list.
Couchbase Server is able to serve information from RAM when one of the following conditions is met during warmup:
The server has finished loading documents for all keys listed in the access log, or
The server has finished loading documents for every key stored on disk for all vBuckets, or
The total number of documents loaded into memory is greater
than, or equal to, the setting for
ep_warmup_min_items_threshold, or
If total % of RAM filled by documents is greater than, or
equal to, the setting for
ep_warmup_min_memory_threshold, or
If total RAM usage by a node is greater than or equal to the
setting for mem_low_wat.
When the server reaches one of these states, this is known as the run level; when Couchbase Server reaches this point, it immediately stops loading documents for the remaining keys. After this point, Couchbase Server will load this remaining documents from disk into RAM as a background data fetch.
In order to adjust warmup behavior, it is also important for you to understand the access log and scanning process in Couchbase Server 2.0. The server uses the access log to determine which documents are most frequently used, and therefore which documents should be loaded first.
The server has a process that will periodically scan every key in
RAM and compile them into a log, named
access.log as well as maintain a backup of
this access log, named access.old. The server
can use this backup file during warmup if the most recent access
log has been corrupted during warmup or node failure. By default
this process runs initially at 2:00 GMT and will run again in 24-
hour time periods after that point. You can configure this process
to run at a different initial time and at a different fixed
interval.
If a client tries to contact Couchbase Server during warmup, the
server will produce a ENGINE_TMPFAIL (0x0d)
error code. This error indicates that data access is still not
available because warmup has not yet finished. For those of you
who are creating your own Couchbase SDK, you will need to handle
this error in your library. This may mean that the client waits
and retries, or the client performs a backoff of requests, or it
produces an error and does not retry the request. For those of you
who are building an application with a Couchbase SDK, be aware
that how this error is delivered and handled is dependent upon the
individual SDKs. For more information, refer to the Language
Reference for your chosen Couchbase SDK.