A Tale of Two Ejection Methods: Value-only vs. Full

Introduction

I would like to thank our Couchbase Server engineering team – especially, Dave Rigby and Jim Walker – for their tremendous help with this article and endless patience with all my questions.  Thanks a lot guys, I really appreciate your vast knowledge and readiness to share it!

In Couchbase Server, data is stored in buckets.  The default bucket type – uncreatively called Couchbase type – assures asynchronous data persistence to disk.  The server will try to have all your data, the active set and the replica set(s), in memory.  However, if the in-memory data reaches 85% of the bucket memory quota (this level is called high water mark), the server will start ejecting (evicting) some of the not-recently-used from memory.  The active documents have 40% chance of being evicted, and the replica documents have 60% chance to be evicted.  The reason behind it is the following: it’s more important to have active document resident in memory; however, we do want some replica documents in memory as well, in case a node fails over, and the replicas will need to become active.  The eviction process continues until the in-memory data settles below 75% of the bucket memory quota (this level is called low water mark).

Just a brief reminder about the structure of the documents stored in Couchbase:

• Document key or ID: variable length – up to 250 bytes, must be unique for the documents stored within the same bucket
• Document metadata: fixed length – 56 bytes for documents stored in Couchbase bucket type, 72 bytes for documents stored in Ephemeral bucket type (this bucket type is outside the scope of this article)
• Document value: variable length – up to 20MB, usually JSON, other formats can also be used

Two Ejection Methods

By default, a Couchbase bucket uses Value-only ejection (or eviction) method.  Many versions ago, this was the only available option.  As the name implies, the ejection process will remove only the value of the documents and will keep the document keys and metadata in memory at all times.

Starting with version 3.0 (Couchbase Server is at version 6.5.1 at the time of this writing), we added the Full ejection method.  It is easy to guess that a bucket configured for full eviction will remove documents’ keys, metadata, and values as part of the ejection process.

Trade-off: Performance vs. Memory Size

So, which option should you pick for your buckets?  Here is what the tooltip explains in the GUI of our latest Couchbase Server version:

• Value Ejection: During ejection, only the value will be ejected (key and metadata will remain in memory).
• Full Ejection: During ejection, everything (including key, metadata, and value) will be ejected.
  Value Ejection needs more system memory, but provides the best performance. Full Ejection reduces the memory overhead requirement.

To be more precise, neither method “needs more system memory“: Full ejection allows for the dataset to exceed memory significantly, whereas Value ejection allows for the dataset to be larger than memory to a certain degree, since it must keep keys and metadata in memory.

If consistent read latency of the applications working with the data in Couchbase Server is (very) important, you should stick with value ejection.  This also requires you to size your cluster properly, assign sufficient memory quotas to your buckets, implement document retention policies (archive or remove old documents regularly), and monitor memory usage.

If staying within budget or existing technology limits is (very) important, then you should consider full eviction for your large buckets.  This may be the case when you are maxed out on your on-premise hardware resources, and there is no way to squeeze any more RAM into your Couchbase nodes.  When the money is tight for on-premise or cloud technology spending, the performance trade-off can save the day.

Another reason to consider full ejection is when you expect your dataset to be large, so that it will exceed reasonable memory configs.  How large is large?  Let’s say billions (yes, plural!) of documents, tens (also plural!) of terabytes of data.  Our Solutions Engineers and Solutions Architects are always here to help you size your Couchbase cluster properly – please, talk to us.

While it is possible to switch from one ejection method to another, it is one of the few operations that requires bucket restart and, therefore, some downtime.  We do show the following warning in red font in the UI when you attempt to change the ejection method: Changing eviction policy will restart the bucket. This will lead to closing all open connections and some downtime.  Thus, it is wise to decide on the ejection method for your buckets before rolling them out into production.

Why the Difference in Performance?

Let’s look at the different scenarios for bucket ejection methods and data residency.  This will help us understand how performance gets affected.

1. Full Data Residency

When your data is fully resident (i.e., it fits into ~85% of the bucket memory quota), the performance of value-only and full ejection buckets should be similar.  Document operations work like the shown below:

• Get (read) a document with key = profile::john-doe::123
  • FAST: check in memory if this key exists.
    • FAST: If yes, return the document from memory.
    • FAST: If not, return “document does not exist” error.
• Insert a document with key = profile::john-doe::123
  • FAST: check in memory if this key exists.
    • FAST: If yes, return “document already exists” error.
    • FAST: If not, write the document into memory.  Then asynchronously persist and replicate it.
• Replace a document with key = = profile::john-doe::123
  • FAST: check in memory if this key exists.
    • FAST: If yes, save changes into memory.  Then asynchronously persist and replicate them.
    • FAST: If not, return “document does not exist” error.

2. Partial Data Residency, Value-only Ejection

When the data residency is below 100% for a bucket with value-only ejection, Couchbase Server knows that document keys and metadata are still in memory.  Therefore, document operations will look like the following:

• Get (read) a document with key = profile::john-doe::123
  • FAST: check in memory if this key exists.
    • FAST: If yes, check if the document value is in memory
      • FAST: If yes, return the document from memory.
      • SLOW: If not, read the document from disk into memory and return it.
    • FAST: If not, return “document does not exist” error.
• Insert a document with key = profile::john-doe::123
  • FAST: check in memory if this key exists.
    • FAST: If yes, return “document already exists” error.
    • FAST: If not, write the document into memory.  Then asynchronously persist and replicate it.
• Replace a document with key = = profile::john-doe::123
  • FAST: check in memory if this key exists.
    • FAST: If yes, save changes into memory.  Then asynchronously persist and replicate them.
    • FAST: If not, return “document does not exist” error.

3. Partial Data Residency, Full Ejection

When the data residency is below 100% for a bucket with full ejection, Couchbase Server cannot rely on the presence of document keys and metadata in memory, because they could have been evicted.  Therefore, document operations will look like the following:

• Get (read) a document with key = profile::john-doe::123
  • FAST: check in memory if this key exists.
    • FAST: If yes, check if the document value is in memory
      • FAST: If yes, return the document from memory.
      • SLOW: If not, check the disk
        • SLOW: If the document is found on the disk, read the document from disk into memory and return it.
        • SLOW: If not, return “document does not exist” error.
• Insert a document with key = profile::john-doe::123
  • FAST: check in memory if this key exists.
    • FAST: If yes, return “document already exists” error.
    • SLOW: If not, check the disk
      • SLOW: If not, write the document into memory.  Then asynchronously persist and replicate it.
      • SLOW: If yes, return “document already exists” error.
• Replace a document with key = = profile::john-doe::123
  • FAST: check in memory if this key exists.
    • FAST: If yes, save changes into memory.  Then asynchronously persist and replicate them.
    • SLOW: If not, check the disk
      • SLOW: If not, return “document does not exist” error.
      • SLOW: If yes, write the document into memory.  Then asynchronously persist and replicate it.

With full eviction buckets that are not 100% memory-resident, many Exists operations have to go to the disk.  The number of background fetches (reading documents from disk to memory, also known as “cache misses”) is also likely to be higher for full ejection buckets.

Are There Ways to Improve Performance?

In addition to Insert and Replace operations, Couchbase SDKs support Upserts.  The upsert operation will either insert a document if one does not exist or replace the existing document.  Due to the append-only nature of handling data mutations in Couchbase Server, the upsert operations do not require corresponding exists operation.  So, the use of upserts instead of inserts and replaces will improve your application performance – provided that your use case allows for such a substitution.

Dedicated fast disks will also improve the performance of full ejection buckets.  It is one of the recommended configurations for Couchbase nodes: dedicated per-node disks over virtualized storage.

When to Switch from Value-only to Full Ejection

Let’s consider the following situation:

• You have a bucket configured with value-only ejection
• The bucket has grown to the point when less than 15-20% of the data is resident in memory, which is our recommended minimum residency ratio
• You anticipate more data coming in, you already archive/delete the documents in line with the use case requirements, and there is no way to provide more RAM to the bucket

Should you switch to full ejection method?

There is a good metric that can help you answer this question: the amount of metadata in RAM, compared to the user data in RAM for the same bucket.  By “metadata”, we mean document keys + metadata, in this particular case.  “User data” in this case means document value.

You can find these metrics under the vBucket Resources section in the UI.  On the screenshot below, we see that the metadata takes up about 42% of the in-memory bucket data:

Metadata overhead stats

At a certain point, you may start seeing messages like below in the UI and your Couchbase logs:

Metadata overhead warning

With so much of the memory quota taken up by document keys and metadata, there is little to no room to keep the document values.  In this case, you will see the server doing a lot of background fetches to load data from disk to memory, just to be ejected in the near future, since the available memory is too small.

If you cannot add more nodes to the cluster and/or reallocate memory from other buckets, then switching to full ejection method for the bucket at hand is advisable.  Just keep in mind: there will be some downtime while your bucket restarts and undergoes the warmup process.

Wrap-up

• Value-only ejection/eviction removes only document values from memory, while full eviction also removes document keys and metadata.
• You should make an informed choice of the ejection method before putting a bucket in production.  Write-heavy use cases with many millions of documents are good candidates for full ejection.  Changing the ejection method later in production can be costly due to some downtime.
• Upsert operations and dedicated fast disks always improve application performance, and even more important for full ejection buckets.
• Monitor the bucket metadata overhead.  It is a good indicator that your cluster may be in need of more RAM.  If you can’t get more memory to your Couchbase cluster, and your metadata overhead is at or higher than 40%, you may consider switching your large write-heavy buckets to full ejection.