Persistence and Replication

Skip to end of metadata
Go to start of metadata

Observe/Persistence Testing

This document will outline proposals for a new OOB command for fine-grained control over persistence and replication settings. This will be used to test our so-called durability operations and ensure they actually report correct status.

The problem with testing durability and persistence on a real server is the fact that the actual replication and/or persisting of a specific key depends on the physical latencies which exist in the disk and network hardware (or the CPU speed, if running in an all-ram, virtual machine environment). Thus there is a lack of certainty in determining what the expected state of the key is and what the expected OBSERVE response should be.

Injecting fine-grained commands in the mock will solve this issue as the persistence and replication status of a key will now be explicitly controlled via OOB command rather than dependent on hardware speed.


At the basic level, there are multiple concepts which need to be dealt with here. We'll list them here

Nodes and Replicas

In a normal environment, a key is first stored to the master's cache (in memory) and is then placed within a replication queue to be distributed to other replicas. Because replication is dependent on the network, and persistence dependent on disk, we may have one of the following scenarios:

  • Key is cached in master, persisted in master, cached in replicas and persisted in replicas. This happens if the time to write to the master's disk is less than the time to send+persist to the replicas; so e.g.:
    1. Key is stored in master's RAM
    2. Key is placed in both master's write queue and replication queue
    3. Key is written to disk
    4. Key is sent over network to replicas
    5. Replicas persist item to disk
  • Key is cached in master, replicated to replicas, persisted to master, persisted to replicas. This happens if the write queue for the master is slower than the network replication queue; so e.g.
    1. Key is stored in master's RAM
    2. Key is placed in both master's write queue and replication queue
    3. Key is sent over network to replicas
    4. Key is written on master's disk
    5. Key is written on replicas' disks.
  • An even more extreme condition is where the master's disk is very slow, and the key is both persisted and replicated to the replicas before it is persisted to the master
    1. Key is stored in master's RAM
    2. Key is placed in both master's write queue and replication queue
    3. Key is sent over to network to replicas
    4. Key is written on replicas' disks
    5. Key is written on master's disk

Controlling Replication and Persistence status

In order to be able to test these situations in a controlled environment, a means for directly manipulating "persistent" and "cached" storage is required. The following commands will be implemented in the CouchbaseMock

The commands will feature options to:

  • Explicitly persist a key to master. A simple command would look like:
    • command: persist
    • payload (Payload for Command):
      • Key: "Some Key"
      • Value: (Optional) The value for the key.
      • Bucket: (Optional) The bucket the key resides on. If not provided, it is default
      • vBucket: The vBucket ID for the key. This must be provided
      • CAS: (Optional) - the CAS value for the key. If not supplied, the true cas will be used
      • OnMaster: Boolean. Whether the key should persist to master
      • OnReplicas: Number. How many replicas should this be persisted to

The full set of commands are:

  • persist - Store an item to disk
  • unpersist - Remove an item from dist
  • cache - Store an item to cache
  • uncache - Remove an item from the cache
  • endure - Store an item in the cache and disk
  • purge - Remove an item from both cache and disk
  • keyinfo - Returns information on the key. For each server (master and replicas) on the cluster, a dictionary is returned containing the key's status on its "Cache Store" and its "Persistent Store"

The formal parameters for these commands (in the payload field are) (fields with a '*' are required):

Name Value Type Description
Key()* String Required. The key to operate on
OnMaster()* Boolean Required. Whether this operation should affect the master for this key
OnReplicas()* Number or list of numbers Required. Which replicas should be affected by this operation. If this is a number,
then n replicas will be affected. If this is a list of numbers, each number in the list
is taken to be a replica index which should have its entry for Key affected
CAS Number New CAS for the operation. Only valid on commands which store data
Bucket String Bucket on which the key resides on. is "default" if this parameter is not supplied
vBucket Number The vBucket for this key. If not supplied, the key is looked up internally
within the Mock's vBucket map
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.