View Source

h1. Observe/Persistence Testing

This document will outline proposals for a new OOB command for fine-grained control over persistence and replication settings. This will be used to test our so-called durability operations and ensure they actually report correct status.

The problem with testing durability and persistence on a real server is the fact that the actual replication and/or persisting of a specific key depends on the physical latencies which exist in the disk and network hardware (or the CPU speed, if running in an all-ram, virtual machine environment). Thus there is a lack of certainty in determining what the _expected_ state of the key is and what the _expected_ OBSERVE response should be.

Injecting fine-grained commands in the mock will solve this issue as the persistence and replication status of a key will now be explicitly controlled via OOB command rather than dependent on hardware speed.

h2. Concepts

At the basic level, there are multiple concepts which need to be dealt with here. We'll list them here

h3. Nodes and Replicas

In a normal environment, a key is first stored to the master's cache (in memory) and is then placed within a replication queue to be distributed to other replicas. Because replication is dependent on the network, and persistence dependent on disk, we may have one of the following scenarios:
* Key is cached in master, persisted in master, cached in replicas and persisted in replicas. This happens if the time to write to the master's disk is less than the time to send+persist to the replicas; so e.g.:
*# Key is stored in master's RAM
*# Key is placed in both master's write queue and replication queue
*# Key is written to disk
*# Key is sent over network to replicas
*# Replicas persist item to disk
* Key is cached in master, replicated to replicas, persisted to master, persisted to replicas. This happens if the write queue for the master is slower than the network replication queue; so e.g.
*# Key is stored in master's RAM
*# Key is placed in both master's write queue and replication queue
*# Key is sent over network to replicas
*# Key is written on master's disk
*# Key is written on replicas' disks.
* An even more extreme condition is where the master's disk is very slow, and the key is both persisted _and_ replicated to the replicas before it is persisted to the master
*# Key is stored in master's RAM
*# Key is placed in both master's write queue and replication queue
*# Key is sent over to network to replicas
*# Key is written on replicas' disks
*# Key is written on master's disk



h3. Controlling Replication and Persistence status

In order to be able to test these situations in a controlled environment, a means for directly manipulating "persistent" and "cached" storage is required. The following commands will be implemented in the CouchbaseMock

The commands will feature options to:
* Explicitly persist a key to master. A simple command would look like:
** *{_}command{_}*: _persist_
** *{_}payload{_}* (Payload for Command):
*** *{_}Key:_* _"Some Key"_
*** *Value:* (Optional) The value for the key.
*** *{_}Bucket{_}*: (Optional) The bucket the key resides on. If not provided, it is _default_
*** *{_}vBucket{_}{*}*:* The vBucket ID for the key. This must be provided
*** *CAS:* (Optional) - the CAS value for the key. If not supplied, the _true_ cas will be used
*** *{_}OnMaster:_* Boolean. Whether the key should persist to master
*** *{_}OnReplicas:_* Number. How many replicas should this be persisted to


The full set of commands are:
* *{_}persist{_}* \- Store an item to disk
* *{_}unpersist{_}* \- Remove an item from dist
* *{_}cache{_}* \- Store an item to cache
* *{_}uncache{_}* \- Remove an item from the cache
* *{_}endure{_}* \- Store an item in the cache and disk
* *{_}purge{_}* \- Remove an item from both cache and disk
* *{_}keyinfo{_}{*}* *\- Returns information on the key. For each server (master and replicas) on the cluster, a dictionary is returned containing the key's status on its "Cache Store" and its "Persistent Store"


The formal parameters for these commands (in the *payload* field are) (fields with a '*' are required):

|| Name || Value Type || Description ||
| *Key\(*)* | String | _Required._ The key to operate on |
| *OnMaster\(*)* | Boolean | _Required._ Whether this operation should affect the master for this key |
| *OnReplicas\(*)* | Number or list of numbers | _Required_. Which replicas should be affected by this operation. If this is a number,\\
then _n_ replicas will be affected. If this is a list of numbers, each number in the list\\
is taken to be a replica index which should have its entry for _Key_ affected |
| *CAS* | Number | New CAS for the operation. Only valid on commands which store data |
| *Bucket* | String | Bucket on which the key resides on. is "_default_" if this parameter is not supplied |
| *vBucket* | Number | The vBucket for this key. If not supplied, the key is looked up internally\\
within the Mock's vBucket map |