Skip to end of metadata
Go to start of metadata
You are viewing an old version of this page. View the current version. Compare with Current  |   View Page History

Observe/Persistence Testing

This document will outline proposals for a new OOB command for fine-grained control over persistence and replication settings. This will be used to test our so-called durability operations and ensure they actually report correct status.

The problem with testing durability and persistence on a real server is the fact that the actual replication and/or persisting of a specific key depends on the physical latencies which exist in the disk and network hardware (or the CPU speed, if running in an all-ram, virtual machine environment). Thus there is a lack of certainty in determining what the expected state of the key is and what the expected OBSERVE response should be.

Injecting fine-grained commands in the mock will solve this issue as the persistence and replication status of a key will now be explicitly controlled via OOB command rather than dependent on hardware speed.

Concepts

At the basic level, there are multiple concepts which need to be dealt with here. We'll list them here

Nodes and Replicas

In a normal environment, a key is first stored to the master's cache (in memory) and is then placed within a replication queue to be distributed to other replicas. Because replication is dependent on the network, and persistence dependent on disk, we may have one of the following scenarios:

  • Key is cached in master, persisted in master, cached in replicas and persisted in replicas. This happens if the time to write to the master's disk is less than the time to send+persist to the replicas; so e.g.:
    1. Key is stored in master's RAM
    2. Key is placed in both master's write queue and replication queue
    3. Key is written to disk
    4. Key is sent over network to replicas
    5. Replicas persist item to disk
  • Key is cached in master, replicated to replicas, persisted to master, persisted to replicas. This happens if the write queue for the master is slower than the network replication queue; so e.g.
    1. Key is stored in master's RAM
    2. Key is placed in both master's write queue and replication queue
    3. Key is sent over network to replicas
    4. Key is written on master's disk
    5. Key is written on replicas' disks.
  • An even more extreme condition is where the master's disk is very slow, and the key is both persisted and replicated to the replicas before it is persisted to the master
    1. Key is stored in master's RAM
    2. Key is placed in both master's write queue and replication queue
    3. Key is sent over to network to replicas
    4. Key is written on replicas' disks
    5. Key is written on master's disk

Controlling Replication and Persistence status

The commands will feature options to:

  • Explicitly persist a key to master. A simple command would look like:
    • Command: persist
    • Payload (Payload for Command):
      • Key: "Some Key"
      • Value: (Optional) The value for the key.
      • Bucket: (Optional) The bucket the key resides on. If not provided, it is default
      • vBucket: The vBucket ID for the key. This must be provided
      • CAS: (Optional) - the CAS value for the key. If not supplied, the true cas will be used
      • OnMaster: Boolean. Whether the key should persist to master
      • OnReplicas: Number. How many replicas should this be persisted to

In addition to the persist command, there will also be:

  • unpersist - explicitly remove an item from persistent store
  • uncache - explicitly remove an item from the cache
  • persist_and_replicate - Store an item in the cache and disk
  • purge - Remove an item from both cache and disk
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.