Skip to end of metadata
Go to start of metadata
You are viewing an old version of this page. View the current version. Compare with Current  |   View Page History

Observe/Persistence Testing

This document will outline proposals for a new OOB command for fine-grained control over persistence and replication settings. This will be used to test our so-called durability operations and ensure they actually report correct status.

The problem with testing durability and persistence on a real server is the fact that the actual replication and/or persisting of a specific key depends on the physical latencies which exist in the disk and network hardware (or the CPU speed, if running in an all-ram, virtual machine environment). Thus there is a lack of certainty in determining what the expected state of the key is and what the expected OBSERVE response should be.

Injecting fine-grained commands in the mock will solve this issue as the persistence and replication status of a key will now be explicitly controlled via OOB command rather than dependent on hardware speed.

Concepts

At the basic level, there are multiple concepts which need to be dealt with here. We'll list them here

Nodes and Replicas

In a normal environment, a key is first stored to the master's cache (in memory) and is then placed within a replication queue to be distributed to other replicas. Because replication is dependent on the network, and persistence dependent on disk, we may have one of the following scenarios:

  • Key is cached in master, persisted in master, cached in replicas and persisted in replicas. This happens if the time to write to the master's disk is less than the time to send+persist to the replicas; so e.g.:
    1. Key is stored in master's RAM
    2. Key is placed in both master's write queue and replication queue
    3. Key is written to disk
    4. Key is sent over network to replicas
    5. Replicas persist item to disk
  • Key is cached in master, replicated to replicas, persisted to master, persisted to replicas. This happens if the write queue for the master is slower than the network replication queue; so e.g.
    1. Key is stored in master's RAM
    2. Key is placed in both master's write queue and replication queue
    3. Key is sent over network to replicas
    4. Key is written on master's disk
    5. Key is written on replicas' disks.
  • An even more extreme condition is where the master's disk is very slow, and the key is both persisted and replicated to the replicas before it is persisted to the master
    1. Key is stored in master's RAM
    2. Key is placed in both master's write queue and replication queue
    3. Key is sent over to network to replicas
    4. Key is written on replicas' disks
    5. Key is written on master's disk

Controlling Replication and Persistence status

The commands will feature options to:

  • Explicitly persist a key to master. A simple command would look like:
    • Command: kvstorage
    • Payload (Payload for Command):
      • Key: "Some Key" (The key to persist. The key does not have to exist yet, but will be set with these properties once a mutator command is sent via memcached)
      • Servers: (A dictionary of servers. The key is the server index. The server for each key must be either a master or a replica for its vBucket)
        • "1":
          • Persist: (Persistence Settings)
            • TimeOffset: 4 (Specify that the key should appear as "persisted" in 4 millis from now). This will not actually sleep but will set an internal timestamp (for which the subsequent 'OBSERVE' will not report anything until it returns). Note that this should be expressed as a Unix timestamp (in millis) rather than a time offset.
          • Replicate: (Replication Settings)
            • TimeOffset: 2 (Specify that the key should not be "replicated" to this node until "2" seconds from now)
        • "3":
          • Persist:
            • TimeOffset: 86
          • Replicate:
            • TimeOffset: 23
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.