Version 10 by Steve Yen
on Jul 14, 2012 22:25.

compared with
Key
This line was removed.
This word was removed. This word was added.
This line was added.

Changes (39)

View Page History
{toc}
The XDCR binary protocol consists of the add with meta, get with meta, set with meta, and delete with meta commands. Their binary format is defined below.

The XDCR binary protocol involves several memcached binary commands to
replicate data and metadata...
h2. Set With Meta

* get meta
* add with meta
* set with meta
* delete with meta
* TAP mutation
* TAP delete

There are other REST/CAPI protocol commands, too, involved in XDCR but they are not detailed in this document.

For the relevant memcached binary commands, their binary format is detailed below.

h2. Status

Status of this document: in review (2012/07/14)

h2. CHANGES

* 2012-07-13 - The nru field was added to the XXX-WITH-META meta
field. Additionally, compared to some unreleased developer-interim
builds, the newCAS and seqNo fields have been reversed, where newCAS
now comes before seqNo.

h2. Set With Meta / Add With Meta / Delete With Meta

These commands (the so-called "XXX-WITH-META" commands) all follow the same layout.

High level layout...

{code:none}
XXX-WITH-META := header + body.
header := the usual 24 byte memcached binary request header, including a header-CAS field.
body := extras + key + value.
extras := flag + expiration + newCAS + meta.
meta := seqno + nru.
flag := 32-bits.
expiration := 32-bits.
newCAS := 64-bits.
seqno := 64-bits.
nru := 8-bits.
{code}

The header-CAS field follows the usual CAS semantics. That is, the
server will successfully process an XXX-WITH-META request only if the
header-CAS matches the item's current CAS value or if the header-CAS
value is 0.

The newCAS field, in contrast, is the CAS value that the server will
accept as the item's new CAS value if the request succeeds, allowing
the client to force a server begin using a given CAS value.

When the command is DELETE-WITH-META, the flag should be 0, the
expiration should be 0, and the value should be 0 length.

The meta field is the same meta field that's also provided by the
TAP_MUTATION command. The meta field is placed at the end of the
extras field so that any new meta information in future versions can
be easily handled by existing protocol users by just treating the meta
field as a variable length field.

Example...

{code:none}
Byte/ 0 | 1 | 2 | 3 |
/ | | | |
0| 0x80 | 0xa2 | 0x00 | 0x05 |
+---------------+---------------+---------------+---------------+
4| 0x194 | 0x00 | 0x00 | 0x00 |
+---------------+---------------+---------------+---------------+
8| 0x00 | 0x00 | 0x00 | 0x250 |
+---------------+---------------+---------------+---------------+
12| 0xde | 0xad | 0xbe | 0xef |
20| 0x00 | 0x00 | 0x00 | 0x00 |
+---------------+---------------+---------------+---------------+
24| 0x00 | 0x00 | 0x00 | 0x010 |
+---------------+---------------+---------------+---------------+
28| 0x00 | 0x00 | 0x00 | 0x0a |
+---------------+---------------+---------------+---------------+
32| 0xca 0xbe | 0xfe 0xef | 0xbca | 0xbfe |
+---------------+---------------+---------------+---------------+
36| 0xde | 0xad | 0xbea | 0xef 0xbe |
+---------------+---------------+---------------+---------------+
40| 0xbe 0xca | 0xef 0xfe | 0xcba | 0xfbe |
+---------------+---------------+---------------+---------------+
44| 0xde | 0xad | 0xbae | 0xbe 0xef |
+---------------+---------------+---------------+---------------+
48| 0x00 | 0x6d ('m') | 0x79 ('y') | 0x6b ('k') |
48| 0x6d ('m') | 0x79 ('y') | 0x6b ('k') | 0x65 ('e') |
+---------------+---------------+---------------+---------------+
52| 0x65 ('e') | 0x79 ('y') | 0x6d ('m') | 0x79 ('y') | 0x76 ('v') |
+---------------+---------------+---------------+---------------+
56| 0x76 ('v') | 0x61 ('a') | 0x6c ('l') | 0x75 ('u') | 0x65 ('e') |
+---------------+---------------+---------------+---------------+
60| 0x65 ('e') |
+---------------+

set with meta command
Opcode (1) : 0xa2
Key length (2,3) : 0x0005
Extra length (4) : 0x19 == 25 = 4 + 4 + 8 + 8 + 1
Extra length (4) : 0x14
Data type (5) : 0x00
Vbucket (6,7) : 0x0000
Total body (8-11) : 0x00000025 == 0x19 + 5 + 7 == 25 + 5 + 7
Total body (8-11) : 0x00000020 == 0x14 + 5 + 7
Opaque (12-15): 0xdeadbeef
CAS (16-23): 0x0000000000000000
Flags (24-27): 0x000000010
Expiration (28-31): 0x0000000a
New CAS (32-39): 0xcafebabedeadbeef
Seqno (40-47): (32-39): 0xbeefcafedeadbabe
NRU (48) : 0x00
Remote CAS (40-47): 0xcafebabedeadbeef
Key (49-53): (48-52): mykey
Value (54-60): (53-59): myvalue
{code}

The "quiet" versions of these commands have the same protocol layout.
Note that the format above can be used with different opcodes to define slightly different behavior. These opcodes define setq with meta (0xa3), add with meta (0xa4), and addq with meta (0xa5).

h3. Command Opcodes
h2. Delete With Meta

* CMD_SET_WITH_META = 0xa2
* CMD_SETQ_WITH_META = 0xa3
* CMD_ADD_WITH_META = 0xa4
* CMD_ADDQ_WITH_META = 0xa5
* CMD_DELETE_WITH_META = 0xa8
* CMD_DELETEQ_WITH_META = 0xa9
{code:none}
Byte/ 0 | 1 | 2 | 3 |
/ | | | |
|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|0 1 2 3 4 5 6 7|
+---------------+---------------+---------------+---------------+
0| 0x80 | 0xa8 | 0x00 | 0x05 |
+---------------+---------------+---------------+---------------+
4| 0x14 | 0x00 | 0x00 | 0x00 |
+---------------+---------------+---------------+---------------+
8| 0x00 | 0x00 | 0x00 | 0x19 |
+---------------+---------------+---------------+---------------+
12| 0xde | 0xad | 0xbe | 0xef |
+---------------+---------------+---------------+---------------+
16| 0x00 | 0x00 | 0x00 | 0x00 |
+---------------+---------------+---------------+---------------+
20| 0x00 | 0x00 | 0x00 | 0x00 |
+---------------+---------------+---------------+---------------+
24| 0x00 | 0x00 | 0x00 | 0x00 |
+---------------+---------------+---------------+---------------+
28| 0x00 | 0x00 | 0x00 | 0x0a |
+---------------+---------------+---------------+---------------+
32| 0xbe | 0xef | 0xca | 0xfe |
+---------------+---------------+---------------+---------------+
36| 0xde | 0xad | 0xba | 0xbe |
+---------------+---------------+---------------+---------------+
40| 0xca | 0xfe | 0xba | 0xbe |
+---------------+---------------+---------------+---------------+
44| 0xde | 0xad | 0xbe | 0xef |
+---------------+---------------+---------------+---------------+
48| 0x6d ('m') | 0x79 ('y') | 0x6b ('k') | 0x65 ('e') |
+---------------+---------------+---------------+---------------+
52| 0x79 ('y') |
+---------------+

h2. TAP_MUTATION engine-specific / meta field.
delete with meta command
Field (offset) (value)
Magic (0) : 0x80
Opcode (1) : 0xa8
Key length (2,3) : 0x0005
Extra length (4) : 0x14
Data type (5) : 0x00
Vbucket (6,7) : 0x0000
Total body (8-11) : 0x00000019 == 0x14 + 5 + 0
Opaque (12-15): 0xdeadbeef
CAS (16-23): 0x0000000000000000
Flags (24-27): 0x00000000
Expiration (28-31): 0x0000000a
Seqno (32-39): 0xbeefcafedeadbabe
Remote CAS (40-47): 0xcafebabedeadbeef
Key (48-52): mykey
{code}

The TAP_MUTATION binary protocol command has an engine-specific field
(also called "engine private field"). See...

http://www.couchbase.com/wiki/display/couchbase/TAP+Protocol

In 1.8, the engine private field is 0 bytes length.

In 2.0, the engine private field is equal to the meta field.

A TAP client can forward the meta field bytes from a TAP stream to
another server by using the XXX-WITH-META commaand.
For deleteq with meta use opcode 0xa9.