Couchbase MutationMetaData fields

[VaibhavTech15]

Hi, is there any good documentation to understand the meaning of the MutationMetaData fields like flags, revSeqno, bySeqNo, lockTime etc?

[david.nault]

Hi @VaibhavTech15 !

Thanks for using the Kafka connector.

Are you asking about the metadata fields emitted by RawJsonWithMetadataSourceHandler and DefaultSchemaSourceHandler ? you can ignore most if not all of them. The connector has a long history, and some metadata fields are present only to prevent breaking existing user code that expects them. That said, here’s a quick reference.

---

bucket – name of the bucket where the event occurred.

partition – zero-based index of the Couchbase partition (also known as a “virtual bucket” or “vbucket”) where the event occurred.

vBucketUuid – opaque token unique to a history branch. (Some kinds of failover operations result in the creation of new history branches.) You can reliably compare two sequence numbers only if they are from the same partition and are associated with the same vBucketUuid.

key – Couchbase document ID of the affected document.

cas – “compare-and-swap” value after the change. Used for optimistic locking. You can divide it by something-or-other to get a timestamp for when the event occurred, but this is an implementation detail you really shouldn’t rely on.

bySeqno – funny name for the DCP sequence number. Each partition has its own sequence. It usually only goes up, but it can get “rolled back” after a failover. Gaps in the sequence can occur for any number of reasons; it’s not a good idea to do arithmetic on a sequence number.

revSeqno – number of times this document has been mutated / deleted. I’m pretty sure this gets reset after a compaction event. Personally, I wouldn’t rely on it. If you’re serious about tracking document versions, it’s best to include a version field in your document content.

expiration – epoch second when the document is scheduled to expire, or 0 if never.

flags – bitfield that hints at the “type” of the document (JSON, String, Binary, Serialized Java object, etc.) This is just advisory, and is not guaranteed to be set correctly. For example, it’s possible for a document containing valid JSON to be flagged as Binary or String. Take a look at CodecFlags.java from the Couchbase Java SDK to see how to interpret the flags.

If you need to check whether a document is JSON before publishing it to Kafka, the most reliable way is to parse it, like this.

lockTime – always zero. Please ignore this field.

---

As long as we’re talking about Couchbase document metadata, I’d like to mention that Couchabse Kafka connector version 4.2.6 added a couchbase.headers config property that lets you attach metadata as Kafka record headers (instead of polluting the record value). This works regardless of which source handler you use, and gives access to additional metadata like the Scope and Collection where the event happened.

Thanks,
David