Correct, you’ll hit issues there with rollback, which will look like a normal update. And there are other potential problems: if a document has transactional metadata changes staged, then the application crashes and cleanup is unable to rollback this document due to the Pending scenario described above - well you don’t want to accidentally filter out any subsequent modifications to that document just because it has a “txn” xattr. So I don’t believe filtering with the currently available information is viable - at least without tracking per-document state to try and disambiguate these cases, but I think that will get very complex, and will require learning about and embedding a lot of details about current implementation internals that could change down the line.
For rollback scenario, I guess, the rollback event will just look like a regular mutation event without any extended attributes. Or, do you think that there is a way to differentiate it from a regular mutation event which is triggered when a document is created in Couchbase ?
The former is correct, and there’s no way to differentiate it just from the DCP packet.
I would like to know - if the next mutation event to the document, when its not part of a transaction, will have the ‘txn’ xattr cleared atomically when the document is updated ? So that, we don’t see additional DCP events other than for the document-update.
When the transaction commits the document it will atomically remove the xattr.
I think you may be talking about cases like those mentioned above, e.g. a crashed transaction in Pending state, and a non-transactional update later is made to that document? In this case, the non-transactional update will not touch the “txn” xattr. Nothing changes xattrs unless the mutation explicitly does so (though user xattrs are removed when a document is deleted). That’s by design - the xattrs are most often used for metadata purposes, so regular non-metadata changes to the document’s body shouldn’t accidentally modify the metadata.
Our main problem is with test-automation. We are not able to reliably know how many events should we expect for assertion. Especially, because the book-keeping is Couchbase’s internal implementation and can change in the future.
Can I request for a feature to mark all events (transaction or otherwise) with a reason-code xattr and an API which can help us know the category of this mutation event, which uses this reason-code; or maybe an alternative solution ?
Ah ok, thanks for clarifying the use-case, which seems reasonable. We do already track this feature request internally, and there has been a good deal of prior discussion on it. Providing a solution isn’t as simple as it perhaps appears, with the interaction between filtering out events and DCP compaction a particularly thorny issue to resolve. So I don’t have a firm commitment or roadmap at this point, but please know that we are tracking it, and I’ll add this use-case and post to that context.
In the interim, I think the only workaround is to code knowledge of the current transactional internals into your tests. While this isn’t ideal, it is relatively safe as there are certain transactional fundamentals that are extremely unlikely to change at the protocol level, e.g. a document will very likely always need two mutations, one for staging and one for committing, unless we fundamentally change how and where we stage changes. And if we do introduce changes at the DCP level that I think for backwards compatibility they would be very likely to be opt-in and your tests would continue to work unchanged.
