Revision conflicts caused by replication

Bart · June 16, 2017, 12:59pm

Hello,

We are currently observing some odd behaviour with our application using CBLite and Sync Gateway.
The application is built around client server synchronization. The server creates a form document which is synced to the client. The client can then (potentially offline) fill in the form and submit it for processing by the server. This submission is done by modifying the status of the document from status “draft” to status “outbox”. The server monitors the channel of all documents in the “outbox” status and deletes them after processing. The clients are using CBLite (1.1.1 for iOS and 1.3 for Android) and the server communicates throught the Sync Gateway 1.5 REST service directly.

What we are observing is that once the document has been deleted by the server, Couchbase Lite on the client is once again starting to transfer old revisions of the document eventually leading to a secondary processing on the server (and sometimes even more than 2 times).

The logs of Sync Gateway show the following trace for a problematic form:

gist.github.com

https://gist.github.com/b-smets/d7afb4b89ba731f542a80b26bdade5df

gistfile1.txt

2017-06-14T15:37:30.870Z HTTP: #2114877: GET /XXX/568536a0-ec3b-46a2-b1b7-0370b8a7fa77 (as server)
2017-06-14T15:37:00.553Z HTTP: #2117640: PUT /XXX/568536a0-ec3b-46a2-b1b7-0370b8a7fa77 (as server)
2017-06-15T12:07:08.745Z HTTP: #2305145: PUT /XXX/568536a0-ec3b-46a2-b1b7-0370b8a7fa77?new_edits=false (as client)
2017-06-15T12:07:42.191Z HTTP: #2302075: GET /XXX/568536a0-ec3b-46a2-b1b7-0370b8a7fa77/default.xml (as server)
2017-06-15T12:07:10.322Z HTTP: #2305150: GET /XXX/568536a0-ec3b-46a2-b1b7-0370b8a7fa77 (as server)
2017-06-15T12:07:43.443Z HTTP: #2302078: DELETE /XXX/568536a0-ec3b-46a2-b1b7-0370b8a7fa77?rev=21-b4e9c67fc3cfd030aa87b0c390b35bfd (as server)

== It should normally stop here but it keeps going ==

2017-06-15T12:08:02.362Z HTTP: #2305244: PUT /XXX/568536a0-ec3b-46a2-b1b7-0370b8a7fa77?new_edits=false (as client)

This file has been truncated. show original

We also dit some capturing of the network traffic and I have extracted all individual requests between Sync Gateway and the client / server:

Dropbox - Error - Simplify your life (sensitive information has obviously been redacted)

Unfortunately we are running 2 reverse proxies so it’s a it tedious to determine the exact order of requests. What we can guarantee is that all requests per reverse proxy are in the order received.

Some peculiarities we observed ourselves:

i00/02 => Device immediately PUTs revision 21 in status outbox without ever putting any intermediate revision (as far as we know).
We assume this may be because the device had been offline for a long time.
i00/05 => Device requests both revision 1 and 22 in the bulk_get call. Why?
i00/06 => Device is PUTing revision 2 again and SG accepts it without issue.
i01/14 => Device is once again PUTin revision 21 in outbox status.
We compared the contents of the default.xml attachment in both revision 21s and observed very minimal changes.

We also looked at the document in Couchbase itself and saw the following:

gist.github.com

https://gist.github.com/b-smets/79fd3e8282f7d32e248aa0af9cc78b26

gistfile1.txt

{
  “_deleted”: true,
  “_id”: “568536a0-ec3b-46a2-b1b7-0370b8a7fa77”,
  “_rev”: “22-f366f03b2d4caf7deae09332e4308c1f”,
  “_sync”: {
    “rev”: “22-f366f03b2d4caf7deae09332e4308c1f”,
    “new_rev”: “22-5f7c55e567d65dca8d05f188901a4130",
    “flags”: 21,
    “sequence”: 159250,
    “recent_sequences”: [

This file has been truncated. show original

The document seems to contain a several conflicting revisions. We are also not sure how this can be caused as the document is at any time only touched by either the client (status draft) or server (status outbox).

We know that this second part all happened without interaction of the user. After the first processing, the user did not do anything to trigger this again. This all seemed to have happened internally in CBLite so I was wondering if you could provide me with some extra information.
Is this intended behaviour? Is there a way to prevent this?

If there is any additional information I can provide, please let me know.

priya.rajagopal · June 17, 2017, 3:35pm

Interesting. When the document is deleted on the server, the deleted revision should be replicated to the device so it is aware of this deleted revision - so unless the device is offline and is continuing to make changes when offline, you shouldn’t be seeing conflicts. It does not appear that your device is offline when the server is deleting the document so what you are observing is intriguing .
There is a bug that may be related to what you are observing so may be worth tracking
https://github.com/couchbase/sync_gateway/issues/2660
(CC @adamf)

Also, this blog post should give you a behind the scenes look at revision control in couchbase mobile which may be a useful read - https://blog.couchbase.com/conflict-resolution-couchbase-mobile/

jens · June 18, 2017, 12:46am

This sounds a lot like a recently-fixed Couchbase Lite bug involving revision pruning — I think the fix will be in the upcoming 1.4.1 release, but I’m not sure of that.

A workaround is to increase the maxRevTreeDepth parameter on the client side, or else to not make so many changes to a single document on the client in between syncing.

Bart · June 19, 2017, 9:00am

Ah very nice. Your remark gave me a consistently reproducable case:

Create a document and sync with SG
Disconnect device from internet
Create >20 revisions on the device (20 being the default maxRevTreeDepth)
Reconnect to internet and sync everything to SG again
=> Conflict

Setting the maxRevTreeDepth on the database to 100 indeed seems to resolve this (our users will never create 100 revisions while offline).

Thanks so much for the hint!