Couchbase/sync_gateway Guidance on how to deal with keeping track of "processed" documents

Mobile
1

I posted this question on github initially as I was not aware of this forum.

Background:
In our mobile app, users can perform actions that can result in processing of the commands in the backend. When we establish a web socket connection to get pending commands (documents), a “since” value has to be provided. This since value has to be tracked somewhere, so that if service goes down, next time it starts from the last sequence that was processed. So I thought I’ll create a doc and after each command is processed, I’ll update value with a sequence of the document processed. Well what I found out, when looking at couchbase server is that it keeps track of each revision. As we are talking about millions of commands, that would yield a doc that is just huge. What I really need is just a document such as “_sync:seq”, or a doc, that does not have all that revision, history etc. just a field with a number.
Any guidance would be highly appreciated.

lastProcessedDocumentSequence doc

{
“value”: 6
"_sync": {
“rev”: “3-688542223bca3bb9d516eb2b5728826d”,
“sequence”: 8,
“recent_sequences”: [
2,
5,
8
],
“history”: {
“revs”: [
“1-f282563eb44318552395c7e34e734aa2”,
“2-8f1d4129dfd6eebf628bd751792bfdeb”,
“3-688542223bca3bb9d516eb2b5728826d”
],
“parents”: [
-1,
0,
1
],
“channels”: [
null,
null,
null
]
},
“time_saved”: “2017-08-30T18:45:00.2041133Z”
},
}

2

Storing your ‘last seq’ value as a document in the bucket is a reasonable approach, but since that document isn’t being replicated, you don’t need or want a revision history (as you’ve identified).

There are a couple of alternatives:

  1. Use the server SDK to interact with that document.
    https://developer.couchbase.com/documentation/server/4.6/sdk/nodejs/start-using-sdk.html

  2. Store the document via Sync Gateway as a local document, using the /db/_local endpoint.
    https://developer.couchbase.com/documentation/mobile/1.4/references/sync-gateway/admin-rest-api/index.html#!/document/put_db_local_local_doc

3

We did a very similar thing. We needed to keep track of the last sequence that was processed when the change is received via the web socket by a back end service.We used the Couchbase SDK instead of the Sync Gateway.

Also, we needed to make sure that that particular sequence was only handled by one process. The document key for the cluster document included the sequence ID, and if two processes attempt to insert the same document, the SDK would report a KeyExists exception, indicating that the change was already being handled by another process.

So far it seems to work. We expire the documents after 5 hours since they are no longer relevant then. This allows for our services to go down for extended periods of time and then check the last sequence that was processed.

1 Like