Limiting revisions on Couchbase Sync

Yes, the replicator will pull the current revision from the server. But if it can’t find the common ancestor, it will treat this as a conflict — it’ll still have its local revision as well as the one from the server.

Usually this isn’t a problem. The server revision will have a higher generation number (deeper tree) so it’ll be used as the default. But if your app checks for conflicts, it’ll se that there is a conflict.

1 Like

Cool, thanks. that would have been my next question. How to deal with conflicts when only pulling. But if the default is the server’s revision when no ancestor is found, it will give me some peace :).

following up on this:

I am not sure if the revs_limit=10 kicks in (I saw related posts, but unfortunately they weren’t conclusive).
What is the best way of debugging? Does the limit work immediate-effectively? Or is there some later pruning going on?

I created manually 63 revisions by repeatedly clicking save in the /_admin view on a specific document.
When I look at
http://localhost:4985/zf2016/?revs=true&revs_info=true
I still see 63 revisions in {"_revisions": “_ids”:[…]}

Setting revs_limit in the Sync Gateway config is correctly limiting the revisions for me, using the same Admin UI-based test you’re using.

What version of Sync Gateway are you using? Any chance you’ve got revs_limit defined in the wrong place in your config?

Here’s my config, which is working against the master branch of SG:

{
   "interface":":4984",
   "adminInterface":":4985",
   "log":["*"],
   "databases": {
      "default": {
        "server": "http://localhost:8091",
        "bucket": "default",
        "revs_limit": 10
      }
   }
}

@adamf
hmm, i put it there into the db document.
We have SG 1.2 - “version”: "Couchbase Sync Gateway/HEAD(nobranch)(3aa3c25)

{
“log”: [“CRUD”, “REST+”, “Access”],
“facebook”: {“register”: true},
“pretty”: true,
“databases”: {

“zf2016”: {
“server”: “http://localhost:8091”,
“bucket”: “zf2016”,
“revs_limit”: 10,
“unsupported”: {
“user_views”: {
“enabled”:true
}
},
a lot of other stuff coming: sync function, etc.

the revisions id array is still increasing beyong 10.

@adamf actually I looked into the DB now again and there were only 10 revisions in _raw as well as
/?revs=true&revs_info=true and _raw/

However, when I click 30 times on it again (within a few seconds). It has 40 entries again.
Is that due to some caching or late garbage collection?
As for now it doesnt seem to be immediate.

it works now immediately. sorry about the confusion, somehow some black magic happened :wink:

now it’s not workign anymore, after restarting couchbase, and sync-gateway. This system is starting to cause a major headache. It’s hard to debug, or to see if the configuration is valid. also once you have 100.000 documents, the sync gateway admin just becomes useless.

If you’ve got any additional information on the failure scenario, pass it along and we can try to reproduce.

You can use the /db/_config endpoint on the Admin REST API to see what the currently running config has defined as the revs_limit. If you’ve got a scenario where that’s showing a revs_limit that’s not being respected when updating docs, please share the details.

The Sync Gateway Admin UI is really just a development time tool at this point. We’re in the middle of work to significantly improve the Admin UI - it’s scheduled for an upcoming Sync Gateway release.

Perfect - I didn’t see the _config endpoint…
So, we had to scrape the couchbase behind, as there have been panic errors (JSON not parsable) and we believe the system just went completely bananas. Then it worked again.

Ok, since we are in production it worked fine for great 12h with 45.000 users. Then we restarted sync_gateway due to changes to the logger.

Then revs_limit has been not functional anymore and we have no clue how to diagnose or fix it.
For new documents it still works, for old it doesnt, and it is completely opaque why, and how that even happenend.
No conflicts, only one service updating documents regularly - how much can go wrong there!?

I am sorry to say this, but cb+sg is certainly not 1.x-ready and we’ve made the call to abandon it and to switch to a hosted alternative. SG is too generic that complexity and pitfalls outweigh its benefits.

I’m sorry to hear that you’ve ended up making that decision. I haven’t actually heard of other cases where users ran into these problems with revs_limit, and it’s unfortunate we haven’t been able to get to the bottom of it. I’ll review what kind of additional diagnostics we could include to help debug this issue in gurue.

On the off-chance you tried any of these before giving up, a few questions to help us get to the bottom of this:

  1. Were you able to retrieve the running config from the _config endpoint when you weren’t seeing revs_limit being applied properly, to validate that it still had the expected revs_limit? The fact that you’re seeing this on restart makes me wonder whether the correct config is being picked up.
  2. When you say ‘for new documents it works, for old it doesn’t’ - can you provide some more details on what’s a ‘new’ document and what’s an ‘old’ document in this scenario?

Thanks.

@adamf we have been facing the same issue. I have not been able to get to the http://1.1.1.1:6666/_admin/custombucket/db/_config endpoint. It just shows me a blank page with the following text and a textbox to enter docID.

0 documents, highlighted documents have access control output with the current sync function.

I have set the revs_limit to 50, but currently seeing 1679 revisions in the “revs” array of the document!!! It’s got a very erratic behaviour. I tested with creating docs before and after changing revs_limit to 10 and updating them and they both stuck to the limit. I then reduced the limit to lesser than the number of revisions the docs already had(reduced it to 5) and the next update showed only 5 revisions. If this is working, why do I have a document in the same bucket with 1679 revisions!!? The change to revs_limit=10 and 5 didnt seem to affect it one bit. There are numerous docs facing the same problem.

This has slowed down replications very significantly. Eagerly waiting for your response, Thank you.

A few comments:

  • The config endpoint is /{db}/_config. So in your case, it look like http://1.1.1.1:6666/custombucket/_config.
  • A document won’t get it’s revisions pruned until it’s updated - so if you have a document with 1679 revisions that hasn’t been updated since you modified the revisions limit, it won’t get updated
  • Sync Gateway prunes to the revs_limit for each non-deleted leaf revision in the revision tree. So if you have many unresolved conflicts in your document, you can still end up with more total revisions than revs_limit.

Ok, so the revs_limit has been set correctly. Thanks for pointing it out!
I updated the doc again, and now it has 1680 revisions.
The documents are created and updated only by the server. It is a single process so I doubt there will be any conflicts from there.
The only difference I found between the docs that stuck correctly to the revs_limit and the ones that did not, was that the latter were getting synced to around 30 users while the other docs were not getting synced to any user. Could it be that the syncing is somehow causing conflicts? There are no updates happening from the users though, only reads.
Can there be any other reason for the revs_limit to be exceeded?

I just checked the docs with ?open_revs=all and there were 2 leafs returned. Revision 1 and revision 1680. I guess that is why all the revisions in between were saved. On deleting the old leaf, the revs_limit is followed properly. Thanks for your help! :slight_smile:

Hi @Abhilash,

I havent looked into this for a while.
can you post the command you used to delete the leaf? Do I understand correctly that the number of revisions returned included all revisions of all leafs?

Best regards
Ulf

Hi @blanke,
This is the command I used:
http://1.1.1.1:6666/custombucket/custom_doc_id?open_revs=all

I read it somewhere on the couchbase Google group, cannot find the reference now. Apparently what it does is return only all leaf revisions of the particular document. It only returns last revision of each leaf. Something of this sort:

–cc60b50dbd410434ab65fe608f53778d9023d6ed100bd93c7983bd06ba75
Content-Type: application/json

{“_id”:“custom_doc_id”,“_rev”:“1-c1c47654dd39dbcfcf219e5e6977c678”,“body”:“body”}
–cc60b50dbd410434ab65fe608f53778d9023d6ed100bd93c7983bd06ba75
Content-Type: application/json

{“_id”:“custom_doc_id”,“_rev”:“1690-dc9f3927f635e548d1e519b1c1ef69da”,“body”:“body”}
–cc60b50dbd410434ab65fe608f53778d9023d6ed100bd93c7983bd06ba75–

What i did was delete the revision with the lower revision number and update the document. So the number of revisions now went down to revs_limit and ??_open_revs=all returned only the doc with revision number 1690. I tried executing the command again after a while though, and both the same revision numbers and docs came up.

Is what I am doing the right way to go about it? What exactly does ?_open_revs=all return?

Thanks in advance!

open_revs=all returns all non-tombstoned leaf revisions.

Can you provide some more details on what’s meant by: [quote=“Abhilash, post:23, topic:8590”]
I tried executing the command again after a while though, and both the same revision numbers and docs came up.
[/quote]

What command did you try to execute after a while, and what was the response?

@adamf, same issue we are facing. we had revision conflicts at 1st revision and we resolved by

https://developer.couchbase.com/documentation/mobile/current/guides/sync-gateway/resolving-conflicts/index.html

and we are resolving by current revision (the one that got picked deterministically by the system) remains the winner
when we update documents and looks like open_revs=all returning older revision which got deleted earlier.

below is _bulk_docs request body where revision 1-a42dc43aefec84c14cf854ecb4ec6995 was deleted earlier . but still we are receiving the same in every request of open_revs=all

{‘docs’: [{’_rev’: u’1-a42dc43aefec84c14cf854ecb4ec6995’, ‘_id’: u’3’, ‘_deleted’: True}, {u’ver’: u’dssdds’, u’_rev’: u’15-df476d53eb1caeaaee678f393296ed3b’, u’_id’: u’3’, u’type’: u’dd’}]}