Re-syncing locally purged documents


We want to limit the size of the couchbase-lite db on the device; so we have setup a time-to-live policy that locally purges the documents after a certain date (the documents still remain on the server).

Now if the user of the app wants to access the old, purged documents (say via scrolling down a bottom-less scroller of a list view that displays a never-ending stream documents), how can be re-get the locally-purged documents from the server? Can we do a REST-API call to the server and manually insert the documents returned from the API call to the couchbase-lite? OR somehow re-sync the old , locally-purged documents via replication?

I think if you create and run a replicator configured to pull only that document ID (or IDs), it will restore the document.

Not sure if this is correct. How will the mobile app code know which document id (or IDs) to pull?

You will have to keep track of them. The only other choice is to reset the replicator and wait for it to sync again from the beginning. Purge is designed to get rid of all traces of a given document.

So what you are telling is that it is not possible to keep the mobile lite database’s size within a certain limit ?

The user of the mobile app can go on generating documents in an unbounded manner and that can fill up significantly high volume on the device. So to solve this problem without “loosing” any data, we need to retain the copy of the data on the couchbase server but delete locally. BUT, at some point the user may want to look at the data that was deleted locally due to on device space considerations. At this time, what do we do?

It is surely possible to keep the mobile lite database small, but the solution is very domain specific. Usually this is accomplished by channels which will classify which data is important to which user / device.

If you expire or purge a document then all traces of it are gone locally. There are three ways to get it back, two involve client side actions and one is server side:

  1. Request that document ID specifically using the method @jens mentioned
  2. Reset the replicator checkpoint and have it start over from the beginning. This will pull down everything that was previously purged
  3. Update the document on the server side. This will cause it to be pulled down to the client as if it were a new document.

Okay thanks for the reply. Can’t I do this :

Let’s say I create channels based on date-ranges and automatically expire the documents locally behind a date.

When the app user scrolls to a date range that is beyond the expiry threshold, then make a synchronous replication call to the replicator to fetch all docs that belong to the date-range-channel. Until the replicator call returns, show a spinner in the UI and make the user wait.

Another approach is, to make a REST API call to the server (instead of the sync replication) to fetch docs belonging the missing date range and then insert them to the local database.

Are any of the two approaches feasible / reasonable ?

Never do this synchronously! Run the replication asynchronously and update the UI as the docs become available.

Another approach is, to make a REST API call to the server (instead of the sync replication) to fetch docs belonging the missing date range and then insert them to the local database.

Please don’t try to re-implement the replicator. CBL 2 lacks a public API that would let you insert these docs properly (preserving their revision metadata.) If you just insert them using the public API you will create new revisions, which won’t match the revIDs on the server; this will result in your client trying to push those revisions back to the server, causing a mess.

This would work if the goal is only to display the document contents in the UI.

However, we also want to display aggregates, for example the sum of values that are part of the documents(We have a view in the local database for this). During the time when documents are getting synced asynchronously , the aggregates will be incorrect until all the documents are downloaded.

Is there any solution for the aggregates problem mentioned in the above post?

You seem to want to have your cake and eat it too. You simultaneously want to get rid of the documents, but still be aware of them. Is there an issue with updating the aggregates as the information becomes available asynchronously?

Yes there is a problem with updating the aggregates asynchronously - accuracy - while all the docs that are needed for the aggregate to be correct are still syncing the aggregate value will be partially updated and hence incorrect.

This accuracy problem does not exist if the goal is to simply show the contents of the doc one by one, say in a list view, because each document is the atomic unit of sync.

Let’s say each document were a trip that the user drove, and the app in addition to showing the list of user’s trips also showed the total miles travelled in a month (which is an aggregate over all trips in that month). Let’s say we purged the trips on the phone after 1 year but we retain them on the cloud. If the user wanted to see the total miles in a month older than 1 year , we want to provide that aggregate value accurately.

Once the downloads are complete the value will be accurate will it not? There is no requirement to update the UI as you go along, but you should perform the replication in the background and then update the UI and everything else once you are done.

Okay - you mean update the UI when the replication of a month completes, as part of the replication completion callback ?

Yes, run a replication in the background (one-shot) and when it completes (you can check when it stops in the status callback) use the new information to update the UI and whatever else you need to update.

Okay thanks that might work.

I had a related question - can there be a callback for each document synced as part of a replication batch ?

The latest release that came out last week introduced this functionality. You have not mentioned your platform so I will just link the C# docs for the subject.

Thanks! Our platform is iOS Swift.