Very slow replication performance when using prebuilt database on couchbase-lite

We are utilizing prebuilt database so that we can spin up our mobile application a little bit faster. We created a user and assigned a single channel to authenticate and replicate our most common documents from sync-gateway. We compressed the cblite2 file and store it in S3 which will then get downloaded by our mobile app (running cblite 2.7), decompressed and copied. Our app on each device will have a different user with that same single channel assigned to the sync gateway user above plus a few other channels.

After copying the prebuilt database onto our mobile application, the app will attempt to synchronize other documents that weren’t included in the prebuilt database. So far, we are seeing the data being replicated taking much longer than if we had not used the prebuilt database. As an example, it would take us 40-50 minutes to replicate 1.07 million records to our mobile application. However when we utilize our prebuilt database (1.05 million documents), it would take no time to download and copy the prebuilt database, but then it would take 2.5 - 3 hours to synchronize the other documents (20,000 records).

I hope this is not expected behavior. Can you help us figure out how to remedy the issue?

I work with @rahimrahman and we’re still trying to get to the bottom of this problem. I suspect the problem lies in the fact that the user account used to create the prebuilt database does not match the one used on the device after the prebuilt database is installed. Furthermore, that 2nd user account has additional channels assigned to it. I suspect the replicator is therefore unable to make use of the last checkpoint and has to download the changes feed starting from scratch.

Could that be the cause of this slow replication?

The problem we’re trying to solve is that we have a large database to bootstrap on the device with 1M - 2M+ documents. The documents are also not entirely the same for all devices. Although the majority of the documents are assigned to a common channel sent to all the devices, some are not. We had hoped to mitigate the slowness of the initial download by creating a prebuilt database with the common set of documents, so after installing it, only those remaining few assigned to other channels would have to be replicated.

Unfortunately, prebuilt databases haven’t helped. It actually takes longer to fully replicate starting with a prebuilt database than starting from scratch.

that 2nd user account has additional channels assigned to it. I suspect the replicator is therefore unable to make use of the last checkpoint

You may be right. The checkpoint ID is computed based on a hash of the replicator’s remote URL, channels, and the docIDs to restrict to (if any). If those are not the same as the ones used to populate the prebuilt database, the replicator will be using a different checkpoint ID, whose value will start out empty, meaning it will replicate from scratch. (It doesn’t download the bodies of docs it already has, but it does have to download their docIDs and revisionIDs and look them up in its database.)

I suggest you define a set of common channels that every user will be pulling, then have your prebuilt database populated by pulling only those channels. On the client you can start a replication using the same set of channels — this will match the checkpoint and efficiently pull in only the new/updated docs.

You’ll also need to start a separate replication that pulls client-specific channels. Hopefully those channels don’t have so many documents.

I’ll give that a try, Jens. Thanks for the suggestion and come back with the result.