Increasing Sync Gateway Throughput

Mobile Sync Gateway
1

Context

I have a Mac app using the Couchbase Lite Swift SDK that connects to Capella and Sync Gateway.

There are ~5,600,000 documents in the bucket, all of which must be synced down to client devices. I’m seeing a throughput of about 2,500 documents-per-second during initial replication, which means it takes almost 40 minutes to fully sync this database from scratch.

Question

Is 2,500 documents-per-second essentially the limit of the Replicator? Or are there ways to substantially increase this?

Details

  1. The documents are all fairly small. No attachments. (Think of an iTunes library with a bunch of songs that have artist, album, etc. fields).

  2. The network connection is not saturated (at least on the client end).

  3. The Database cluster is the lowest paid tier, paired with the lowest paid tier for the App Service node. (Together, these cost $470/mo, which is not insignificant and is about 4x more expensive than the equivalent setup at MongoDB, which is what we’re migrating from.)

  4. The Sync Gateway is connected to only one client for this test: me.

  5. Delta sync is disabled (I don’t think it would improve the initial sync times?)

  6. Unfortunately, syncing every document to the client is a requirement. I cannot improve sync times by limiting what syncs.

2

We faced similar issues with slow initial sync times and found a highly effective approach using Couchbase Lite’s prebuilt database capability, which dramatically reduces the time it takes for users to get up and running.

Here’s how it works, in simple terms:

On the backend (you do this once):

  1. Create a Couchbase Lite database locally.
  2. Connect it to your Sync Gateway URL and let it sync fully.
  3. Once all the data is pulled in, package the database as a zip.
  4. Include this zip file inside your app binary.

On the client (first app launch):

  1. Unzip the packaged database and initialize the local Couchbase Lite instance with it.
  2. Point it to the Sync Gateway URL.
  3. From there, it only syncs incremental changes, drastically reducing the amount of time spent waiting for the initial sync.

Official docs for this feature:

This approach has worked really well for us, especially in environments where first-time sync needs to be fast (e.g., mobile apps with lots of preloaded content or users in low-connectivity areas).

Curious to hear how others are handling this. Especially if anyone has used delta sync or compression techniques on top of this.

1 Like