Couch base Mobile Sync speed enhancement

Hi All,

I have around 65000 documents on the couch base server. It is hosted on AWS. Total 2 nodes of server, 2 sync gateway and 1 Nginx load balancer.
We have good internet speed (~50 MBPS lease line) All VM on AWS are 8 Core and 16 GB RAM.
All the documents take around 4 minutes to sync in one shot.

App cant be used until all documents are downloaded.

I have found that bulk get call is sent for 100 documents at a time. Is it possible to increase it on Android?

Or if there is some optimizations like creation of index on Server, Or OS level tuning. We are using ubuntu 14.XX.

I need to bring down this to seconds.

Please help

Regards
Pankaj Sharma

The size of the bulk_get request isn’t usually a limiting factor. Usually the bottlenecks are CPU and storage write speeds on the device. I suggest profiling the Android app to see where the time is going.

Are you certain the app is unusable without all the data? Can you present the data incrementally as it arrives? Or can you partition the documents in channels, so that pulling one channel first will allow the app to proceed, while you then pull the rest of the docs in the background?

to check this I developed a Console APP which runs on a work station. That took around 5 mins. Are you suggesting that bulk get call is asynchronous. If yes then how many threads can be created. Or is it sequential.

Right now we have partitioned the data into channels and then it has come down to 65k docs. We are looking to partition it more and give priority to documents which can give us Must have data for App to start working.

But in this case issue is the data we are assuming on less priority. If that data is required on App before it gets sync then what is the best way to handle it ?

Again is there any option available on Android to boost up Sync and use all resources possible. Because our App will be exclusive for the device.

to check this I developed a Console APP which runs on a work station. That took around 5 mins. Are you suggesting that bulk get call is asynchronous. If yes then how many threads can be created. Or is it sequential.

It’s not in itself synchronous or asynchronous; it’s an HTTP request, so it depends on how it’s implemented. Sync Gateway is highly parallel and services every TCP socket in a separate thread (goroutine). I’m not familiar with the HTTP client library we use on Android, but I’m sure it’s multithreaded and uses multiple sockets.

Increasing the number of docs per request would lower the per-request overhead (i.e. generating and parsing HTTP headers), but that’s pretty small. The downside is that if the response is buffered in RAM instead of being streamed, it will consume more memory and have higher latency. I’m not sure whether this applies to Sync Gateway or Android; it does on iOS.

Another factor is that CBL has to know which docs to request, and those are coming in from the server changes feed. After that, CBL has to look up every doc in the database to see whether it already has that revision or not. So there’s a pipeline involved, and we have to regulate the flow control between stages to avoid buffering tons of stuff in memory if one stage runs slower than the others.

Again, I suggest you profile the app to see where the bottlenecks are, as that will help a lot in narrowing down how to optimize it.