New replication using CBLite/Sync Gateway takes up lot of memory and CPU, causing the service to restart

[AakashJavia]

Couchbase server High CPU with sync gateway

Hi, I have the following installation : 1 server with Couchbase server EE 6.0.3 and sync gateway 2.5.0. 16 Gb RAM and 8 CPUs. Since yesterday, sync gateway seems to have an heavy impact on couchbase server. I started to see a huge CPU usage on one of the cbq-engine process (500-600%), and there are like 20-30K ops. per seconds on the statistics dashboard. It finally calmed down, I don’t know why. I stopped sync gateway this morning in order to start a import (7k documents), and then starte…

This post mentions that we can do batch imports. But so far I havent been able to find any direct documentation on how to do that.

We are using continuous replication through CBLite, and our sync gateway version is 2.0.0 and Couchbase version is 5.0.1 Community.

Any help is welcome. Thanks in advance!

[priya.rajagopal]

First off, would recommend upgrading Sync Gateway to latest version. There are a significant number of improvements to the performance of import processing in 2.7 . 2.0.0 is a very dated release.

Also, the batch imports is enforced through the server side. Ensure that your mutations on server are done in batches. i.e. instead of a UPDATE statement that updates 1 M documents in one shot, stagger the updates on the server bucket. Sync Gateway will process all changes as fast as it can.

[AakashJavia]

Hi Priya,
Thanks for the answer.
Our updates are performed asynchronously, and we ensure that at a time no more than 4 records are being updated via sync gateway.
Our issue is during the initial replication using CBLite when a new app is installed on the mobile, and all the necessary channel restricted data (120k docs in current case) needs to be pushed to the mobile. It is at this time that the performance issues are seen.
Thank you for suggesting the upgrade to SG2.7. I will look into it.
Have a great day,

[priya.rajagopal]

Then this is not an issue related to import processing as alluded to in the original post. It appears to do with replication with Couchbase Lite clients trying to sync a very large channel (of 120K documents). If your sync gateway is crashing, then it is likely that is running into OOM issues with the channel cache. But it is not clear if that’s what you are seeing - subject line indicates that you are observing a crash but your recent post indicates “performance issues”. Which is it ?
Either way, first part of response still holds true = upgrade to 2.7.0 version of Sync Gateway. There are enhancements to channel cache management that was introduced in 2.6 that should alleviate the issue of the cache potentially growing unbounded . There are also a number of config options that can be tuned.
Also, look into the sizing of your sync gateway and ensure that it is sized (RAM, cores) to handle the load. Track the CPU and RAM usage on Sync Gateway machine to ensure it is within thresholds.