Sync gateway - DOC filtering

Hi Team,
we are trying to filtering some documents in sync-gw sync function. For example: if the document type is “XXX” we want to filter out that document at sync-gw level. Below is the function,

var mstDomains=[“Customer”,“Manifest”,“Route”]
console.log(“This is masterdata domain doc:”+doc._id+",so don’t persist in trandb");
throw ({ forbidden: "Can’t persist Masterdata domain doc in transaction domain " });
This works fine and in the replicator side we are getting this error captured. But the problem is performance, without this filter we can sync 100,000 document < 20 sec but if we add this the total time taken is around 3 to 4 min. Is there a better way to do this.

we know we can add the filter in mobile replicator itself but we are trying to achieve the same in sync-gw level as well.


Filtering on client side is the preferred design approach. Avoids unnecessary use of network bandwidth and processing overhead on the sync gateway as well as messaging overhead (the rejection notification needs to be sent back to client and processed as well) . Otherwise, you will have to scale up the sync gateway to handle extra load

Priya, thank you for that answer. I’ve been working with Nagarajan on this and agree with your advise on Filtering.

In this case, the purpose is really for Validation and we are checking the doc type as a failsafe to ensure only valid documents are accepted. The client side is already filtering as you suggested, this validation is intended as a failsafe in the event a new doc type is introduced which doesn’t get filtered out by the client.

We’ve done some additional testing and have observed that the bulk of the time consumed in the sync script is in the aforementioned “messaging overhead”, the rejection notification via throw().

Are there any alternative script options that would reject a document without sending notification? We would like to reject (or simply skip) the document without incurring the messaging overhead.

If you are filtering on client side, then I am confused about the performance concern that you cite. As I understand the check on the sync gateway is a second level check just as a failsafe mechanism - so on normal operation, you shouldn’t be seeing any of these documents on sync gateway.
Can you elaborate on how a new domain type is introduced on the client side without the corresponding app and/or server change . Who is creating these domains? If a document on client is created with a new domain type, then shouldn’t the app be able to process the document appropriately? If you envision the domain types to change, then perhaps the app should filter on only domain that needs to sync (i.e reject all documents that are not in a specific domain)