Sync - why do I receive these docs. on mobile - and can I stop them?

I have a setup where on mobile I create certain documents that I only want to sync. TO the server. However, for one doc. type I can see that it is sync’ed to the device (if I set up the app on a new device and log in). I thought I had controlled that via the import_filter function -but I see that these docs (for the specific user) is sent to the device.

The docs in question has type="Image". This is my filter_import and sync functions:

       "import_filter": `
            function(doc) {
               // Some document types not allowed on mobile
               if (doc.type == 'EnvLake' || doc.type == 'EnvMeasurement' || doc.type == 'ActivityLog' || doc.type == 'Feedback' || doc.type == 'Image') {
                return false;
                }
                if ((doc.type == 'FishingTrip' || doc.type == 'Catch') && typeof doc.userkey === 'undefined') {
                    return false;
                }
                return true;
            }
            `,
       "sync": `
            function (doc, oldDoc) {
                function _log(t) {
                    // Write to sg_info.log
                    console.log('SG: ' + t);
                }
                function _getUserKey(d) {
                    var key = null;
                    if (d) {
                        if (d.type == 'User') {
                            key = d.key;
                        } else {
                            key = d.userkey;
                        }
                    }
                    return key;
                }
            
               if (doc && doc._deleted) {
                    // Doc. deleted -> if public then require
			if(oldDoc){
	                    var userkey = _getUserKey(oldDoc);
	                    _log('delete doc id: ' + doc._id + ', userkey=' + userkey);
    	                if (userkey != null) {
		                    requireUser(userkey);
            	        } else {
                    	    requireAdmin();
			}
                    }
                    _log('doc deleted, id: ' + (doc._id || 'no id!') + ', ' + (oldDoc ? ('old key=' + oldDoc.key + ', userkey=' + userkey) : 'no oldDoc'));
                    return;
                }
                _log('doc id: ' + (doc._id || 'no id!') + ', ispublic: ' + doc.ispublic + ', userkey=' + userkey + ', ' + (oldDoc ? (oldDoc._deleted ? 'oldDoc is deleted' : ('old key=' + oldDoc.key + ', oldDoc.userkey=' + oldDoc.userkey + ' update')) : ' creation'));
                // Document type is mandatory
                if (typeof doc.type === 'undefined') {
                    _log('Document type missing: ' + JSON.stringify(doc));
                    throw ({ forbidden: "Document type is required. id=" + doc._id });
                }
                // Document key is mandatory
                if (typeof doc.key === 'undefined') {
                    _log('Document key missing: ' + JSON.stringify(doc));
                    throw ({ forbidden: "Document key is required. id=" + doc._id });
                }
                // Update: Cannot allow change of type or key
                if (oldDoc != null && !oldDoc._deleted) {
                    // Update
                    if (oldDoc.type != doc.type) {
                        throw ({ forbidden: "Can't change doc type" });
                    }
                    if (oldDoc.key != doc.key) {
                        throw ({ forbidden: "Can't change doc key" });
                    }
                }
                // Document sync is disabled (used for type Image - but generic implementation)
                if (doc.issyncdisabled) {
                    throw ({ forbidden: "Sync. disabled for id=" + doc._id });
                }
                // All public docs are available in the app
                if (doc.ispublic) {
                    _log('public, id: ' + (doc._id || 'no id!'));
                    channel('!');
                }
                // All fishing trips and catches from users (not external reporting) are available (for stats)
                if ((doc.type == 'FishingTrip' || doc.type == 'Catch') && typeof doc.userkey !== 'undefined') {
                    _log('Trip/catch for user: ' + doc.userkey + ', id: ' + (doc._id || 'no id!'));
                    channel('!');
                 }
                // All users are available (for stats)
                if (doc.type == 'User' && doc.deleted != true) {
                    _log('User doc, id: ' + (doc._id || 'no id!'));
                    channel('!');
                 }

                // Allow anyone to create a Feedback or Observation on the server
                if (oldDoc == null && doc.userkey == null && (doc.type == 'Feedback' || doc.type == 'Observation')) {
                    _log('Created ' + doc.type + ': ' + (doc._id || 'no id!') + ', key: ' + doc.key + ' as anonymous user ');
                    return;
                }

                // Only non-public docs "owned" by user can be created/updated (and replicated)
                var userkey = _getUserKey(doc);
                if (userkey != null) {
                    if (oldDoc != null && ! oldDoc._deleted) {
                        // Update
                        if (oldDoc.userkey && oldDoc.userkey != doc.userkey) {
                            throw ({ forbidden: "Can't change user key" });
                        }
                    }
                    _log('User owned, id: ' + (doc._id || 'no id!') + ', type: ' + doc.type + ', user: ' + userkey);
                    channel('channel.' + userkey);
                    access(userkey, 'channel.' + userkey);
					requireUser(userkey);
                } else if (doc.ispublic) {
	                requireAdmin();
                } else {
                    // Creation/update without user
                    _log('Document type cannot be created without user key: ' + (doc.type === 'Image' ? doc._id : JSON.stringify(doc)));
                    throw ({ forbidden: "This document type cannot be created without user key. id=" + doc._id });
                }
             }
		`,

NB! I have a field issyncdisabled that I use to control when a doc. is synced from the device to the server. This is used to save data locally while the user creates it but not sync until the entry is complete - just as a note as it also mentions “Image” :slight_smile:

So question is: Can I modify the filter to make sure that the docs don’t sync to the device if they only exist on the server?
… or should I handle that in the sync function - and if so, then how?

The mechanism for defining the set of documents being replicated to clients is channel assignment. Clients will only pull documents that are assigned to channels they have access to.

If the application on a new device is replicating a document from the server, this means that document is getting assigned to a channel that the device user has access to.

More details here: https://docs.couchbase.com/sync-gateway/current/sync-gateway-channels.html

Ok, so the importFilter is not used to entirely exclude them from the sync. function?

That was the impression I had earlier from previous discussions here…

The import process generates mobile metadata for documents that are NOT written via Sync Gateway (e.g. via a Couchbase SDK as part of a server-side application). The import filter only applies to these documents.

Documents being replicated from Couchbase Lite via Sync Gateway are not in this category.

Hmmm… Ok. These documents are being replicated TO Couchbase Lite via Sync Gateway which is why I thought they were covered…

As you’ve seen, import is write-time filtering (for documents not originating from Couchbase Lite), and not a read-time filtering (pull replications to Couchbase Lite). Hope that helps.

1 Like

Ok, so what do I do to make sure that I can send a document from CB Lite to the server - but not send a document of the same type back to CB Lite?

From what you say I understand that leaving the “Image” document out of any channels should avoid it being send to CB Lite. But I still need it to be able to send from CB Lite to the server (along with some other doc. types, e.g. feedback). How can I allow that?

These topics - the difference between read and write security, data routing - are all well covered in the “Access Control” sections of our documentation. Have you already reviewed these?

https://docs.couchbase.com/sync-gateway/2.7/read-access.html
https://docs.couchbase.com/sync-gateway/2.7/write-access.html

1 Like

Well I thought so :innocent:

… but apparently I need to revisit them… as it is not working as I thought (which is why I ask). But I understand that the import filter has nothing to do with the sync. between CB Lite and the server, which I thought by mistake…

I’ll have another look at the new versions of the docs. and see if they are clearer than at the time I worked with this last time…

Have you tried setting a pull replication filter in the ReplicatorConfiguration?
ReplicationFilter Docs
ReplicatorConfiguration Docs

1 Like

No, I haven’t… I have stopped using Pull replication after discovering the issues with pull-only type repliccation.

Wouldn’t that be on the “client side”? Some of these documents are big (the “Image” documents) so I’ld rather stop them server side but perhaps the sync. is “clever” enough to not let the entire docs. travel down to the device prior to cancelling them?

But thanks for the links. I’ll have a look at them :+1:

Thanks for the suggestions.

I ended up making an exception to adding the docs to the user’s channel if type is “Image”. Then did a rebuild on Sync.Gateway (which took a looong time).

That seems to work :+1:

1 Like