Deleted documents reappearing

[ferreirajohanh]

I’m facing issues where deleted docs are reappearing after a time in my bucket. I suspect it has something to do with my sync gateway but I’m unsure where to even start troubleshooting here.

Current Versions

1. Couchbase Server Community Edition 7.0.2 build 6703
2. couchbase-lite-android:2.8.6
3. sync-gateway:2.8.3-community

I have three types of docs:

1. config - Generated and edited on server but only sync down to devices
2. device_generated - Generated on devices, only sync up to server
3. server_generated - Generated and edited on server, never sync down to devices

Here is a illustration of the flow:

Sync gateway config

{
    "log": [
        "*"
    ],
    "interface": "0.0.0.0:5500",
    "adminInterface": "0.0.0.0:5501",
    "databases": {
        "bucket": {
            "server": "node1.cluster",
            "bucket": "bucket_name",
            "username": "username",
            "password": "password",
            "enable_shared_bucket_access": true,
            "import_docs": "continuous",
            "use_views": true,
            "users": {
                "username": {
                    "password": "password",
                    "admin_channels": [
                        "*"
                    ],
                    "disabled": false
                }
            },
            "sync":
            `
            function sync(doc, oldDoc) {
                /* sanity check */
                // check if document was removed from server or via SDK
                // In this case, just return
                if (isRemoved()) {
                        return;
                }
                //Only sync documents that are created on the server
                if (doc.type == "device_generated") {
                        channel("server");
                } else {
                        channel("devices");
                }
                // This is when document is removed via SDK or directly on server
                function isRemoved() {
                        return (isDelete() && oldDoc == null);
                }
        
                function isDelete() {
                        return (doc._deleted == true);
                }
        }
            `
        }
    }
}

On Couchbase Lite I’ve setup my Android app to only sync on the “devices” channel.

Here is the problem I’m facing:

1. device_generated and server_generated docs are being deleted by users on the server
2. The docs are no longer present on the server
3. A few days later they magically reappear

Can anyone suggest a reason why this might be happening? Is there an issue in my sync gateway config file?

Thanks in advance!

[borrrden]

Appreciate the details in this post, but one detail that is missing is how exactly the delete operation is being performed, and where they are reappearing (on the server or on the device?).

[ferreirajohanh]

The documents that reappeared were deleted by running a delete query directly on the Couchbase Web Console and also by using the Java SDK.

[borrrden]

I noticed that you did not specify a channel for the deletion. This is likely to cause unexpected behavior since the deletion will not be picked up by any couchbase lite clients. After that, depending on how you have your conflict resolution set up, it might be the case that the lite client is editing a document and pushing it back up again. You should put the channel call on deletions as well so that the deletions themselves are replicated.

[ferreirajohanh]

@ferreirajohanh

{
    "log": [
        "*"
    ],
    "interface": "0.0.0.0:5500",
    "adminInterface": "0.0.0.0:5501",
    "databases": {
        "bucket": {
            "server": "node1.cluster",
            "bucket": "bucket_name",
            "username": "username",
            "password": "password",
            "enable_shared_bucket_access": true,
            "import_docs": "continuous",
            "use_views": true,
            "users": {
                "username": {
                    "password": "password",
                    "admin_channels": [
                        "*"
                    ],
                    "disabled": false
                }
            },
            "sync":
            `
            function sync(doc, oldDoc) {
                /* sanity check */
                // check if document was removed from server or via SDK
                // In this case, just return
                if (isRemoved()) {
                        return;
                }
                //Only sync documents that are created on the server
                if (doc.type == "device_generated") {
                        channel("server");
                } else {
                        channel("devices");
                }
                // This is when document is removed via SDK or directly on server
                function isRemoved() {
                        return (isDelete() && oldDoc == null);
                }
        
                function isDelete() {
                        return (doc._deleted == true);
                }
        }
            `
        }
    }
}

OK great thanks!

Just to confirm is this what you propose?

I’ve made an edit inside the if (isRemoved()) statement.

function sync(doc, oldDoc) {
    /* sanity check */
    // check if document was removed from server or via SDK
    // In this case, just return
    if (isRemoved()) {
        // --- NEW EDIT ---
        channel("server");
        // -----------------
        return;
    }
    //Only sync documents that are created on the server
    if (doc.type == "device_generated") {
        channel("server");
    } else {
        channel("devices");
    }
    // This is when document is removed via SDK or directly on server
    function isRemoved() {
        return (isDelete() && oldDoc == null);
    }

    function isDelete() {
        return (doc._deleted == true);
    }
}

Also, is there a standard sync function that I can reference somewhere? Or other examples?

Thanks for the help!

[borrrden]

There is no “standard” sync function because what is standard? It’s 100% domain specific. The default is documented though.

What you proposed seems in line with what I suggested, if that is what is going out to the devices (it’s a little hard to follow with “device generated” documents being put in the “server” channel and I guess non device generated being put in “devices”.

Let me tag @bbrks to see if he has anything else to share, since sync function is not my main area.

[ferreirajohanh]

Great, thanks!

@bbrks could you maybe confirm if my edit makes sense?

[ferreirajohanh]

When you delete a docment manually on the Enterprise Edition 5.0.1 build 5003 bucket，but this docment won't be synced to android client!

If you remove or delete a document via SDK or directly couchbase server, the oldDoc will be null. You don’t have to do anything specific in terms of putting deleted documents into channel - the SGW will take care of it for you. The document is already in a channel and would now be updated with the _deleted flag as true. This test would be the check to see if document is removed. function isRemoved() { return( isDelete() && oldDoc == null); } So this is all you would have to do in your…

I’ve been doing some more research and it doesn’t seem that my current sync function is incorrect. I’m using the same method as suggested by @priya.rajagopal

[ferreirajohanh]

This occurred again today and I was able to get the logs from the sync gateway:

2023-08-07T12:51:18.728Z [WRN] Null doc body/rawBody docId/ from  -- db.(*Document).BodyBytes() at document.go:306
2023-08-07T12:51:18.728Z [WRN] Null doc body/rawBody docId/ from  -- db.(*Document).Body() at document.go:245
2023-08-07T12:51:18.731Z [WRN] Null doc body/rawBody docId/ from  -- db.(*Document).Body() at document.go:245
2023-08-07T12:51:18.732Z [INF] CRUD: 	Doc "docId" / "2-961e0ef91b16bacaa63f6817100d6237" in channels "{server}"

I can see that it happens at this line in the code:

couchbase/sync_gateway/blob/ab6e24f365e8116ea6224066c7f338fab3af014a/db/document.go#L305

But I’m unsure where to follow the code from there…

The same error message is also referenced here:

Indexes suddenly getting hundred of thousands of mutations and increasing

I am using Couchbase Server Community Edition 6.6.0 build 7909 on a Ubuntu server and a few times in the past month, the indexes on my server suddenly getting thousands of mutations and this increases into the millions. The indexes are showing as ‘ready’ and are queryable. This does eventually go back to normal but when it is processing all the mutations any new documents aren’t returned when the index is queried. The sync gateway requests timeout when there are loads of mutations and then o…

But I’m not sure if this is related. Can anyone maybe advise how I can debug further?

[adamf]

At first glance I don’t see how that particular warning would be related to the described behaviour.

I don’t think I’ve seen an answer above to @borrrden 's original question of “where are they reappearing (on the server or on the device)” - which one is it?

[ferreirajohanh]

The docs are reappearing on the server side.

[adamf]

There are a few possibilities here, but the most likely sounds like the one Jim mentioned earlier - that clients are modifying the document, and that update is being replicated to the server and recreating the documents.
I think there are a few areas you could investigate based on the answer to a few questions:

• Do you intend to allow clients to modify the documents that are reappearing? If no, what protections are you using to avoid that? Your sync function doesn’t have any restrictions on who can write documents.

• If you’re using client application logic to prevent updates once a document has been deleted, is this a case of the delete not being replicated to the client in the first place? From the Sync Gateway debug logs you would be able to see changes to the document, and work out which client performed the update. That might give you a hint about where the mutations are coming from

[ferreirajohanh]

Hi @adamf

Sorry for the late reply!

No, the intention is not that the clients are able to modify the docs. Currently I’m enforcing this by placing the correspondings docs in a different channel and on the client side not subscribing to that specific channel.

So what I’m struggling to understand is how the doc is ending up on the clients and then syncing back up from the clients through the sync gateway to the server.

Here is the details of the doc after being deleted when requesting it from the sync gateway api ({urll}/db/_raw/doc_id):

{
    "_deleted": true,
    "_sync": {
        "rev": "4-e562147b3f6158c31f54f7dcb8e1d894",
        "flags": 17,
        "sequence": 28937228,
        "recent_sequences": [
            27810644,
            27810683,
            27811331,
            28937185,
            28937228
        ],
        "history": {
            "revs": [
                "3-5bc29163dd8ea0c143c21396fd6dbd50",
                "4-e562147b3f6158c31f54f7dcb8e1d894",
                "2-1a06a8e1fe590794f67042e792fa36af",
                "1-3f4fad7f56be04018b9ddb75f2f1c3a2",
                "2-8210b787adbc1680e1a6510e3b1e5f87",
                "1-11f294a2c3689d7664a66f1781eac87c"
            ],
            "parents": [
                2,
                0,
                5,
                -1,
                3,
                -1
            ],
            "deleted": [
                1,
                4
            ],
            "bodymap": {
                "4": "{\"_deleted\":true}"
            },
            "channels": [
                [
                    "server"
                ],
                null,
                [
                    "server"
                ],
                [
                    "server"
                ],
                null,
                null
            ]
        },
        "channels": {
            "server": {
                "seq": 28937228,
                "rev": "4-e562147b3f6158c31f54f7dcb8e1d894",
                "del": true
            }
        },
        "cas": "0x0000257bda817f17",
        "value_crc32c": "0x00",
        "tombstoned_at": 1693214760,
        "time_saved": "2023-08-28T09:26:00.28276148Z"
    }
}

It seems there is something weird happening with my sync function when looking at the history part of the various channels it has been assigned to?

[ferreirajohanh]

I’m starting to thing there is something wrong with how I’ve set up the user permissions.

There is currently only a single user set up on the sync gateway and all of the clients use the same user credentials to connect to the sync gateway.

"username": {
    "password": "password",
    "admin_channels": [
        "*"
    ],
    "disabled": false
}

Here is my replicator config on my android clients:

ReplicatorConfiguration oneShotConfig = new ReplicatorConfiguration(DatabaseUtil.getDatabase(), endpoint)
                    .setReplicatorType(ReplicatorConfiguration.ReplicatorType.PUSH_AND_PULL)
                    .setContinuous(false)
                    .setChannels(SyncUtil.syncChannels)
                    .setPullFilter(replicationFilter);

I’ve set to only replicate from specific channels and also a replication filter which should not pull down all of the docs I’m not expecting on the clients.

But I’m thinking that because of the wild card channel in the user setup some of the docs are still making their way down to the devices.

Does this make sense at all?

[borrrden]

It sounds like you might be under the false impression that channels are applied to push. They only have to do with pull. Couchbase Lite has no concept of channels, they are purely a Sync Gateway concept. Assignment to channels happens in the sync function because Couchbase Lite has no idea about what documents are in what channels. But if you already knew that then nevermind.

Are you suggesting that because the pull logic is not working, documents are being replicated to the device when they shouldn’t be and then pushed back up to resurrect them? If you have channels set on a pull replicator then only documents in those channels will be pulled. Furthermore there is no user set on your replication (unless you omitted that part) so admin_channels would be even less relevant in that situation.

EDIT Also just a small reminder that as of today 2.8.x is end of life for both lite and sync gateway.

[ferreirajohanh]

Good day @borrrden and thanks for the reply!

Sorry I forgot to add this:

oneShotConfig.setAuthenticator(new BasicAuthenticator(SYNC_GATEWAY_USERNAME, SYNC_GATEWAY_PASSWORD));

I am definitely authenticating with the specified username and password.

But yes that is my thinking at the moment, I’m somehow pulling all of the docs down to the device, regardless of channel and then those docs are being pushed up again.

[borrrden]

There should be some evidence of that in either the Sync Gateway or Couchbase Lite logs. Without that evidence I would say it’s unlikely that all channels are being pulled regardless of setting a channel on the replicator.

[ferreirajohanh]

@borrrden thanks for the reply!

I’ll dive into the logs and check if I see anything there!