SG does not push down attachment to client (403 Attachment's doc not being synced)

Mobile Sync Gateway
1

Setup:
CB Server 6 CE
SG 2.7
CBL Android 2.7

I have been running into issues with the old attachments created with CBL 1.x which I detailed here:

Now I observe that SG does not push down attachments, too. Here’s the SG log:

[INF] SyncMsg: c:[1f6b1d87] #4: Type:getAttachment Digest:sha1-GL8jR+MlmM4+wqqKe3372FkLL70= 
[INF] SyncMsg: c:[1f6b1d87] #4: Type:getAttachment   --> 403 Attachment's doc not being synced Time:13.39µs
...
[INF] SyncMsg: c:[1f6b1d87] #3: Type:getAttachment Digest:sha1-8IE/kBltWJR4lUs7ynBdj0G9+Pw= 
[INF] SyncMsg: c:[1f6b1d87] #3: Type:getAttachment   --> 403 Attachment's doc not being synced Time:6.867µs

I also saved the SG debug log but it’s over 5k lines.

On the Android client I see this:

E/CouchbaseLite/REPLICATOR: {N8litecore4repl12IncomingBlobE#3}==> N8litecore4repl12IncomingBlobE for doc 'Qh9zvlESWUeJKbRw3LukBfCu7LV2::userProfile'._attachments["Qh9zvlESWUeJKbRw3LukBfCu7LV2::large"] [sha1-8IE/kBltWJR4lUs7ynBdj0G9+Pw=] @0x6f4961b0c8
E/CouchbaseLite/REPLICATOR: {N8litecore4repl12IncomingBlobE#3} Got error response: HTTP 403 'Attachment's doc not being synced'
E/CouchbaseLite/REPLICATOR: {N8litecore4repl12IncomingBlobE#3} Got error response: HTTP 403 'Attachment's doc not being synced'

To solve the issue via code I tried:

  • resetting the checkpoint and one-shot pull it
  • adding a field to the document via web request and Java SDK. The update should make the SG push it down to the client
  • creating the document on the client … it wasn’t updated on the server. Likely because the revision on the client starts with 1… while the one on the server has likely a higher revision number and thus it’s rejected. Or because of another reason it is rejected. That’s a guess on my part

In testing it helped to delete the attachment part of the document in the document viewer in the browser of Couchbase Server. It’s not confirmed by the affected user in production yet. As of now this is a manual effort. I might have to do this with all documents of all users who run into this issue. Additionally their attachements are gone.

2

Then the SG pushed down the document to the client (instantly).

3

This is the same bug as you’ve found when pulling documents (CBG-741). We’re working on a fix for this, but in the meantime, I’ve advised that documents affected by this bug can be fixed by writing a new revision through Sync Gateway.

The following curl command retrieves the active revision of a given document, and writes it back to Sync Gateway, creating a new revision with the same data:

curl -s http://sghost:4985/DB_NAME/DOC_ID | curl -X PUT http://sghost:4985/DB_NAME/DOC_ID -H 'Content-Type: application/json' -d @-

Example usage (assuming doc1 is at rev 13-...):

$ curl -s http://localhost:4985/db1/doc1 | curl -X PUT http://localhost:4985/db1/doc1 -H 'Content-Type: application/json' -d @-
{"id":"doc1","ok":true,"rev":"14-60e3d80d226eda5ed1aff7bec46773c5"}

This will force the old 1.x-2.1 style attachment metadata to be upgraded, and should allow clients to pull revisions after this.

4

I’m importing a backup on my dev server cluster just now and will try this and follow up in a few hours. Thanks!

5

This command does fix the issue!

Comparing two documents, one that was fixed with the command and one with blobs show differences though:
The fixed document has no attachment/blob field in the json. It has this in the meta data:

"attachments": {
        "Qh9zvlESWUeJKbRw3LukBfCu7LV2::large": {
          "content_type": "image/jpeg",
          "digest": "sha1-8IE/kBltWJR4lUs7ynBdj0G9+Pw=",
          "length": 174354,
          "revpos": 5,
          "stub": true
        },
        "Qh9zvlESWUeJKbRw3LukBfCu7LV2::small": {
          "content_type": "image/jpeg",
          "digest": "sha1-GL8jR+MlmM4+wqqKe3372FkLL70=",
          "length": 49768,
          "revpos": 5,
          "stub": true
        }
      }

The document with a blob has this in the json body:

"1584460640751": {
    "@type": "blob",
    "content_type": "image/jpeg",
    "digest": "sha1-lNSKqQbyROdnxBRdGkh9MvsxA7A=",
    "length": 183541
  }

And this in the meta data:

 "attachments": {
        "blob_/1584460640751": {
          "content_type": "image/jpeg",
          "digest": "sha1-lNSKqQbyROdnxBRdGkh9MvsxA7A=",
          "length": 183541,
          "revpos": 2,
          "stub": true
        }
      }

Q1) With these observed differences is it still safe to use this command on all “broken” documents?

Q2) I was planning to update all old documents with attachments: delete the attachments and add them as blobs on the client. Is it planned to provide a SG command to do this for the whole database?

6

I compared it with another document which also has only old attachments. It looks the same. So it’s safe to use the command. I cannot remember if before SG 2.x the attachments were also (or only) in the json body.

7

In case someone else runs into the same issue, here’s how I solved it for all documents:

  1. Create index
CREATE INDEX `idx_attachment` ON `my_bucket`(`_attachments`)
  1. Run query
SELECT RAW(meta().id)
FROM my_bucket
WHERE _attachments IS NOT MISSING;
  1. Create script (left as an exercise for the reader) and print results to file like so
curl -s http://localhost:4985/my_bucket/docA | curl -X PUT http://localhost:4985/my_bucket/docA -H 'Content-Type: application/json' -d @-;
sleep 2;
curl -s http://localhost:4985/my_bucket/docB | curl -X PUT http://localhost:4985/my_bucket/docB -H 'Content-Type: application/json' -d @-;
sleep 2;
...
  1. Test script on dev server and check if issues are solved, backup bucket on production server, run script on production server