Puzzled about many _sync* documents

JulianBuss · September 29, 2022, 4:13pm

Using this query:

select count(1) from ticketingsystem WHERE ((meta().`id`) like "\\_sync:%")

I’m counting the amount of Sync Gateway “Meta” documents in a bucket. The “ticketingsystem” bucket in the development environment counts about 2.065 _sync* documents with 398.456 documents in the bucket altogether.

In the related bucket in our production environment I’m getting quite different results, ranging from ~1.200.000 _sync documents to ~250.000 _sync documents with a range of ~4.000.000 to ~2.000.000 documents altogether.

My questions:

Are such numbers to be expected?
I suspect the _sync documents have something to do with the amount of mutations in the bucket, is that correct?
Do the _sync documents depend on the import function? In another bucket we have ~500.000 documents from which we only want to replicate a very small subset (below 1.000 documents) to Couchbase Lite. In that bucket we observed a huge amount of _sync documents (>1.000.000) although the import function defines that only <1.000 documents should be processed by Sync Gateway. Is that to be expected?

Thanks!

adamf · September 29, 2022, 8:31pm

There are several types of_sync: prefixed documents used by Sync Gateway, primarily to support security and replication of data in the associated bucket. Typically the highest volume of documents are those with the _sync:rev prefix - these are temporary backups of revisions used to support replication. These are set to expire (the TTL depends on whether delta sync is enabled), but can result in a large volume of documents for highly mutated documents.

What _sync document types are you seeing in the case where you’ve got 1M _sync docs in a bucket with ~1K documents being proccessed by SG?

JulianBuss · September 30, 2022, 5:27am

Hi Adam,

the bucket “synsystem” has about 21k documents for which the SG import function returns true (not ~1k as I wrote previously, sorry). I deleted the db from SG and created it again, counting the _sync documents in various states. Results as follows:

just after db create, without any config:

  {
    "$1": 1,
    "$2": "_sync:dbconf"
  },
  {
    "$1": 191,
    "$2": "_sync:dcp_ck"
  },
  {
    "$1": 1,
    "$2": "_sync:cfgind"
  },
  {
    "$1": 2,
    "$2": "_sync:cfgnod"
  },
  {
    "$1": 1,
    "$2": "_sync:cfgver"
  },
  {
    "$1": 1619167,
    "$2": "_sync:unused"
  },
  {
    "$1": 1,
    "$2": "_sync:cfgsgr"
  },
  {
    "$1": 1,
    "$2": "_sync:cfgpla"
  },
  {
    "$1": 1,
    "$2": "_sync:heartb"
  },
  {
    "$1": 1,
    "$2": "_sync:seq"
  },
  {
    "$1": 1,
    "$2": "_sync:syncda"
  }

5 minutes after db create, still without any config:

{
      "$1": 1,
      "$2": "_sync:cfgver"
    },
    {
      "$1": 1,
      "$2": "_sync:seq"
    },
    {
      "$1": 1,
      "$2": "_sync:cfgind"
    },
    {
      "$1": 4633580,
      "$2": "_sync:unused"
    },
    {
      "$1": 310,
      "$2": "_sync:dcp_ck"
    },
    {
      "$1": 1,
      "$2": "_sync:heartb"
    },
    {
      "$1": 1,
      "$2": "_sync:cfgpla"
    },
    {
      "$1": 1,
      "$2": "_sync:dbconf"
    },
    {
      "$1": 2,
      "$2": "_sync:cfgnod"
    },
    {
      "$1": 1,
      "$2": "_sync:cfgsgr"
    },
    {
      "$1": 1,
      "$2": "_sync:syncda"
    }

after setting the following config:

{
    "name":"synsystem",
    "enable_shared_bucket_access": true,
    "import_docs": true,
    "delta_sync": {
        "enabled": true,
        "rev_max_age_seconds": 28800
    },
    "guest": {
        "disabled": true
    },
    "allow_conflicts": false,
    "revs_limit": 20,
    "compact_interval_days": 1
}

 {
      "$1": 1,
      "$2": "_sync:cfgpla"
    },
    {
      "$1": 1,
      "$2": "_sync:cfgind"
    },
    {
      "$1": 1,
      "$2": "_sync:seq"
    },
    {
      "$1": 27,
      "$2": "_sync:rev:sy"
    },
    {
      "$1": 1,
      "$2": "_sync:dbconf"
    },
    {
      "$1": 1,
      "$2": "_sync:cfgver"
    },
    {
      "$1": 1,
      "$2": "_sync:rev:e2"
    },
    {
      "$1": 1,
      "$2": "_sync:cfgsgr"
    },
    {
      "$1": 1024,
      "$2": "_sync:dcp_ck"
    },
    {
      "$1": 937195,
      "$2": "_sync:unused"
    },
    {
      "$1": 135,
      "$2": "_sync:rev:sc"
    },
    {
      "$1": 1,
      "$2": "_sync:rev:au"
    },
    {
      "$1": 1,
      "$2": "_sync:heartb"
    },
    {
      "$1": 1,
      "$2": "_sync:rev:1e"
    },
    {
      "$1": 1,
      "$2": "_sync:syncda"
    },
    {
      "$1": 2,
      "$2": "_sync:cfgnod"
    }

after setting the import function:

function(doc) {
    if (!doc.form && doc._deleted === true) {
        return true;
    }
    if (!doc.$_meta) return false;
    if (doc.$_meta.type === "notification") {
        return true;
    }

    return false
}

  {
      "$1": 1,
      "$2": "_sync:rev:1a"
    },
    {
      "$1": 1,
      "$2": "_sync:cfgind"
    },
    {
      "$1": 275,
      "$2": "_sync:rev:sc"
    },
    {
      "$1": 4,
      "$2": "_sync:rev:51"
    },
    {
      "$1": 1,
      "$2": "_sync:cfgver"
    },
    {
      "$1": 1,
      "$2": "_sync:cfgsgr"
    },
    {
      "$1": 1,
      "$2": "_sync:dbconf"
    },
    {
      "$1": 64,
      "$2": "_sync:rev:sy"
    },
    {
      "$1": 1,
      "$2": "_sync:seq"
    },
    {
      "$1": 1,
      "$2": "_sync:rev:08"
    },
    {
      "$1": 1,
      "$2": "_sync:rev:5f"
    },
    {
      "$1": 1,
      "$2": "_sync:rev:8c"
    },
    {
      "$1": 3,
      "$2": "_sync:rev:1e"
    },
    {
      "$1": 1,
      "$2": "_sync:rev:95"
    },
    {
      "$1": 1,
      "$2": "_sync:rev:e2"
    },
    {
      "$1": 1024,
      "$2": "_sync:dcp_ck"
    },
    {
      "$1": 1,
      "$2": "_sync:rev:3a"
    },
    {
      "$1": 1,
      "$2": "_sync:cfgpla"
    },
    {
      "$1": 1,
      "$2": "_sync:syncda"
    },
    {
      "$1": 3,
      "$2": "_sync:rev:au"
    },
    {
      "$1": 1,
      "$2": "_sync:rev:49"
    },
    {
      "$1": 2,
      "$2": "_sync:cfgnod"
    },
    {
      "$1": 1,
      "$2": "_sync:heartb"
    },
    {
      "$1": 1,
      "$2": "_sync:rev:b4"
    },
    {
      "$1": 470928,
      "$2": "_sync:unused"
    }

focused on rev and unused types, after 2 minutes:

unused: 45324
rev: 397

focused on rev and unused types, after 4 minutes:

unused: 60
rev: 484
total _sync documents: 1578
total docs imported by SG: 21170

It’s very early in the morning and there is few activity in our applications. Yesterday in the late afternoon I had millions of _sync documents in that bucket before I deleted the db from SG, and that number seemed to be persistent. I will continue monitoring the numbers during the day.

One interesting question is: what are the _sync:unused documents?

JulianBuss · September 30, 2022, 1:02pm

I monitored the situation during the day and so far the count of _sync docs in all buckets is around some thousands, that feels just fine.

Yesterday late afternoon, after I found the millions of _sync documents, I stopped the SG gateways and cleared all sync data in order to start from scratch:

deleted all documents with meta().id like _sync% from all buckets
removed all xattrs_sync data from all documents
started SG, create dbs, set config, import function and sync function

As said above, so far it looks fine beside the temporary 1.619.167 _sync:unused as described in my post above.