Sync Gateway doesn't return from request to go _offline

Couchbase Server
1

Couchbase Server Community Edition 7.6.4 build 5146
Couchbase Sync Gateway/2.8.2(1;4df7a2d) CE

Trying to take database offline for resync, but it doesn’t return, seems to be stuck:

curl -kSsi -H “content-type: application/json” -H “accept: application/json” -X POST “https://localhost:4985/licensees/_offline”

Checking from another shell:

safedoorpm@sdpmwdeusd0:~$ curl -kSs “https://localhost:4985/licensees/” | jq ‘.’
{
  "db_name": "licensees",
  "update_seq": 372673,
  "committed_update_seq": 372673,
  "instance_start_time": 1759899276436420,
  "compact_running": false,
  "purge_seq": 0,
  "disk_format_version": 0,
  "state": "Stopping",
  "server_uuid": "c5d5d896cc3219dffa7c762b2c0d7906"
}

I’ve let it sit for over an hour, no response.

I previsously managed to do _offline followed by _resync and _online, and have even restarted Sync Gateway, Couchbase, and the entire server (Ubuntu) and it still does this.

Restarting SG doesn’t report any problems that I can see:

safedoorpm@sdpmwdeusd0:~$ sudo systemctl restart sync_gateway
safedoorpm@sdpmwdeusd0:~$ sudo systemctl status sync_gateway
● sync_gateway.service - Couchbase Sync Gateway server
     Loaded: loaded (/lib/systemd/system/sync_gateway.service; enabled; vendor preset: enabled)
     Active: active (running) since Fri 2025-10-10 19:40:56 UTC; 7s ago
    Process: 483212 ExecStartPre=/bin/mkdir -p /home/sync_gateway/logs (code=exited, status=0/SUCCESS)
    Process: 483220 ExecStartPre=/bin/chown -R sync_gateway:sync_gateway /home/sync_gateway/logs (code=exited, status=0/SUCCESS)
    Process: 483225 ExecStartPre=/bin/mkdir -p /home/sync_gateway/data (code=exited, status=0/SUCCESS)
    Process: 483226 ExecStartPre=/bin/chown -R sync_gateway:sync_gateway /home/sync_gateway/data (code=exited, status=0/SUCCESS)
   Main PID: 483227 (sync_gateway)
      Tasks: 15 (limit: 19081)
     Memory: 49.9M
     CGroup: /system.slice/sync_gateway.service
             └─483227 /opt/couchbase-sync-gateway/bin/sync_gateway --defaultLogFilePath /home/sync_gateway/logs /home/sync_gateway/sync_gateway.json

Oct 10 19:41:01 sdpmwdeusd0 bash[483227]: 2025-10-10T19:41:01.105Z [DBG] Changes+: c:[560b4392] MultiChangesFeed sending {Seq:372672, ID:_user/kent.mcphee@safedoorpm.com, Changes:[]}   (t>
Oct 10 19:41:01 sdpmwdeusd0 bash[483227]: 2025-10-10T19:41:01.105Z [DBG] Changes+: c:[560b4392] MultiChangesFeed waiting...   (to kent.mcphee@safedoorpm.com)
Oct 10 19:41:01 sdpmwdeusd0 bash[483227]: 2025-10-10T19:41:01.105Z [DBG] Changes+: No new changes to send to change listener.  Waiting for "bucket-licensees"'s count to pass 0
Oct 10 19:41:01 sdpmwdeusd0 bash[483227]: 2025-10-10T19:41:01.105Z [DBG] Sync+: c:[560b4392]     Sending 1 changes
Oct 10 19:41:01 sdpmwdeusd0 bash[483227]: 2025-10-10T19:41:01.105Z [DBG] Sync+: c:[560b4392]     Sending 0 changes
Oct 10 19:41:01 sdpmwdeusd0 bash[483227]: 2025-10-10T19:41:01.105Z [DBG] WS+: c:#002 Queued MSG#1~
Oct 10 19:41:01 sdpmwdeusd0 bash[483227]: 2025-10-10T19:41:01.105Z [DBG] WSFrame+: c:#002 Push MSG#1~
Oct 10 19:41:01 sdpmwdeusd0 bash[483227]: 2025-10-10T19:41:01.105Z [INF] Sync: c:[560b4392] Sent all changes to client
Oct 10 19:41:01 sdpmwdeusd0 bash[483227]: 2025-10-10T19:41:01.105Z [DBG] WSFrame+: c:#002 Sending frame: MSG#1~ (flags=  101000, size=   51)
Oct 10 19:41:03 sdpmwdeusd0 bash[483227]: 2025-10-10T19:41:03.657Z [INF] Replicate: Starting sg-replicate replications...

If I try _offline again, it still hangs

I have another server with basically identical configuration and it doesn’t have this issue. Basically the same .json configuration file, except with different credentials and a slightly different sync function as one is our dev system, one is QA.

Could it be something specific about the data? Or maybe some subtle configuration difference? Where should I look?

2

I’ve not seen this issue, though this code has been extensively reworked in Sync Gateway 3.0+. In Sync Gateway 2.8.2 or newer, can you use the workaround of updating the database configuration to have ”offline”: true?

This isn’t an API we are likely to fix if there is an issue, since calling /db/_offline is only node local. In Sync Gateway 2.8, this doesn’t matter, but using persistent configuration in Sync Gateway 3.0+ means that updating the database configuration with ”offline”: true is the only way to have this act across all Sync Gateway nodes.

3

I only needed this for the one node so the configuration setting worked for me, thanks.

I wish I could figure out why this node keeps doing that. I can’t see anything in the SG logs

4

My guess is that it is Jira - calling /{db}/_offline will block until all HTTP connections have been terminated.

Until Jira was addressed, this also included all Couchbase Lite continuous replications.

However, the behavior of blocking until all HTTP requests are finished might not be desired anyway - it is blocked behind a read-write mutex, and is not a draining operation. If frequent HTTP requests were occurring, it would be possible that /{db}/_offline would not acquire the write part mutex. All other requests will acquire the read mutex.

Changing the db config to offline will have the the effect of immediately terminating the connections and turning the database into offline mode. As I said before, we are likely to leave the behavior of /{db}/_offline as is, with all the flaws, since using /{db}/_config with {“offline": true}is the only thing that works with a multi node Sync Gateway cluster.

5

Ok, thank you for the clarification. The configuration file method is sufficient for my use.