Cbbackupmgr restore needs longer timeout

Couchbase Server
1

We are getting timeouts from cbbackupmgr restore due to 180 second timeout. How do we increase that?

2

Hi @hockeynut,

This might be possible, cbbackupmgr does have quite a few timeouts/configurable options, however, it’s difficult to know exactly which one you’re hitting.

Please could you provide the following information:

  1. What version of cbbackupmgr (or couchbase-server) are you using?
  2. What do the error message/logs say ($ARCHIVE_PATH/logs/backup-0.log)?

Thanks,
James

1 Like
3

Thanks for the answer James.

cbbackupmgr version 6.5.0-4960

Looking to see if that file is around…

4

Hi @hockeynut,

The 6.5.0-4960 release of cbbackupmgr does not have as many tunable options as some of the later releases (with regards to timeouts).

I’ve had a look through the 6.5.0-4960 release and only see one timeout which is 180 seconds; this is the timeout when we’re requesting DCP streams during a backup (which is not configurable in 6.5.0-4960).

Looking to see if that file is around…

That file should be located in the archive directory that you’re trying to restore (the path that you supply to the -a/--archive flag). If you’re unable to provide that file, could you provide the full error message; the one which indicated that you’d hit a 180 second timeout?

Thanks,
James

5

Error restoring cluster: Timeout occurred (180 sec) on operation
Restore bucket “union” failed
Mutations restored: 3615590, Mutations failed to restore: 0
Deletions restored: 10484321, Deletions failed to restore: 1
Skipped due to purge number or conflict resolution: Muatations: 41 Deletions: 596

6

Hi @hockeynut,

Thanks for the update, that’s perfect; I see the issue now. Unfortunately, this timeout isn’t configurable in 6.5.0-4960.

This timeout can be configured from 6.6.0 onwards by using the CB_MEMCACHED_OPERATION_TIMEOUT environment variable with a “duration” style value i.e. 1s, 5s, 60s, 1m, 24h. For example, env CB_MEMCACHED_OPERATION_TIMEOUT=5m cbbackupmgr restore ... will set the timeout to five minutes.

You can assert that this timeout has been changed, be looking for the following statement in the cbbackpumgr logs (as previously discussed, these are created in the archive e.g. $ARCHIVE/logs/backup-0.log).

2022-01-28T15:49:41.260+00:00 (Pool) Set operation timeout to: 5m0s

As per the compatibility matrix, you should be able to restore your 6.5.0 backup using 6.6.x or 7.0.0 (both of these versions support reading the 6.5.x backup format, and can restore to a 6.5.x cluster).

Thanks,
James