So this morning I decided to create the backup again, using a different Docker Couchbase container. Result of the backup seemed about the same, almost the same size, about 1.1GB zipped. Difference is the restore worked and actually restored the data. So the issue was something about the first backup itself.
Comparing the logs of the successful one with the previous which while looking successful doesn’t actually restore properly, the command was identical. The newer one didn’t have those WARN messages though that I saw in the older one. I’m not sure why those would matter, but they are what stand out as the difference. Here’s the block of logs with the warnings.
2023-05-09T16:24:07.055+00:00 WARN: (REST) (Attempt 1) (GET) Request to endpoint ‘/api/v1/backup’ failed due to error: failed to prepare request: failed to get host for service ‘Analytics’: failed to get host for service ‘Analytics’: Analytics Service is not available – logging.(*ToolsCommonLogger).Log() at tools_common.go:28
2023-05-09T16:24:07.055+00:00 (Couchbase) (Source) Will not be transferring Analytics Cluster Metadata because Analytics Service is not available
2023-05-09T16:24:07.055+00:00 (Plan) (Analytics) Successfully transferred Analytics metadata | {“number”:2,“duration”:“325.2µs”}
2023-05-09T16:24:07.057+00:00 (Plan) (Eventing) Transferring Eventing metadata
2023-05-09T16:24:07.057+00:00 WARN: (REST) (Attempt 1) (GET) Request to endpoint ‘/api/v1/backup’ failed due to error: failed to prepare request: failed to get host for service ‘Eventing’: failed to get host for service ‘Eventing’: Eventing Service is not available – logging.(*ToolsCommonLogger).Log() at tools_common.go:28
2023-05-09T16:24:07.057+00:00 (Couchbase) (Source) Will not be transferring Eventing Metadata because Eventing Service is not available
2023-05-09T16:24:07.057+00:00 (Plan) (Eventing) Successfully transferred Eventing metadata | {“number”:3,“duration”:“52.5µs”}
2023-05-09T16:24:07.061+00:00 (Plan) (Search) Transferring full text index aliases
2023-05-09T16:24:07.061+00:00 WARN: (REST) (Attempt 1) (GET) Request to endpoint ‘/api/v1/backup’ failed due to error: failed to prepare request: failed to get host for service ‘Search’: failed to get host for service ‘Search’: Search Service is not available – logging.(*ToolsCommonLogger).Log() at tools_common.go:28
The backup which fails to restore does have this line that verifies the data for the bucket I’m concerned most with (halix2) did back up properly.
2023-05-09T16:24:46.503+00:00 (Plan) (Data) Successfully transferred key value data for bucket ‘halix2’ | {“number”:23,“duration”:“27.8491111s”,“stats”:{“estimated_total_items”:2903457,“total_items”:2551614,“total_vbuckets”:1024,“vbuckets_complete”:1024,“bytes_received”:2335192821,“snapshot_markers_received”:1024,“failover_logs_received”:1024,“mutations_received”:2511047,“deletions_received”:40567,“started_at”:1683649458669907000,“finished_at”:1683649486503197200,“complete”:true}}
So I’m not sure why that data still didn’t restore for that backup. I’d love to know what happened because it is of course essential to be able to trust our backups when I move these scripts over into production. We cannot afford to have random backups looking ok, but DOA if we need to restore it at some point. I cannot upload the logs unfortunately as a new user, but would those WARN messages above explain why the backup won’t restore? What else from the log should I look at that might explain why? Thanks.