Hi,
I’m trying to backup all buckets from our 3 node cluster and is failing. I was also trying to backup a single bucket one by one to se which is failing and discovered that they almost al are failing and only 3 of 20 buckets could be back-upped. The error is the following:
Blockquote
Backing up to ‘2022-09-14T15_24_55.066290901Z’
Transferring key value data for ‘statistic’ at 308.93KB/s (about 0s remaining) 1573 items / 1.14MB
[=======================================================================================================================] 100.00%
Error backing up cluster: EOF
Backed up bucket “statistic” failed
Mutations backed up: 1573, Mutations failed to backup: 0
Deletions backed up: 0, Deletions failed to backup: 0
Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
Blockquote
The contents from backup-0.log are:
2022-09-15T11:49:31.741+00:00 (Cmd) Backing up cluster 87b44da8d54726c094cba99fa2e6a27a
2022-09-15T11:49:31.743+00:00 (Stats) Starting stat gathering - stat timestamp: 1663242571
2022-09-15T11:49:31.743+00:00 (Plan) Transferring cluster configuration
2022-09-15T11:49:31.747+00:00 (Rest) GET http://127.0.0.1:8091/pools 200
2022-09-15T11:49:31.747+00:00 (Plan) Successfully transferred cluster config
2022-09-15T11:49:31.748+00:00 (Plan) Transferring Eventing metadata
2022-09-15T11:49:31.749+00:00 (Rest) GET http://127.0.0.1:8091/pools/default/nodeServices 200
2022-09-15T11:49:31.812+00:00 (Rest) GET http://x.x.x.x:8096/api/v1/export 200
2022-09-15T11:49:31.813+00:00 (Plan) Successfully transferred Eventing metadata
2022-09-15T11:49:31.813+00:00 (Plan) Transferring full text index aliases
2022-09-15T11:49:31.814+00:00 (Rest) GET http://127.0.0.1:8091/pools/default/nodeServices 200
2022-09-15T11:49:31.815+00:00 (Plan) Successfully transferred full text index aliases
2022-09-15T11:49:31.815+00:00 (Plan) Checking if remote bucket 'statistic' exists
2022-09-15T11:49:31.815+00:00 (Plan) Successfully ensured remote bucket 'statistic' exists
2022-09-15T11:49:31.815+00:00 (Plan) Transferring bucket configuration for 'statistic' to 'statistic'
2022-09-15T11:49:32.191+00:00 (Rest) GET http://127.0.0.1:8091/pools/default/buckets 200
2022-09-15T11:49:32.204+00:00 (Plan) Successfully transferred bucket configuration for 'statistic' to 'statistic'
2022-09-15T11:49:32.204+00:00 (Plan) Transferring Analytics metadata
2022-09-15T11:49:32.205+00:00 (Rest) GET http://127.0.0.1:8091/pools/default/nodeServices 200
2022-09-15T11:49:32.205+00:00 (Plan) Successfully transferred Analytics metadata
2022-09-15T11:49:32.206+00:00 (Plan) Transferring full text index definitions for 'statistic' to 'statistic'
2022-09-15T11:49:32.207+00:00 (Rest) GET http://127.0.0.1:8091/pools/default/nodeServices 200
2022-09-15T11:49:32.207+00:00 (Plan) Successfully transferred full text index definitions for 'statistic' to 'statistic'
2022-09-15T11:49:32.208+00:00 (Plan) Transferring GSI index definitions for statistic to statistic
2022-09-15T11:49:32.210+00:00 (Rest) GET http://127.0.0.1:8091/pools/default/nodeServices 200
2022-09-15T11:49:32.730+00:00 (Rest) GET http://x.x.x.x:9102/getIndexMetadata?bucket=statistic 200
2022-09-15T11:49:32.730+00:00 (Plan) Successfully transferred GSI index definitions for statistic to statistic
2022-09-15T11:49:32.731+00:00 (Plan) Transferring views definitions from 'statistic' to 'statistic'
2022-09-15T11:49:33.071+00:00 (Rest) GET http://127.0.0.1:8091/pools/default/buckets 200
2022-09-15T11:49:33.083+00:00 (Rest) GET http://127.0.0.1:8091/pools/default/nodeServices 200
2022-09-15T11:49:33.087+00:00 (Rest) GET http://x.x.x.x:8091/pools/default/buckets/statistic/ddocs 200
2022-09-15T11:49:33.087+00:00 (Plan) Successfully transferred views definitions from 'statistic' to 'statistic'
2022-09-15T11:49:33.087+00:00 (Plan) Checking for data movement restrictions between statistic and statistic
2022-09-15T11:49:33.447+00:00 (Rest) GET http://127.0.0.1:8091/pools/default/buckets 200
2022-09-15T11:49:33.511+00:00 (Rest) GET http://127.0.0.1:8091/pools/default 200
2022-09-15T11:49:33.511+00:00 Transferring from Couchbase Server 6.6.1
2022-09-15T11:49:33.512+00:00 (Plan) Successfully checked for data movement restrictions between statistic and statistic
2022-09-15T11:49:33.512+00:00 (Plan) Deciding which key value data to transfer for statistic
2022-09-15T11:49:33.513+00:00 (Rest) GET http://127.0.0.1:8091/pools/default/nodeServices 200
2022-09-15T11:49:34.397+00:00 (Plan) Successfully decided which key value data to transfer for statistic
2022-09-15T11:49:34.397+00:00 (Plan) Transferring new key value data for statistic to statistic
2022-09-15T11:49:34.441+00:00 (Rest) GET http://127.0.0.1:8091/pools/default/nodeServices 200
2022-09-15T11:49:35.142+00:00 (DCP) (statistic) (vb 820) Creating DCP stream {"uuid":268719278969323,"start_seqno":0,"end_seqno":28,"snap_start":0,"snap_end":0,"retries":0}
2022-09-15T11:49:35.143+00:00 (DCP) (statistic) (vb 895) Creating DCP stream {"uuid":191001500686165,"start_seqno":0,"end_seqno":27,"snap_start":0,"snap_end":0,"retries":0}
2022-09-15T11:49:35.144+00:00 (DCP) (statistic) (vb 404) Creating DCP stream {"uuid":50407715982084,"start_seqno":0,"end_seqno":26,"snap_start":0,"snap_end":0,"retries":0}
2022-09-15T11:49:35.145+00:00 (DCP) (statistic) (vb 619) Creating DCP stream {"uuid":222629674129219,"start_seqno":0,"end_seqno":27,"snap_start":0,"snap_end":0,"retries":0}
2022-09-15T11:49:35.147+00:00 (DCP) (statistic) (vb 397) Creating DCP stream {"uuid":102638502664807,"start_seqno":0,"end_seqno":25,"snap_start":0,"snap_end":0,"retries":0}
2022-09-15T11:49:35.149+00:00 (DCP) (statistic) (vb 520) Creating DCP stream {"uuid":260207993773150,"start_seqno":0,"end_seqno":34,"snap_start":0,"snap_end":0,"retries":0}
2022-09-15T11:49:35.151+00:00 (DCP) (statistic) (vb 635) Creating DCP stream {"uuid":85078990972823,"start_seqno":0,"end_seqno":42,"snap_start":0,"snap_end":0,"retries":0}
2022-09-15T11:49:35.152+00:00 (DCP) (statistic) (vb 404) Stream closed because all items were streamed | {"uuid":50407715982084,"snap_start":26,"snap_end":26,"snap_complete":true,"last_seqno":26,"retries":0}
2022-09-15T11:49:35.153+00:00 (DCP) (statistic) (vb 644) Creating DCP stream {"uuid":56126078317242,"start_seqno":0,"end_seqno":23,"snap_start":0,"snap_end":0,"retries":0}
2022-09-15T11:49:35.154+00:00 (DCP) (statistic) (vb 962) Creating DCP stream {"uuid":144769815367427,"start_seqno":0,"end_seqno":23,"snap_start":0,"snap_end":0,"retries":0}
2022-09-15T11:49:35.160+00:00 (DCP) (statistic) (vb 619) Stream closed because all items were streamed | {"uuid":222629674129219,"snap_start":27,"snap_end":27,"snap_complete":true,"last_seqno":27,"retries":0}
2022-09-15T11:49:35.168+00:00 (DCP) (statistic) (vb 397) Stream closed because all items were streamed | {"uuid":102638502664807,"snap_start":25,"snap_end":25,"snap_complete":true,"last_seqno":25,"retries":0}
2022-09-15T11:49:35.177+00:00 (DCP) (statistic) (vb 520) Stream closed because all items were streamed | {"uuid":260207993773150,"snap_start":34,"snap_end":34,"snap_complete":true,"last_seqno":34,"retries":0}
2022-09-15T11:49:35.184+00:00 (DCP) (statistic) (vb 635) Stream closed because all items were streamed | {"uuid":85078990972823,"snap_start":42,"snap_end":42,"snap_complete":true,"last_seqno":42,"retries":0}
2022-09-15T11:49:35.191+00:00 (DCP) (statistic) (vb 644) Stream closed because all items were streamed | {"uuid":56126078317242,"snap_start":23,"snap_end":23,"snap_complete":true,"last_seqno":23,"retries":0}
2022-09-15T11:49:35.607+00:00 (Gocbcore) memdClient read failure: read tcp x.x.x.x:42462->y.y.y.y:11210: read: connection reset by peer
2022-09-15T11:49:35.607+00:00 WARN: (DCP) (statistic) (vb 820) Stream closed due to unexpected error 'EOF | {"bucket":"statistic","last_dispatched_to":"y.y.y.y:11210","last_dispatched_from":"x.x.x.x:42462","last_connection_id":"0605afd1f59f7d7a/e1557ba748642d37"}' | {"uuid":268719278969323,"snap_start":0,"snap_end":28,"snap_complete":true,"last_seqno":27,"retries":0} -- couchbase.(*DCPAsyncWorker).End() at dcp_async_worker.go:457
2022-09-15T11:49:35.610+00:00 WARN: (DCP) (statistic) (vb 820) Received an unexpected error whilst streaming, beginning teardown: EOF | {"bucket":"statistic","last_dispatched_to":"y.y.y.y:11210","last_dispatched_from":"x.x.x.x:42462","last_connection_id":"0605afd1f59f7d7a/e1557ba748642d37"} -- couchbase.(*DCPAsyncWorker).handleDCPError() at dcp_async_worker.go:550
2022-09-15T11:49:35.610+00:00 WARN: (DCP) (statistic) (vb 895) Stream closed by backup client | {"uuid":191001500686165,"snap_start":0,"snap_end":27,"snap_complete":true,"last_seqno":0,"retries":0} -- couchbase.(*DCPAsyncWorker).End() at dcp_async_worker.go:455
2022-09-15T11:49:35.610+00:00 WARN: (DCP) (statistic) (vb 962) Closing DCP stream because client is shutting down | {"uuid":144769815367427,"snap_start":0,"snap_end":0,"snap_complete":true,"last_seqno":0,"retries":-1} -- couchbase.(*DCPAsyncWorker).openStream() at dcp_async_worker.go:174
2022-09-15T11:49:35.631+00:00 (Stats) Stopping stat collection
2022-09-15T11:49:35.631+00:00 (Cmd) Error backing up cluster: failed to execute cluster operations: failed to execute bucket operations: failed to transfer bucket data for bucket 'statistic': failed to transfer key value data: failed to transfer key value data: EOF | {"bucket":"statistic","last_dispatched_to":"y.y.y.y:11210","last_dispatched_from":"x.x.x.x:42462","last_connection_id":"0605afd1f59f7d7a/e1557ba748642d37"}
2022-09-15T11:49:35.631+00:00 (Cmd) Backed up bucket "statistic" failed
2022-09-15T11:49:35.631+00:00 (Cmd) Mutations backed up: 204, Mutations failed to backup: 0
2022-09-15T11:49:35.631+00:00 (Cmd) Deletions backed up: 0, Deletions failed to backup: 0
2022-09-15T11:49:35.631+00:00 (Cmd) Skipped due to purge number or conflict resolution: Mutations: 0 Deletions: 0
Could anybody to advise what could it be or what to check?