Restore Data from couch files using cbdatarecovery

I had a mishap and need to transfer couch files from a dead node back into a cluster. I copied the data to a location on one of the nodes of the cluster and executed the following:

/opt/couchbase/bin/cbdatarecovery -c 127.0.0.1:8091 -u Administrator -p mypassword -d /home/ubuntu/data --include-data mybucket

The result is:
Transferring key value data for bucket 'mybuc… at 0B/s (about 28h31m40s remaining) 0 items / 44B
[ ] 0.29%
Recovery failed: operation failed: timeout reached

Running verbose gives these errors:

2024-01-22T14:35:38.364-05:00 WARN: (Pool) (mybucket) Failed to send document with key ‘ValueResult<rc=0x0, key=‘docid’, value=669460, cas=0x14fb0e760c230000, flags=0x0>’ due to a timeout: ambiguous timeout | {“InnerError”:{“InnerError”:{“InnerError”:{},“Message”:“ambiguous timeout”}},“OperationID”:“SetMeta”,“Opaque”:“0”,“TimeObserved”:300001115784,“RetryReasons”:null,“RetryAttempts”:0,“LastDispatchedTo”:“”,“LastDispatchedFrom”:“”,“LastConnectionID”:“”,“Internal”:{“ResourceUnits”:null}} – couchbase.(*MemcachedWorker).processOperation() at pool_worker.go:279

2024-01-22T14:35:38.364-05:00 (Cmd) Recovery failed: failed to execute cluster operations: failed to execute bucket operation for bucket ‘mybucket’: failed to transfer bucket data for bucket ‘mybucket’: failed to transfer key value data: failed to transfer key value data: failed to transfer data from file at ‘/home/ubuntu/data/mybucket/980.couch.1’: failed to traverse by sequence number: failed to follow node pointer: failed to follow node pointer: failed to follow node pointer: failed to run callback: operation failed: timeout reached
Recovery failed: operation failed: timeout reached

Can you please check that the target bucket is not running out of memory?

It might be beneficial to open a case with Customer Support. Or a jira issue at issues.couchbase.com

Yes - that was exactly the problem. I added in a 2nd node (this was a 1-node cluster), rebalanced, and re-attempted the operation and it was successful.

1 Like