Cbrecovery error: unable to access REST API

Hi there,

I’m trying to make cbrecovery tool work, but so far with no luck. Maybe someone would help me finding the problem, any help would be greatly appreciated.
The problems looks like this:

/opt/couchbase/bin/cbrecovery http://Administrator:password@remotehost:8091 http://Administrator:password@failedhost:8091 -b testbuck -B testbuck -n -v
2016-09-28 12:56:17,871: MainThread error: unable to access REST API: failedhost:8091/pools/default/buckets/testbuck/controller/startRecovery; please check source URL, server status, username (-u) and password (-p); reason: start_recovery
error: unable to access REST API: failedhost:8091/pools/default/buckets/testbuck/controller/startRecovery; please check source URL, server status, username (-u) and password (-p); reason: start_recovery

One important thing to mention is that the testbuck bucket is configured on a dedicated port (different from 11211) which means it’s auth-less. The fact that I’m using a dedicated port for the bucket made the Couchbase REST API unprotected as well, which was a huge surprise to me (see Is REST API in fact protected with Basic Authentication for more details). My guess is that it’s somehow related to the problem, as I get the same problem with cbbackup:

/opt/couchbase/bin/cbbackup http://remotehost:8091 /backup/couchbase -u Administrator -p password -b testbuck -n
error: unable to access REST API: remotehost:8091/pools/default/buckets; please check source URL, server status, username (-u) and password (-p)

But if I remove username and password parameters it works (!):

/opt/couchbase/bin/cbbackup http://remotehost:8091 /backup/couchbase -b testbuck -n
done, but no data written due to dry-run

I tried to do the same trick with cbrecovery tool but it didn’t help, get the same error:

/opt/couchbase/bin/cbrecovery http://remotehost:8091 http://failedhost:8091 -b testbuck -B testbuck -n -v
2016-09-28 13:27:32,633: MainThread error: unable to access REST API: failedhost:8091/pools/default/buckets/testbuck/controller/startRecovery; please check source URL, server status, username (-u) and password (-p); reason: start_recovery
error: unable to access REST API: failedhost:8091/pools/default/buckets/testbuck/controller/startRecovery; please check source URL, server status, username (-u) and password (-p); reason: start_recovery

Another thing, that cbrecovery tool depends on XDCR, but the documentation basically explains nothing about how exactly XDCR should be configured to allow the tool work. Here is a quote from Recover Partitions from a Remote Cluster section:

Data recovery from remote clusters requires an XDCR environment and adequate amount of memory and disk space to support the workload and recovered data.

Which really explains nothing.

My assumption is that replication should be configured on the remotehost server, i.e. replication from remotehost to failedhost. I did that but again that didn’t fix the problem.

Any suggestions, ideas? Does it look as a bug somewhere?

Maybe a comment from developers?
Could anyone explain how {{cbrecovery}} tool should work in regards to XDCR configuration?