Issues with backup of a couchbase cluster deployed on openshift cluster

Couchbase Server Kubernetes
1

Hello,

I’ve created an openshift cluster and deployed a couchbase cluster using redhat openshift container platform. It created 3 pods named cb-example-0000, cb-example-0001 and cb-example-0002.

I’ve also created a route cb-route-001 using which i can access couchbase webconsole.

on a local linux machine with couchbase installed, using the route’s hostname and port 80, i was able to run bucket-list command and it listed the available buckets on the cluster.

/opt/couchbase/bin/couchbase-cli bucket-list -c cb-route001-hostname:80 --username Administrator --password

I’m trying to run a backup command using the same hostname and it’s failing :

/opt/couchbase/bin/cbbackupmgr backup -c cb-route001-hostname:80 -u Administrator -p -a /data/backup27 -r cluster

Error backing up cluster: lookup cb-example-0000.cb-example.couchbase001.svc on 172.16.191.50:53: no such host

and i see below loglines in backup-0.log :

2022-12-28T14:52:11.581+05:30 (Cmd) cbbackupmgr version 7.0.3-7031 Hostname: oc1 OS: linux Version: 4.18.0-80.el8.x86_64 Arch: amd64 vCPU: 1 Memory: 8173346816 (7.61GiB)
2022-12-28T14:52:11.581+05:30 (Cmd) backup -c cb-route001-hostname:80 -u Administrator -p ***** -a /data/backup27 -r cluster
2022-12-28T14:52:11.581+05:30 (Cmd) mounted archive with id: cd9f64bf-fc2d-4501-86d6-6fecbb9949af
2022-12-28T14:52:11.581+05:30 (REST) (Attempt 1) (GET) Dispatching request to ‘http://cb-route001-couchbase001.apps.new1509.gcp-cv.commvault.com:80/pools/default/nodeServices’
2022-12-28T14:52:13.726+05:30 (REST) (Attempt 1) (GET) (200) Received response from ‘http://cb-route001-couchbase001.apps.new1509.gcp-cv.commvault.com:80/pools/default/nodeServices’
2022-12-28T14:52:13.727+05:30 (REST) (Attempt 1) (GET) Dispatching request to ‘http://cb-example-0000.cb-example.couchbase001.svc:8091/pools’
2022-12-28T14:52:13.773+05:30 ERRO: (REST) (Attempt 1) (GET) Failed to dispatch request to ‘http://cb-example-0000.cb-example.couchbase001.svc:8091/pools’: Get “http://cb-example-0000.cb-example.couchbase001.svc:8091/pools”: dial tcp: lookup cb-example-0000.cb-example.couchbase001.svc on 172.16.191.50:53: no such host – rest.(*Request).execute() at request.go:228
2022-12-28T14:52:13.773+05:30 (Cmd) Error backing up cluster: failed to get backup transferable: failed to create REST client: failed to get cluster information: failed to get cluster metadata: failed to execute request: http client failed to dispatch/receive request/response: Get “http://cb-example-0000.cb-example.couchbase001.svc:8091/pools”: dial tcp: lookup cb-example-0000.cb-example.couchbase001.svc on 172.16.191.50:53: no such host

Eventhough route’s hostname is used, we’re not sure why it’s using pod’s hostname “cb-example-0000.cb-example.couchbase001.svc” internally and failing.

Can somebody please check and help us get backups working?

Regards,
Sruthi Maddileti.

2

Hi Sruthi,

Apologies for missing this post.

This is actually due to MB-50897 (“Allow using network=external option in cbbackupmgr, cbexport, cbimport”) - these tools do not have the option to use Alternate Addresses in the same way as the Couchbase SDKs do, so it is not possible to backup from the outside of a Kubernetes/OpenShift environment.

We are working to target MB-50897 as soon as possible, and also have communicated this separately to your colleagues via email.

Thanks,
Chris Malarky (Product Manager, Couchbase)

3

Hello Chris,

As discussed earlier through emails, i’ve installed couchbase 7.2 on a client and tried running bucket-list and backup commands on the couchbase cluster deployed using redhat openshift container platform using below commands :

/opt/couchbase/bin/couchbase-cli bucket-list -c cb-route001-hostname:80 --username Administrator --password

bucket-list command worked fine as earlier

tried backup command using “network=external” tag as below :

/opt/couchbase/bin/cbbackupmgr backup -a /data/backup -r repo -c cb-route001-hostname:80?network=external -u Administrator -p test123

and it failed with below error again :

Error backing up cluster: lookup cb-example-0004.cb-example.gcp-ocp.svc on 172.16.69.10:53: no such host

Can you please let me know if the usage of network=external tag proper? If yes, can you please let me know why is it still failing with couchbase-7.2 as well?

Thanks in advance :slight_smile:

Regards,
Sruthi

4

Hi @sruthimaddileti,

The tools from Server 7.2 do now support the network=external connection strings, but they still need to be able to resolve the individual node names. This is achieved by using “External DNS”, as discussed in Configure Client SDKs.

Thanks,
Chris