Issues with backup of a couchbase cluster deployed on openshift cluster

Hello,

I’ve created an openshift cluster and deployed a couchbase cluster using redhat openshift container platform. It created 3 pods named cb-example-0000, cb-example-0001 and cb-example-0002.

I’ve also created a route cb-route-001 using which i can access couchbase webconsole.

on a local linux machine with couchbase installed, using the route’s hostname and port 80, i was able to run bucket-list command and it listed the available buckets on the cluster.

/opt/couchbase/bin/couchbase-cli bucket-list -c cb-route001-hostname:80 --username Administrator --password


I’m trying to run a backup command using the same hostname and it’s failing :

/opt/couchbase/bin/cbbackupmgr backup -c cb-route001-hostname:80 -u Administrator -p -a /data/backup27 -r cluster

Error backing up cluster: lookup cb-example-0000.cb-example.couchbase001.svc on 172.16.191.50:53: no such host


and i see below loglines in backup-0.log :

2022-12-28T14:52:11.581+05:30 (Cmd) cbbackupmgr version 7.0.3-7031 Hostname: oc1 OS: linux Version: 4.18.0-80.el8.x86_64 Arch: amd64 vCPU: 1 Memory: 8173346816 (7.61GiB)
2022-12-28T14:52:11.581+05:30 (Cmd) backup -c cb-route001-hostname:80 -u Administrator -p ***** -a /data/backup27 -r cluster
2022-12-28T14:52:11.581+05:30 (Cmd) mounted archive with id: cd9f64bf-fc2d-4501-86d6-6fecbb9949af
2022-12-28T14:52:11.581+05:30 (REST) (Attempt 1) (GET) Dispatching request to ‘http://cb-route001-couchbase001.apps.new1509.gcp-cv.commvault.com:80/pools/default/nodeServices
2022-12-28T14:52:13.726+05:30 (REST) (Attempt 1) (GET) (200) Received response from ‘http://cb-route001-couchbase001.apps.new1509.gcp-cv.commvault.com:80/pools/default/nodeServices
2022-12-28T14:52:13.727+05:30 (REST) (Attempt 1) (GET) Dispatching request to ‘http://cb-example-0000.cb-example.couchbase001.svc:8091/pools
2022-12-28T14:52:13.773+05:30 ERRO: (REST) (Attempt 1) (GET) Failed to dispatch request to ‘http://cb-example-0000.cb-example.couchbase001.svc:8091/pools’: Get “http://cb-example-0000.cb-example.couchbase001.svc:8091/pools”: dial tcp: lookup cb-example-0000.cb-example.couchbase001.svc on 172.16.191.50:53: no such host – rest.(*Request).execute() at request.go:228
2022-12-28T14:52:13.773+05:30 (Cmd) Error backing up cluster: failed to get backup transferable: failed to create REST client: failed to get cluster information: failed to get cluster metadata: failed to execute request: http client failed to dispatch/receive request/response: Get “http://cb-example-0000.cb-example.couchbase001.svc:8091/pools”: dial tcp: lookup cb-example-0000.cb-example.couchbase001.svc on 172.16.191.50:53: no such host


Eventhough route’s hostname is used, we’re not sure why it’s using pod’s hostname “cb-example-0000.cb-example.couchbase001.svc” internally and failing.

Can somebody please check and help us get backups working?

Regards,
Sruthi Maddileti.

Hi Sruthi,

Apologies for missing this post.

This is actually due to MB-50897 (“Allow using network=external option in cbbackupmgr, cbexport, cbimport”) - these tools do not have the option to use Alternate Addresses in the same way as the Couchbase SDKs do, so it is not possible to backup from the outside of a Kubernetes/OpenShift environment.

We are working to target MB-50897 as soon as possible, and also have communicated this separately to your colleagues via email.

Thanks,
Chris Malarky (Product Manager, Couchbase)

Hello Chris,

As discussed earlier through emails, i’ve installed couchbase 7.2 on a client and tried running bucket-list and backup commands on the couchbase cluster deployed using redhat openshift container platform using below commands :

/opt/couchbase/bin/couchbase-cli bucket-list -c cb-route001-hostname:80 --username Administrator --password

bucket-list command worked fine as earlier

tried backup command using “network=external” tag as below :

/opt/couchbase/bin/cbbackupmgr backup -a /data/backup -r repo -c cb-route001-hostname:80?network=external -u Administrator -p test123

and it failed with below error again :

Error backing up cluster: lookup cb-example-0004.cb-example.gcp-ocp.svc on 172.16.69.10:53: no such host

Can you please let me know if the usage of network=external tag proper? If yes, can you please let me know why is it still failing with couchbase-7.2 as well?

Thanks in advance :slight_smile:

Regards,
Sruthi

Hi @sruthimaddileti,

The tools from Server 7.2 do now support the network=external connection strings, but they still need to be able to resolve the individual node names. This is achieved by using “External DNS”, as discussed in Configure Client SDKs.

Thanks,
Chris

Hello Chris,

As per the article, we’ve provided a valid dns name at cluster level and used the same in connection string as below :

/opt/couchbase/bin/cbbackupmgr backup -c couchbases://console.gcp-cv.commvault.com?network=external -u Administrator -p test123 -a /data/backup -r clusternew09

Still the backups are failing. We feel that the usage of connection string is not proper and the cluster details are not mentioned anywhere. Can you please guide us ahead?

Regards,
Sruthi.

Please show the error.

Here’s the error :

But the username and password is correct. I’m able to login to the couchbase web console using the same creds.

The username/password is just a possibility. Check the documentation for any command line options that might provide more information or any log files

Below are the log lines from backup-0.log :

2024-01-17T09:18:15.368+05:30 (Cmd) cbbackupmgr version 7.2.2-6401 Hostname: cbrhel OS: linux Version: 4.18.0-80.el8.x86_64 Arch: amd64 vCPU: 1 Memory: 3941294080 (3.67GiB)
2024-01-17T09:18:15.379+05:30 (Cmd) config -a /data/backup -r clusternew17
2024-01-17T09:18:15.379+05:30 (Cmd) mounted archive with id: dec9119d-ae56-4734-9d9b-f7df3b59791e
2024-01-17T09:18:15.383+05:30 (Cmd) Backup repository clusternew17 created successfully in archive /data/backup
2024-01-17T09:18:35.833+05:30 (Cmd) cbbackupmgr version 7.2.2-6401 Hostname: cbrhel OS: linux Version: 4.18.0-80.el8.x86_64 Arch: amd64 vCPU: 1 Memory: 3941294080 (3.67GiB)
2024-01-17T09:18:35.833+05:30 (Cmd) backup -c couchbases://console.gcp-cv.commvault.com?network=external -u Administrator -p ***** -a /data/backup -r clusternew17
2024-01-17T09:18:35.833+05:30 (Cmd) mounted archive with id: dec9119d-ae56-4734-9d9b-f7df3b59791e
2024-01-17T09:18:46.870+05:30 ERRO: (REST) (Attempt 1) (GET) Failed to perform request to ‘https://console.gcp-cv.commvault.com:18091/pools/default/nodeServices’: Get “https://console.gcp-cv.commvault.com:18091/pools/default/nodeServices”: context deadline exceeded – logging.(*ToolsCommonLogger).Log() at tools_common.go:30
2024-01-17T09:18:46.870+05:30 WARN: (REST) failed to bootstrap client, will retry: failed to execute request: Get “https://console.gcp-cv.commvault.com:18091/pools/default/nodeServices”: context deadline exceeded – logging.(*ToolsCommonLogger).Log() at tools_common.go:28
2024-01-17T09:18:46.870+05:30 (Cmd) Error backing up cluster: failed to bootstrap client: failed to connect to any host(s) from the connection string, check username and password (CLI user reminder – if the password includes special characters, the password should be enclosed in quotes to prevent issues due to interaction with the shell)If using with a Couchbase Capella database, the database may be off or hibernated