Cbbackup failing with Python error

Hi There,

I am getting an error when I am doing cbbackup on Community Edition 6.6.0 build 7909. here are the error details below. Anyone facing this kind of issue? How do I make cbbackup work in Couchbase CE ?

sudo /opt/couchbase/bin/cbbackup http://ipaddress:8091 ~/backup -u username -p password
Exception in thread w0:
Traceback (most recent call last):
  File "/opt/couchbase/lib/python/runtime/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/opt/couchbase/lib/python/runtime/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/couchbase/lib/python/pump.py", line 446, in run_worker
    curx).run()
  File "/opt/couchbase/lib/python/pump.py", line 504, in run
    rv_batch, batch = self.source.provide_batch()
  File "/opt/couchbase/lib/python/pump_dcp.py", line 249, in provide_batch
    rv = self.get_dcp_conn()
  File "/opt/couchbase/lib/python/pump_dcp.py", line 533, in get_dcp_conn
    self.opts.no_ssl_verify, self.opts.cacert)
  File "/opt/couchbase/lib/python/pump.py", line 1199, in get_mcd_conn
    conn = cb_bin_client.MemcachedClient(host, port, use_ssl=use_ssl, verify=verify, cacert=ca_cert)
  File "/opt/couchbase/lib/python/cb_bin_client.py", line 91, in __init__
    socket.SOCK_STREAM):
  File "/opt/couchbase/lib/python/runtime/lib/python3.7/socket.py", line 748, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

Exception ignored in: <function MemcachedClient.__del__ at 0x7f31d9a49510>
Traceback (most recent call last):
  File "/opt/couchbase/lib/python/cb_bin_client.py", line 120, in __del__
    self.close()
  File "/opt/couchbase/lib/python/cb_bin_client.py", line 117, in close
    self.s.close()
AttributeError: 'MemcachedClient' object has no attribute 's'

Hi Itssrinadh,
Thanks for using our products.
I try to reproduce you issue but I could not. I am able to backup travel-sample bucket using cbbackup as show in run log below

Blockquote
[root@s44015 ~]# /opt/couchbase/bin/cbbackup -u Administrator -p password couchbase://localhost:8091 /tmp/backup/
[####################] 100.0% (31591/estimated 31591 msgs)
bucket: travel-sample, msgs transferred…
: total | last | per sec
byte : 36196306 | 36196306 | 1971031.1
2021-02-22 16:53:41,623: mt (0, None)
done
[root@s44015 ~]# more /opt/couchbase/etc/runtime.ini
; Configuration Settings at install-time and runtime. Couchbase may
; modify this file, for example, at runtime when the server is secured
; and taken out of Admin Party mode.
[product]
title = Couchbase Server
version = 6.6.0-7909
license = community
[root@s44015 ~]#
Here is backup dir content
[root@s44015 ~]# ls /tmp/backup/
2021-02-23T005323Z
[root@s44015 ~]# ls /tmp/backup/2021-02-23T005323Z/
2021-02-23T005323Z-full
[root@s44015 ~]#

I use centos 7.6 server to install Couchbase server 6.6.0
If you have any question, drop me a line at thuan at couchbase dot com
Thanks

Hi @thuan Thank you for your reply.

I believe I have a lot of document revisions and I don’t know if it is the cause. Is there any way to find the root cause at my end? are logs helpful for you to check this issue at your end with CC 6.6.0 ?

Here are my server details.

NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"

I will try on Amazon Linux 2 and reply to you soon.
Thanks

1 Like

Sure, Thank you.

FYI, I got the same issue with macOS Catalina CC 6.6.0

Tried again in AWS with Amazon linux 2, I could not reproduce your issue.

[root@ip-172-31-10-133 ec2-user]# /opt/couchbase/bin/cbbackup -u Administrator -p password couchbase://localhost:8091 /tmp/backup/
[####################] 100.0% (31591/estimated 31591 msgs)
bucket: travel-sample, msgs transferred…
: total | last | per sec
byte : 36196306 | 36196306 | 11383021.9
2021-02-23 06:11:29,511: mt (0, None)
done
[root@ip-172-31-10-133 ec2-user]# /opt/couchbase/bin/cbbackup -u Administrator -p password couchbase://localhost:8091 /tmp/backup/ -v
2021-02-23 06:11:34,990: mt cbbackup…
2021-02-23 06:11:34,990: mt source : couchbase://localhost:8091
2021-02-23 06:11:34,990: mt sink : /tmp/backup/
2021-02-23 06:11:34,990: mt opts : {‘bucket_source’: None, ‘single_node’: False, ‘mode’: ‘full’, ‘id’: None, ‘key’: None, ‘vbucket_list’: None, ‘dry_run’: False, ‘username’: ‘’, ‘password’: ‘’, ‘username_dest’: None, ‘password_dest’: None, ‘ssl’: False, ‘no_ssl_verify’: True, ‘cacert’: None, ‘threads’: 4, ‘verbose’: 1, ‘silent’: False, ‘extra’: {‘batch_max_size’: 1000.0, ‘batch_max_bytes’: 400000.0, ‘cbb_max_mb’: 100000.0, ‘max_retry’: 10.0, ‘report’: 5.0, ‘report_full’: 2000.0, ‘recv_min_bytes’: 4096.0, ‘try_xwm’: 1.0, ‘nmv_retry’: 1.0, ‘rehash’: 0.0, ‘data_only’: 0.0, ‘design_doc_only’: 0.0, ‘conflict_resolve’: 1.0, ‘seqno’: 0.0, ‘mcd_compatible’: 1.0, ‘uncompress’: 0.0, ‘backoff_cap’: 10.0, ‘flow_control’: 1.0, ‘dcp_consumer_queue_length’: 1000.0, ‘allow_recovery_vb_remap’: 0.0}, ‘collection’: None, ‘force_txn’: False}
2021-02-23 06:11:34,992: mt Starting new HTTP connection (1): localhost
2021-02-23 06:11:34,997: mt bucket: travel-sample
2021-02-23 06:11:38,403: w0 source : couchbase://localhost:8091(travel-sample@127.0.0.1:8091)
2021-02-23 06:11:38,403: w0 sink : /tmp/backup/(travel-sample@127.0.0.1:8091)
2021-02-23 06:11:38,403: w0 : total | last | per sec
2021-02-23 06:11:38,403: w0 batch : 91 | 91 | 26.8
2021-02-23 06:11:38,403: w0 byte : 36196306 | 36196306 | 10649828.5
2021-02-23 06:11:38,403: w0 msg : 31591 | 31591 | 9294.8
[####################] 100.0% (31591/estimated 31591 msgs)
bucket: travel-sample, msgs transferred…
: total | last | per sec
batch : 91 | 91 | 26.5
byte : 36196306 | 36196306 | 10526079.0
msg : 31591 | 31591 | 9186.8
2021-02-23 06:11:38,441: mt Starting new HTTP connection (1): localhost
2021-02-23 06:11:38,443: mt Starting new HTTP connection (1): 127.0.0.1
2021-02-23 06:11:38,455: mt Starting new HTTP connection (1): localhost
2021-02-23 06:11:38,457: mt Starting new HTTP connection (1): 127.0.0.1
2021-02-23 06:11:38,462: mt Starting new HTTP connection (1): localhost
2021-02-23 06:11:38,463: mt Starting new HTTP connection (1): 127.0.0.1
2021-02-23 06:11:38,465: mt (0, None)
done
[root@ip-172-31-10-133 ec2-user]# more /opt/couchbase/etc/runtime.ini
; Configuration Settings at install-time and runtime. Couchbase may
; modify this file, for example, at runtime when the server is secured
; and taken out of Admin Party mode.
[product]
title = Couchbase Server
version = 6.6.0-7909
license = community
[root@ip-172-31-10-133 ec2-user]#

I will ask an engineer who is working in backup to get more information.
Thanks

Hi @itssrinadh,

I had a look over the issue the error produced:

 socket.gaierror: [Errno -2] Name or service not known

Normally indicates an issue with address resolution, where the most likely case is the node you are running the command on cannot resolve the hostnames of the other nodes. To make sure that this is the case would you mind re-runing the command but using -vvv and returning the output, this will print debug information. The command would be:

sudo /opt/couchbase/bin/cbbackup http://ipaddress:8091 ~/backup -u username -p password -vvv

Hi @Carlos_Gonzalez Thanks for your response. following is debug info.

So here are my observations. I have two servers in two different regions in AWS and if I take backup within the same server it is working fine. but if I try to take a backup from one server to another with two different regions we are getting the following error.

So how can we perform cbbackup or cbtransfer in this scenario?

[ec2-user@ip ~]$ sudo /opt/couchbase/bin/cbbackup http://ipaddress:8091 ~/backup -u username -p password -vvv
2021-02-23 15:04:24,171: mt cbbackup...
2021-02-23 15:04:24,172: mt  source : http://ipaddress:8091
2021-02-23 15:04:24,172: mt  sink   : /home/ec2-user/backup
2021-02-23 15:04:24,172: mt  opts   : {'bucket_source': None, 'single_node': False, 'mode': 'full', 'id': None, 'key': None, 'vbucket_list': None, 'dry_run': False, 'usernam
e': '<xxx>', 'password': '<xxx>', 'username_dest': None, 'password_dest': None, 'ssl': False, 'no_ssl_verify': True, 'cacert': None, 'threads': 4, 'verbose': 3, 'silent': Fa
lse, 'extra': {'batch_max_size': 1000.0, 'batch_max_bytes': 400000.0, 'cbb_max_mb': 100000.0, 'max_retry': 10.0, 'report': 5.0, 'report_full': 2000.0, 'recv_min_bytes': 4096
.0, 'try_xwm': 1.0, 'nmv_retry': 1.0, 'rehash': 0.0, 'data_only': 0.0, 'design_doc_only': 0.0, 'conflict_resolve': 1.0, 'seqno': 0.0, 'mcd_compatible': 1.0, 'uncompress': 0.
0, 'backoff_cap': 10.0, 'flow_control': 1.0, 'dcp_consumer_queue_length': 1000.0, 'allow_recovery_vb_remap': 0.0}, 'collection': None, 'force_txn': False}
2021-02-23 15:04:24,172: mt source_class: <class 'pump_dcp.DCPStreamSource'>
2021-02-23 15:04:24,178: mt Starting new HTTP connection (1): ipaddress
2021-02-23 15:04:24,329: mt "GET /pools/default/buckets HTTP/1.1" 200 20745
2021-02-23 15:04:24,401: mt sink_class: <class 'pump_bfd.BFDSink'>
2021-02-23 15:04:24,401: mt source_buckets: bucket1,bucket2
2021-02-23 15:04:24,401: mt bucket: bucket1
2021-02-23 15:04:24,401: mt  source_nodes: ipaddress:8091
2021-02-23 15:04:24,402: mt  enqueueing node: ipaddress:8091
2021-02-23 15:04:24,402: mt rest_request: <ud>username</ud>@ipaddress:8091/pools/default/buckets/bucket1/stats/curr_items; reason: total_msgs
2021-02-23 15:04:24,402: w0  node: ipaddress:8091
2021-02-23 15:04:24,403: w0  Use alternate addresses: {'source': False, 'sink': False}
2021-02-23 15:04:24,403: w0 sink_bucket: bucket1
2021-02-23 15:04:24,404: w0   DCPStreamSource connecting mc: ipaddress:11210
Exception in thread w0:
Traceback (most recent call last):
  File "/opt/couchbase/lib/python/runtime/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/opt/couchbase/lib/python/runtime/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/opt/couchbase/lib/python/pump.py", line 446, in run_worker
    curx).run()
  File "/opt/couchbase/lib/python/pump.py", line 504, in run
    rv_batch, batch = self.source.provide_batch()
  File "/opt/couchbase/lib/python/pump_dcp.py", line 249, in provide_batch
    rv = self.get_dcp_conn()
  File "/opt/couchbase/lib/python/pump_dcp.py", line 533, in get_dcp_conn
    self.opts.no_ssl_verify, self.opts.cacert)
  File "/opt/couchbase/lib/python/pump.py", line 1199, in get_mcd_conn
    conn = cb_bin_client.MemcachedClient(host, port, use_ssl=use_ssl, verify=verify, cacert=ca_cert)
  File "/opt/couchbase/lib/python/cb_bin_client.py", line 91, in __init__
    socket.SOCK_STREAM):
  File "/opt/couchbase/lib/python/runtime/lib/python3.7/socket.py", line 748, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

Exception ignored in: <function MemcachedClient.__del__ at 0x7fe003d65598>
Traceback (most recent call last):
  File "/opt/couchbase/lib/python/cb_bin_client.py", line 120, in __del__
    self.close()
  File "/opt/couchbase/lib/python/cb_bin_client.py", line 117, in close
    self.s.close()
AttributeError: 'MemcachedClient' object has no attribute 's'
2021-02-23 15:04:24,534: mt rest_request: <ud>usrename</ud>@ipaddress:8091/pools/default/buckets/bucket1/stats/vb_active_resident_items_ratio; reason: total_msgs

Hi @itssrinadh ,

I imagine you are replacing the hosts and IPs with ipaddress. Could you tell me if the IP address in the debug line

DCPStreamSource connecting mc: ipaddress:11210

is the same one you provided in the command line?
Also, is it the same one that shows in the REST endpoint /pools/default for that node?

Hi @Carlos_Gonzalez For security reasons I changed all IP addresses, bucket names and endpoints when i shared with you. but i used original details when I did debug mode in the server.

Yes that makes sense.

What I ask is if the IP you put in the command line and the one in the log line that starts with DCPStreamSource connecting mc: ... the same?

I ask this as if they are not the same then this means that you are hitting a DNS issue and the fix would be to configure DNS/ the node names correctly. To confirm this is the case we have to check that those two values are not the same (we do not need to know the values, just if they are the same or not).

Ok Here are few more inputs.

My couchbase cluster setup server name address used as ip-xx-xx-xx-xx.ec2.internal (This is a EC2 Private IPv4 DNS)

So when i do a backup from other server i do using like this

sudo /opt/couchbase/bin/cbbackup http://ipaddress:8091 ~/backup -u username -p password -vvv

Note: here ipaddress is a public ip of ec2

DCPStreamSource connecting mc: ip-xx-xx-xx-xx.ec2.internal:11210

Note: here ip address is a EC2 private Ip4 DNS

Ok, so what is happening is the following:

When cbbackup connects to the cluster it will boostrap by getting the node map from the node you provided. It then proceeds to get the configured hostname for each node and use that address to communicate. In this case the nodes are configured with the private addresses thus we try to communicate using those addresses, which cannot be resolved from you other instance, which results in the error you are seeing.

The are several possible solutions:

  1. Configure alternate address so that it will use the public address when in the command is given the public address and the internal when given the internal IP. You can read the docs on how to configure
  2. Update you DNS or /etc/hosts so that the internal hosts get resolved to the public ones
  3. Set up the cluster with the external hosts.

I also recommend having a look at the documentation page on deploying on different cloud providers:
https://docs.couchbase.com/server/5.1/cloud/couchbase-aws-best-practices.html#aws-network

Hope that helps.

1 Like

Hi @Carlos_Gonzalez Sure. i will check it and thanks for your help.

Hi @Carlos_Gonzalez As you said configure an alternate address. will the following work in my case? as i configure hostname or ipaddress for cluster setup with ip internal.

curl -v -X PUT -u Administrator:password \
http://internalprivateipaddress:8091/node/controller/setupAlternateAddresses/external \
-d hostname=publicipaddress

So always when we setup cluster, is it good idea to use public ipaddress for cluster hostname or ip address?

The command looks correct, note that you probably want to do this for every node in the cluster.

So always when we setup cluster, is it good idea to use public ipaddress for cluster hostname or ip address?

This dependes on your use cases and your network configuration, for example if you don’t want anything outside of your private network to be able to communicate with the node then you probably want to stick with internal IPs if you want to be able to access it from the outside then you probably want to do public/external ones. And if you want to do both then you can use alternate address configuration.

There is a section on the docs that cover naming nodes and explains using hostnames and IPs.

Hey @Carlos_Gonzalez Thanks for your response. it is working good.