JSON error (str vs bytes) in cbbackup and cbbackupwrapper

Couchbase Server
1

I’ve been experimenting with backing up a CB server from the command line.
On a fresh ubuntu 16.04 VM (with python3 installed and not python2), after cloning the couchbase-cli repo and installing pre-requisites, I’m getting the following error:

...
File "/home/ubuntu/couchbase-cli/pump_dcp.py", line 743, in total_msgs
reason="total_msgs", verify=opts.no_ssl_verify, cacert=opts.cacert)
File "/home/ubuntu/couchbase-cli/pump.py", line 1048, in rest_request_json
return None, rest_json, json.loads(rest_json)
File "/usr/lib/python3.5/json/__init__.py", line 312, in loads
s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'

This occurs in both pump.py and cbbackupwrapper. cbbackupwrapper, line 199, I see:

data = json.loads(response.read())

It seems that the data coming back from the /pools/default/buckets url are bytes rather than strings. By changing the line to:

data = json.loads(response.read().decode("utf-8"))

everything works again. Similarly for pump.py on line 1048.
Is my server configuration screwy that could be causing bytes to be returned? (p.s. For a proper solution, I probably should check the type of the result before decoding…)

Side Note: all the scripts expect /usr/bin/python. But only /usr/bin/python3 exists on a clean ubuntu 16.04 instance in AWS. Creating an alias doesn’t help because of the #! usage in CLI scripts. An ugly workaround is to: ln -s /usr/bin/python3 /usr/bin/python. This is usually frowned upon because you dont want python2 scripts accidentally running python3, but 2 is not installed.

2

Hello @paulr,

It’s great to see you cloning the repo and trying out the bleeding edge. We are in the middle of moving our code from Python2 to Python3.

  • Which SHA is your repo on?
  • Are you building Couchbase too?

From a quick glance at the code it looks like you have found a case where we not yet moved over to Python 3.

Is my server configuration screwy that could be causing bytes to be returned?

Pretty sure the server is all good and the issue is with couchbase-cli.

Side Note: all the scripts expect /usr/bin/python. But only /usr/bin/python3 exists on a clean ubuntu 16.04 instance in AWS. Creating an alias doesn’t help because of the #! usage in CLI scripts. An ugly workaround is to: ln -s /usr/bin/python3 /usr/bin/python . This is usually frowned upon because you dont want python2 scripts accidentally running python3, but 2 is not installed.

In the future we are changing the way the python code is packaged and will not be using the system python to avoid these issues.

3

@paulr

I just tested this using the latest version of couchbase-cli and it works:

 ./cbbackupwrapper http://localhost:9000 ~/backup-2 -u Administrator -p asdasd
Waiting for the backup to complete...
SUCCESSFULLY COMPLETED!

It would be good to know more about the environment you hit this in.

4

Hi @pvarley
Humble apologies! I missed your response (I didn’t notice the little (2) by the icon in the top corner.

I have updated couchbase-cli from the github repo:

mv couchbase-cli couchbase-cli.old
git clone https://github.com/couchbase/couchbase-cli.git

Looks like there were a fair number of changes:

diff -r couchbase-cli couchbase-cli.old
<lots of stuff>

But running the backup is still failing for me:

./cbbackupwrapper http://10.1.0.10:8091 /data/backups -u admin -p <my super secret password>

Traceback (most recent call last):
  File "./cbbackupwrapper", line 286, in <module>
    node, rest, args.username, args.password)
  File "./cbbackupwrapper", line 199, in getBuckets
    data = json.loads(response.read())
  File "/usr/lib/python3.5/json/__init__.py", line 312, in loads
    s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'

I’m positive I’m doing something stupid here!
As to the install of couchbase… I didn’t build it. I just downloaded it:

wget https://packages.couchbase.com/releases/6.0.0/couchbase-server-community_6.0.0-ubuntu16.04_amd64.deb
sudo dpkg -i couchbase-server-community_6.0.0-ubuntu16.04_amd64.deb

One other thing… using cbbackup from the db machine (rather than from the backup server) works fine:

./cbbackup --username=admin --password=<supersecretpassword> -b data http://localhost:8091 /data/db/backups/
5

Ok… to be clear - I’m running two VMs here - one for DB and one for backups.

The DB server was built from:
ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20180522
It then had couchbase installed. It has both python3 and python2 installed, with 2 being the default (python --version reports 2.7).

The backup server was built from
ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20181114
It then had couchbase-cli installed using git clone.

Note the dates are the only difference between the two. Both are fully up to date.
This is most weird.

I’m going to start from scratch with the backup server and try again.
I’ll report back.

6

Created a new instance - 16.04 Ubuntu as above.

sudo apt-get update
sudo apt-get upgrade
git clone https://github.com/couchbase/couchbase-cli.git
sudo apt install python3
sudo apt install python3-pip
sudo pip3 install snappy
cd couchbase-cli
python3 ./cbbackupwrapper http://10.1.0.10:8091 /data/backups -u admin -p <supersecretpassword>

gives the same error as before:

Traceback (most recent call last):
  File "./cbbackupwrapper", line 286, in <module>
    node, rest, args.username, args.password)
  File "./cbbackupwrapper", line 199, in getBuckets
    data = json.loads(response.read())
  File "/usr/lib/python3.5/json/__init__.py", line 312, in loads
    s.__class__.__name__))
TypeError: the JSON object must be str, not 'bytes'