{"id":7726,"date":"2019-10-13T14:34:34","date_gmt":"2019-10-13T21:34:34","guid":{"rendered":"https:\/\/www.couchbase.com\/blog\/?p=7726"},"modified":"2025-06-13T21:25:16","modified_gmt":"2025-06-14T04:25:16","slug":"tips-and-tricks-for-upgrading-from-python-2-to-python-3","status":"publish","type":"post","link":"https:\/\/www.couchbase.com\/blog\/tips-and-tricks-for-upgrading-from-python-2-to-python-3\/","title":{"rendered":"How to Update Python 2 to Python 3 & The Differences Between Them"},"content":{"rendered":"

Introduction<\/span><\/h2>\n

The last major Python upgrade — to version 3 — arrived in Dec. 2008, nearly 12 years ago. And yet there is a good chance that you are still working on the Python 2 product or test code. If so, then you may be\u00a0 seeing the below deprecation message as a reminder to update the Python version you’re working with.<\/span><\/p>\n

“DEPRECATION: Python 2.7 will reach the end of its life on January 1st, 2020. Please upgrade your Python as Python 2.7 won\u2019t be maintained after that date. A future version of pip will drop support for Python 2.7.”<\/span><\/p>\n

Please take this seriously and plan accordingly: Python updates to 3.x are not backward compatible. What you write with Python 2.x versions may not function properly when using 3.x<\/span>.<\/span><\/p>\n

Be sure to also read the fine print. According to the website for the programming language, the final Python upgrade release date is still TBD: \u201cBeing the last of the 2.x series, 2.7 will receive bugfix support until 2020. Support officially stops January 1 2020, but the final release will occur after that date.\u201d [1<\/a>]<\/span><\/p>\n

So, Python 2 is entering into unsupported mode by the end of this year. If you haven\u2019t yet done so, now is a good time to migrate the current Python 2 code to Python 3 syntax and stick to Python 3 going forward. <\/span><\/p>\n

Why don\u2019t teams just jump start on this Python 2 to 3 migration? One of the major hurdles is that the majority of working code simply breaks (read more at <\/span>why-was-python-3-made-incompatible-with-python-2<\/span><\/a>), either because of the direct language syntax or issues with third party APIs. Let\u2019s be fair here: few of us would bother with migration if the new Python updates were backwards compatible. Instead, version 2 will go unsupported, forcing many — including us here at Couchbase — to prioritize migration. Even if the team crosses the bug fix support deadline, it’s ok (because your code is still working). As a team, we decided it\u2019s better to migrate as close to this date as possible so that we are on the same page with other Python community members and learn alongside them.\u00a0\u00a0<\/span><\/p>\n

This document is a collection of tips and tricks we learned while upgrading to Python 3 along with common problems we encountered during the Couchbase test infra migration process. As you’ll see we manually update python by command line after kick-starting with an automated process. Your approach may be different. Regardless, start as soon as you can. Updating python from version 2 to version 3 is important.<\/span><\/p>\n

Couchbase<\/span><\/a> is an open source Enterprise-class MultiCloud to Edge NoSQL Database. The Couchbase functional testing framework, TestRunner has been developed in Python 2. The TestRunner git\u00a0 repository can be found at<\/span> https:\/\/github.com\/couchbase\/testrunner<\/span><\/a> .\u00a0 Our goal now is to completely switch to Python 3 runtime instead of co-running with both Python 3 and Python 2.<\/span><\/p>\n

As part of the Python upgrade process, we have identified the major changes needed to successfully port to version 3. Some of the problems you’ll read about we identified during the porting process. Our aim in sharing our learnings is to help you with your own migration. You can pick the latest Python 3.x version (it depends on the pre-release, stable, security-fixes version on a specific platform, 3.7 or 3.6), which we are referring to as Python 3 throughout this blog. See more details on the release at <\/span>Python releases download<\/span><\/a> and\u00a0 <\/span>Python 3 documentation<\/a>.<\/span><\/p>\n

Cheat Sheet<\/p>\n

 <\/p>\n

Major Changes: Python 2 vs. Python 3<\/span><\/h2>\n

To get an idea of the key differences between Python 2 and Python 3, here is the summary list of code changes needed.<\/p>\n

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
Python 2<\/b><\/td>\nPython 3<\/b><\/td>\n<\/tr>\n
Text utf-8 : str<\/span>
\n‘<\/span>‘<\/span><\/td>\n		Text is unicode : str<\/span>
\nu”<\/span><\/td>\n<\/tr>\n
Binary is same as Text: bytes\/str<\/span>
\nExample:<\/span>
\nfile.read(6) == ‘GIF89a’<\/span><\/td>\n		Binary data is represented as b prefix: bytes<\/span>
\n\u00a0b”<\/span>
\nUse decode() to get the string, encode() to get bytes.<\/span>Examples:\u00a0<\/span>
\nfile.read(6) == b’GIF89a’<\/span>
\nb’hello’.decode() \u2192 ‘hello’<\/span>
\n\u2018hello\u2019.encode() \u2192 b’hello’<\/span>
\nstr(b\u2019hello\u2019) \u2192 “b’hello’<\/span><\/span>“<\/td>\n<\/tr>\n
Print statement<\/span>
\nExample: print ‘ ‘<\/span><\/td>\n		Print function<\/span>
\nExample:print(‘ ‘)<\/span><\/td>\n<\/tr>\n
Integer division<\/span>
\nExample: 5\/2=2<\/span><\/td>\n		Floor Division. Use 2 slashes<\/span>
\nExample: 5\/\/2 = 2 and 5\/2=2.5<\/span><\/td>\n<\/tr>\n
Float division<\/span>
\nExample: 5\/2.0 = 2.5 or 5.0\/2 = 2.5<\/span><\/td>\n		Float Division. Use single slash<\/span>
\nExample: 5\/2 = 2.5<\/span><\/td>\n<\/tr>\n
Long type is different from int<\/span>
\nlong\u00a0<\/span><\/td>\n		There is no long type. It is same as int<\/span><\/td>\n<\/tr>\n
xrange()<\/span><\/td>\nrange()<\/span><\/td>\n<\/tr>\n
Iteration functions had iter prefix. iterxxx()<\/span>
\nExample: iteritems()<\/span><\/td>\n		Dropped iter prefix. xxxx()<\/span>
\nExample: items()<\/span><\/td>\n<\/tr>\n
Lists are directly loaded (all elements loaded into memory when list is used)<\/span>
\nExample: for i in []\u00a0<\/span><\/td>\n		Lists are lazy loaded (when an element is accessed, then only loaded into memory)<\/span>
\nExample: for i in list([])\u00a0<\/span><\/td>\n<\/tr>\n
Dictionaries can be compared by default or against 2 dict.<\/span>
\nExample: sorted(dict)\u00a0<\/span><\/td>\n		Dictionaries can\u2019t be compared directly. sorted() should have key.<\/span><\/p>\n

Example: <\/span>sorted(expected_result,key=(lambda x: x[bucket.name][‘name’]))<\/span>For general dict\/list comparison, you can use below:\u00a0<\/span>
\n\u00a0from deepdiff import DeepDiff<\/span>
\n\u00a0<\/span>diffs = DeepDiff(actual_result[‘results’], expected_result[<\/span>‘results’<\/b>], <\/span>ignore_order<\/span>=<\/span>True<\/b>)<\/span>if <\/b>diffs:<\/span>
\n\u00a0\u00a0\u00a0<\/span>self<\/span>.assertTrue(<\/span>False<\/b>, diffs)<\/span><\/p>\n

Bytes and strings as values:<\/span>
\ndiffs = DeepDiff(<\/span>set<\/span>(actual_indexes), <\/span>set<\/span>(indexes_names), <\/span>ignore_order<\/span>=<\/span>True<\/b>, <\/span>ignore_string_type_changes<\/span>=<\/span>True<\/b>)<\/span><\/td>\n<\/tr>\n

string.replace(data[i],…)<\/span><\/td>\ndata[i].replace(..)<\/span><\/td>\n<\/tr>\n
urllib.urlencode()<\/span><\/td>\nNew modules<\/span><\/p>\n
    \n
  • http.client<\/span><\/li>\n
  • urllib.request, urllib.error, urllib.parse<\/span><\/li>\n
  • sgmllib3k<\/span><\/li>\n<\/ul>\n

    Examples:\u00a0<\/span>
    \nurllib.parse.urlencode()<\/span><\/td>\n<\/tr>\n

string.lowercase<\/span><\/td>\nAttributes:<\/span>
\nstring.ascii_lowercase<\/span>
\nstring.ascii_uppercase<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n

See the <\/span>testrunner py3 commits<\/span><\/a>\u00a0 for changes<\/span><\/p>\n

<\/h2>\n

Python 3 Setup<\/span><\/h2>\n

To setup Python 3 from scratch, run the below commands on a new host with major supported platforms.<\/span>
\nLater during the runtime, either use python 3 command or python in python 3 virtual env. Use either pip3 or pip3.x (pip3.6 for example) to install packages based on the installed Python 3 version.<\/span><\/p>\n

\n\n\n\n\n\n\n
Mac OS<\/b>
\n(Example: Your laptop)
\nDirect setup (pip3 automatically installed):<\/span>(https:\/\/wsvincent.com\/install-python3-mac\/)<\/p>\n
brew install python3<\/pre>\n
Virtual environment setup:<\/span><\/p>\n
$ pip3 install virtualenv\r\n$ mkdir ~\/environments\r\n$ virtualenv -p python3 ~\/environments\r\n$ source ~\/environments\/bin\/activate\r\n(environments) jmunta-mac:~ jagadeshmunta\r\n$ python -V\r\nPython 3.7.3\r\n(environments) jmunta-mac:~ jagadeshmunta\r\n<\/pre>\n
Install required libraries:<\/p>\n
$ pip3 install couchbase\r\n$ pip3 install sgmllib3k\r\n$ pip3 install paramiko\r\n$ pip3 install httplib2\r\n$ pip3 install pyyaml\r\n$ pip3 install Geohash\r\n$ pip3 install python-geohash\r\n$ pip3 install deepdiff\r\n$ pip3 install pyes\r\n\r\n<\/pre>\n
For now, the below modification is required to the common Python 3 http client otherwise, you would hit an error.<\/p>\n
$ vi \/usr\/lib64\/python3.6\/http\/client.py\u00a0\u00a0 \r\nchunk to chunk.encode() in similar to the below:\r\n1078 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 chunk = f'{len(chunk):X}\\r\\n'.encode('ascii') + chunk.encode() \\<\/pre>\n
 <\/td>\n<\/tr>\n
CentOS\u00a0\u00a0<\/b>
\n(Example node: Jenkins Slave)<\/span>
\nDirect setup and virtual environment:<\/p>\n
# yum -y install https:\/\/centos7.iuscommunity.org\/ius-release.rpm\r\n# yum list available > \/tmp\/available_pkgs.txt\r\n# cat \/tmp\/available_pkgs.txt\u00a0 |egrep python3\r\n# yum -y install python36u\r\n# python3.6 --version\r\n# yum -y install python36u-pip\r\n# yum -y install python36u-devel\r\n# python3.6 -V\r\n# mkdir ~\/environments\r\n# cd ~\/environments\/\r\n# python3.6 -m venv my_env\r\n# source ~\/environments\/my_env\/bin\/activate\r\n# python -V<\/pre>\n
Install required libraries:<\/p>\n
# pip3.6 install requests\r\n# pip3.6 install sgmllib3k\r\n# pip3.6 install paramiko\r\n# pip3.6 install httplib2\r\n# pip3.6 install pyyaml\r\n# pip3.6 install pytz\r\n# pip3.6 install Geohash\r\n# pip3.6 install python-geohash\r\n# pip3.6 install deepdiff\r\n# pip3.6 install pyes<\/pre>\n
 <\/p>\n
Perform Couchbase CSDK and Python SDK installation on new slave:<\/span><\/p>\n
# yum -y install wget git\r\n# wget https:\/\/packages.couchbase.com\/releases\/couchbase-release\/couchbase-release-1.0-6-x86_64.rpm\r\n# rpm -iv couchbase-release-1.0-6-x86_64.rpm\r\n# yum install libcouchbase-devel libcouchbase2-bin gcc gcc-c++\r\n# pip3.6 install couchbase<\/pre>\n
For now, the below modification is required to the common Python 3 http client otherwise, you would hit an error.<\/p>\n
$ vi \/usr\/lib64\/python3.6\/http\/client.py\u00a0\u00a0 \r\nchunk to chunk.encode() in similar to the below: \r\n1078 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 chunk = f'{len(chunk):X}\\r\\n'.encode('ascii') + chunk.encode() \\<\/pre>\n
 <\/td>\n<\/tr>\n
Ubuntu slave using for Python 3 runtime verification<\/b>
\nDirect Setup:<\/p>\n
# apt install python3-pip<\/pre>\n
 <\/p>\n
Install the required libraries:<\/p>\n
# pip3 install sgmllib3k\r\n# pip3 install paramiko\r\n# pip3 install httplib2\r\n# pip3 install pyyaml\r\n# pip3 install pytz (NOTE: used by xdcr tests)\r\n# pip3 install Geohash\r\n# pip3 install python-geohash\r\n# pip3 install deepdiff\r\n# pip3 install pyes<\/pre>\n
Install CSDK and Python SDK installation: (Ref:<\/span>\u00a0<\/span>https:\/\/docs.couchbase.com\/c-sdk\/2.10\/start-using-sdk.html )<\/span><\/a><\/p>\n
# wget -O- https:\/\/packages.couchbase.com\/ubuntu\/couchbase.key | sudo apt-key add -\u00a0\r\n# cat \/etc\/apt\/sources.list.d\/couchbase.list\r\ndeb [ arch=amd64 ] https:\/\/packages.couchbase.com\/releases\/couchbase-server\/enterprise\/deb\/ bionic bionic\/main\r\ndeb [ arch=amd64 ] https:\/\/packages.couchbase.com\/releases\/couchbase-server\/community\/deb\/ bionic bionic\/main\r\ndeb https:\/\/packages.couchbase.com\/ubuntu bionic bionic\/main\u00a0\r\n# apt-get update\u00a0\r\n#\u00a0 apt-get install libcouchbase-dev libcouchbase2-bin build-essential\u00a0\r\n#\u00a0 pip3 install couchbase<\/pre>\n
For now, the below modification is required to the common Python 3 http client otherwise, you would hit an error.<\/p>\n
$ vi \/usr\/lib64\/python3.6\/http\/client.py\u00a0\u00a0 \r\nchunk to chunk.encode() in similar to the below: \r\n1078 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 chunk = f'{len(chunk):X}\\r\\n'.encode('ascii') + chunk.encode() \\<\/pre>\n
 <\/td>\n<\/tr>\n
Windows<\/b>
\nDownload and install: <\/span>https:\/\/www.python.org\/ftp\/python\/3.7.4\/python-3.7.4.exe<\/span><\/a><\/p>\n
C:\\Python\\Python37-32>python.exe -VPython 3.7.4\r\nD:\\py3porting\\testrunner>set PATH=C:\\Python\\Python37-32;C:\\Python\\Python37-32\\Scripts;%PATH%\r\nD:\\py3porting\\testrunner>pip3 -V\r\npip 19.0.3 from c:\\python\\python37-32\\lib\\site-packages\\pip (python 3.7)<\/pre>\n
Install required libraries:<\/p>\n
pip3 install libcouchbase\r\npip3 install sgmllib3k\r\npip3 install paramiko\r\npip3 install httplib2\r\npip3 install pyyaml\r\npip3 install Geohash\r\npip3 install python-geohash\r\npip3 install deepdiff\r\npip3 install pyes<\/pre>\n
 <\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<\/div>\n
Porting Process<\/span><\/h2>\n
At a high level, the porting is a three step process. 1) Auto conversion 2) Manual changes 3) Runtime validation and fix<\/p>\n
At first, clone the original repository and have the basic automatic conversion changes. Checkin the changes as a new repository until full conversion is done. This way, the current regression cycles can go without interruption.<\/p>\n
1. Auto conversion<\/span><\/h3>\n
There is an automated tool called <\/span>2to3<\/span><\/a> tool, provided by the Python 3 team that helps in taking care of a few common patterns like print, exception, list wrapping, relative imports etc.\u00a0\u00a0<\/span><\/p>\n
You can start with a single directory in the locally cloned workspace to do double check. Later, the conversion can be done entirely on entire code so that basic porting is taken care of.<\/span><\/p>\n
Below are some of the sample 2to3 conversion commands on the MacOS. In the last command, note that all idioms were applied. This way, the first time conversion can take care of key changes.<\/p>\n
(myenv) jmunta-mac:myscripts jagadeshmunta$ 2to3 . -o new -n -w .\r\nhq-mac:testrunner jagadeshmunta$ cd lib; mv lib\/couchbase_helper ..\/couchbase_helper\r\nhq-mac:testrunner jagadeshmunta$ 2to3 -f all -f buffer -f idioms -f set_literal -f ws_comma\u00a0 -n -o ~\/p3testrunner_3 -w . |tee ~\/2to3_3.txt\r\nhq-mac:testrunner jagadeshmunta$ time 2to3 -f all -f buffer -f idioms -f set_literal -f ws_comma\u00a0 -n -w . |tee ~\/2to3_4.txt\r\n$ 2to3 -f all -f buffer -f idioms -f set_literal -f ws_comma\u00a0 -n -o ~\/p3testrunner_helper -w ..\/couchbase_helper |tee ~\/2to3_helper.txt\r\ncp -R ~\/p3testrunner_helper\/* .<\/pre>\n
 <\/p>\n
2. Manual changes<\/span><\/h3>\n
The auto conversion doesn\u2019t do the complete porting. The below common problems might be experienced during the porting process than the common syntax changes done by the auto conversion 2to3 tool.\u00a0<\/span><\/p>\n
Run the test class and see if there are any errors and fix appropriately, deciding whether to switch from bytes to str or str to bytes or some sort\/comparison issue where one has to fix the key name in the sorted function. This is an iterative process until all the code runtime has been validated.<\/span><\/p>\n
Once a common pattern for sure is clear, then you can do grep and sed to replace across many class files. If you are not sure about other code until runtime, then defer until that test class is executed.\u00a0<\/span><\/p>\n
There might be issues with third party libraries\/modules might have changed, those need to be searched on the web and used appropriately.<\/span><\/p>\n
Make sure all the code path is covered by running across all supported platforms and parameters.<\/span><\/p>\n
3. Runtime Validation and Fix<\/h3>\n
Once the conversion is done, then perform a lot of code runtime as Python is a dynamic language. Otherwise, the changes can break the things if you do just visual static code inspection\/changes. You can start with basic sanity tests, acceptance tests and then select full tests from a single module of tests. <\/span><\/p>\n
Once you’re comfortable, then go with all other modules one by one. Keep checking in the changes to the new repository. In addition, you need to make sure there are no regressions with ported changes from this new repository by running sanity tests on the newer builds. Also, the validation should include all the supported platforms with Python 3.<\/span><\/p>\n
 <\/p>\n
Python 3 Ported Code and Status<\/span><\/h2>\n
Below is where to find the new repository for Python 3 ported code until it is merged to the main repository. The plan is to do one cycle of porting or intermediately take the changes from the main repo and do a manual merge to this.<\/span><\/p>\n
https:\/\/github.com\/couchbaselabs\/testrunner-py3\/<\/span><\/a><\/p>\n
(Branch: master)<\/span><\/p>\n
Many common changes were already done but not completed as there might be some other runtime issues. Fixes in common can also be regressed to the earlier fixes because of assumptions on input value type conversions. There is still some more ported code to be validated with Python 3 and the effort is still in progress.<\/span><\/p>\n
Now, let me show you the common issues that occurred<\/span> during the runtime validation. You can use this as a reference when you hit an issue to see if you are having the similar issue. You can apply the same solution and see if it works for you. And if you have any<\/span> new ideas, you can put them in the comments.<\/p>\n
Common Runtime Problems<\/span><\/h2>\n
\u00a0<\/span><\/p>\n
1. Problem(s): <\/span><\/h4>\n