[MB-8153] [Doc'd] cbworkloadgen shows error import sqlite3 module Created: 24/Apr/13  Updated: 16/May/13  Resolved: 16/May/13

Status: Closed
Project: Couchbase Server
Component/s: documentation, tools
Affects Version/s: 2.1.0
Fix Version/s: 2.1.0
Security Level: Public

Type: Bug Priority: Critical
Reporter: Thuan Nguyen Assignee: Thuan Nguyen
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: centos 5.7 64 bit


 Description   
Install couchbase server 2.0.2-773
cbworkloadge does not work

Error:

root@cen-0408 thuan]# /opt/couchbase/bin/cbworkloadgen -h
Error: could not import sqlite3 module
[root@cen-0408 thuan]# cd /opt/couchbase/bin
[root@cen-0408 bin]# ./cbworkloadgen -h
Error: could not import sqlite3 module

[root@cen-0408 bin]# ./cbworkloadgen
Error: could not import sqlite3 module



 Comments   
Comment by Pavel Paulau [ 25/Apr/13 ]
Interesting:

# cat /etc/redhat-release
CentOS release 5.8 (Final)

# cat /opt/couchbase/VERSION.txt
2.0.2-774-rel

# /opt/couchbase/bin/cbworkloadgen -h
Usage: cbworkloadgen [options]

Generate workload to destination.

Examples:
  cbworkloadgen -n localhost:8091
  cbworkloadgen -n 10.3.121.192:8091 -r .9 -i 100000 \
         -s 100 -b my-other-bucket --threads=10

Options:
  -h, --help show this help message and exit
  -r .95, --ratio-sets=.95
                        set/get operation ratio
  -n 127.0.0.1:8091, --node=127.0.0.1:8091
                        node's ns_server ip:port
  -b default, --bucket=default
                        insert data to a different bucket other than default
  -i 10000, --max-items=10000
                        number of items to be inserted
  -s 10, --size=10 minimum value size
  --prefix=pymc prefix to use for memcached keys or json ids
  -j, --json insert json data
  -l, --loop loop forever until interrupted by users
  -u USERNAME, --username=USERNAME
                        REST username for cluster or server node
  -p PASSWORD, --password=PASSWORD
                        REST password for cluster or server node
  -t 1, --threads=1 number of concurrent workers
  -v, --verbose verbose logging; more -v's provide more verbosity
Comment by Pavel Paulau [ 25/Apr/13 ]
Thuan, could you provide output of:

# ls -l /opt/couchbase/lib/python/
Comment by Thuan Nguyen [ 26/Apr/13 ]
this vm is available at 10.1.3.140 using key to login
python version 2.7
Comment by Bin Cui [ 26/Apr/13 ]
Looks like we cannot load sqlite again for this python version.
Comment by Bin Cui [ 26/Apr/13 ]
-bash-3.2$ python
Python 2.7 (r27:82500, Jul 29 2012, 09:49:59)
[GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sqlite3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/python27/lib/python2.7/sqlite3/__init__.py", line 24, in <module>
    from dbapi2 import *
  File "/opt/python27/lib/python2.7/sqlite3/dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ImportError: No module named _sqlite3

-bash-3.2$ cd /opt/couchbase/lib/python
-bash-3.2$ python
Python 2.7 (r27:82500, Jul 29 2012, 09:49:59)
[GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from pysqlite2 import dbapi2 as sqlite3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pysqlite2/dbapi2.py", line 27, in <module>
    from pysqlite2._sqlite import *
ImportError: pysqlite2/_sqlite.so: undefined symbol: PyUnicodeUCS4_DecodeUTF8
>>>
Comment by Bin Cui [ 26/Apr/13 ]
Quote:
This is usually caused by a mismatch in the Unicode mode of the python
interpreter and the extension module.

Python can be built to use either 2-byte or 4-byte Unicode code points.
If you build an extension on a Python interpreter that uses one, but use
it on another, this error is the most common result.

Comment by Maria McDuff (Inactive) [ 29/Apr/13 ]
Per Bin, this is a release blocker... he's working with Pavel on this issue.
Comment by Pavel Paulau [ 29/Apr/13 ]
Thuan, just a quick question. How did you install Python 2.7 on this machine? Was it installed after Couchbase Server?
Comment by Pavel Paulau [ 29/Apr/13 ]
http://review.couchbase.org/#/c/25938/
Comment by Thuan Nguyen [ 29/Apr/13 ]
On this vm, I just install couchbase server only. So I think python 2.7 is pre-installed before
Comment by Maria McDuff (Inactive) [ 06/May/13 ]
pls verify / close.
Comment by Thuan Nguyen [ 09/May/13 ]
I still repro this bug in build 2.0.2-793 on centos 5.7 64 bit server 10.1.3.140 (same vm)

couchbase-server-enterprise_x86_64_2.0.2-793-rel.rpm
[root@cen-0408 thuan]# rpm -i couchbase-server-enterprise_x86_64_2.0.2-793-rel.rpm
Minimum RAM required : 4 GB
System RAM configured : 3945388 kB

Minimum number of processors required : 4 cores
Number of processors on the system : 4 cores

Starting couchbase-server[ OK ]

You have successfully installed Couchbase Server.
Please browse to http://cen-0408:8091/ to configure your server.
Please refer to http://couchbase.com for additional resources.

Please note that you have to update your firewall configuration to
allow connections to the following ports: 11211, 11210, 11209, 4369,
8091, 8092 and from 21100 to 21299.

By using this software you agree to the End User License Agreement.
See /opt/couchbase/LICENSE.txt.

[root@cen-0408 thuan]# /opt/couchbase/bin/cbworkloadgen -h
Error: could not import sqlite3 module
[[root@cen-0408 thuan]# /opt/couchbase/bin/sqlite3
SQLite version 3.7.2
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite>
[1]+ Stopped /opt/couchbase/bin/sqlite3
[root@cen-0408 thuan]# cd /opt/couchbase/bin/
[root@cen-0408 bin]# ./cbworkloadgen -h
Error: could not import sqlite3 module
[root@cen-0408 bin]# ./sqlite3
SQLite version 3.7.2
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite>
[2]+ Stopped ./sqlite3
[root@cen-0408 bin]# python -V
Python 2.7
Comment by Pavel Paulau [ 10/May/13 ]
Normally Python 2.7 includes sqlite3, however you have custom installation that was compiled without sqlite support. So it obviously fails.

Expected:
$ python2.7 -c "import sqlite3"
$ echo $?
0

Your machine:
$ python2.7 -c "import sqlite3"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/python27/lib/python2.7/sqlite3/__init__.py", line 24, in <module>
    from dbapi2 import *
  File "/opt/python27/lib/python2.7/sqlite3/dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ImportError: No module named _sqlite3

Normal recommendation in such cases is to install sqlite-devel and rebuild Python. Or to use default OS setup.

Addressing such edge cases is too expensive effort IMHO. This is my input, PMs may have other suggestions.
Comment by Bin Cui [ 10/May/13 ]
The current assumption is that we support the following python environments:

1. python 2.4 which doesn't have sqlite3 bundled. we will install our bundled version.
2. python 2.5 and above. python will have its own version of sqlite3 installed. But if it doesn't meet our sqlite3 version requirement, we will install our bundled version.
3. This QA setup is something that we never meet before and it is not a standard environment, to say the least.
Comment by Anil Kumar [ 10/May/13 ]
tony to test this on clean VM to verify if this repro.

1. installing python 2.7 on clean vm
2. check if it comes with sqllite3 already
Comment by Pavel Paulau [ 11/May/13 ]
0. Package "sqlite-devel" must be installed *before* Python 2.7 installation.
Comment by Maria McDuff (Inactive) [ 13/May/13 ]
karen, pls doc sqlite-devel need to be installed first prior to python 2.7. thanks.

tony, pls verify / close. thanks.
Comment by kzeller [ 16/May/13 ]
Added to RN 2.0.2:

<rnentry>

<version ver="2.0.0m"/>

<class id="fix"/>

<issue type="cb" ref="MB-8153"/>


<rntext>

<para>
In the past when you used <command>cbworkloadgen</command> you see this error <literal>ImportError: No module named _sqlite3</literal>.
This has been fixed.</para>


</rntext>

</rnentry>
Comment by Thuan Nguyen [ 16/May/13 ]
Test on build 2.0.2-804 with 3 vms with python 2.4.3, 2.6.5 and 2.7.1
cbworkloadgen works as expected

/opt/couchbase/bin/cbworkloadgen -h
Usage: cbworkloadgen [options]

Generate workload to destination.

Examples:
  cbworkloadgen -n localhost:8091
  cbworkloadgen -n 10.3.121.192:8091 -r .9 -i 100000 \
         -s 100 -b my-other-bucket --threads=10



Options:
  -h, --help show this help message and exit
  -r .95, --ratio-sets=.95
                        set/get operation ratio
  -n 127.0.0.1:8091, --node=127.0.0.1:8091
                        node's ns_server ip:port
  -b default, --bucket=default
                        insert data to a different bucket other than default
  -i 10000, --max-items=10000
                        number of items to be inserted
  -s 10, --size=10 minimum value size
  --prefix=pymc prefix to use for memcached keys or json ids
  -j, --json insert json data
  -l, --loop loop forever until interrupted by users
  -u USERNAME, --username=USERNAME
                        REST username for cluster or server node
  -p PASSWORD, --password=PASSWORD
                        REST password for cluster or server node
  -t 1, --threads=1 number of concurrent workers
  -v, --verbose verbose logging; more -v's provide more verbosity
Generated at Wed Nov 26 04:34:52 CST 2014 using JIRA 5.2.4#845-sha1:c9f4cc41abe72fb236945343a1f485c2c844dac9.