ubuntu 10 & 12.04 32-bit unable to upgrade 1.8.1 to 2.0 latest build ( 1969 ) because cbtransfer is unable to open database file

Farshid Ghods added a comment - 30/Nov/12 4:59 PM same for ubuntu 10.04 Copying /opt/couchbase/var/lib/couchbase/config/config.dat.debsave     cp /opt/couchbase/var/lib/couchbase/config/config.dat.debsave /opt/couchbase/bin/install/../../var/lib/couchbase/config/config.dat Ensuring bucket data directories. Ensuring bucket data directory: /opt/couchbase/var/lib/couchbase/data/default     mkdir -p /opt/couchbase/var/lib/couchbase/data/default Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data     chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data     chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data Upgrading buckets. Upgrading bucket: default     /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couchbase/var/lib/couchbase/data -b default --source-vbucket-state=active --destination-vbucket-state=active cbdbupgrade pid: 915   [ ] 0.0% (0/194164 msgs) bucket: default, msgs transferred...        : total | last | per sec  batch : 1025 | 1025 | 453.4  byte : 0 | 0 | 0.0  msg : 0 | 0 | 0.0 done cbdbupgrade err: 0 Upgrading bucket: default     /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couchbase/var/lib/couchbase/data -b default --source-vbucket-state=replica --destination-vbucket-state=replica cbdbupgrade pid: 924 .............................................................................................................................................................................................................. bucket: default, msgs transferred...        : total | last | per sec  batch : 1025 | 1025 | 440.2  byte : 0 | 0 | 0.0  msg : 0 | 0 | 0.0 done cbdbupgrade err: 0 Upgrading bucket: default     /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couchbase/var/lib/couchbase/data -b default --source-vbucket-state=pending --destination-vbucket-state=pending cbdbupgrade pid: 933 .............................................................................................................................................................................................................. bucket: default, msgs transferred...        : total | last | per sec  batch : 1025 | 1025 | 440.3  byte : 0 | 0 | 0.0  msg : 0 | 0 | 0.0 done cbdbupgrade err: 0 Upgrading bucket: default     /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couchbase/var/lib/couchbase/data -b default --source-vbucket-state=dead --destination-vbucket-state=dead cbdbupgrade pid: 942 cbdbupgrade err: 11 ERROR: upgrade failed for bucket: default dpkg: error processing couchbase-server (--install):  subprocess installed post-installation script returned error exit status 1 Processing triggers for ureadahead ... Errors were encountered while processing:  couchbase-server

Steve Yen added a comment - 30/Nov/12 5:38 PM The files in the data directory seemed to have been in a mixed ownership between couchbase and root user. (sorry, didn't get a "ls -al" snapshot) At the moment, running cbtransfer manually as root seemed to work...      root@domU-12-31-39-09-A0-B2 :~# /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default stdout: | head -n 4     set 1154 0 0 1478     ???????????? [...clipped...]     set 1626 0 0 3806     ???????????? [...clipped...] But, I was also running /opt/couchbase/bin/sqlite3 against that default dir, so that might have inadvertently tweaked things.

Bin Cui added a comment - 30/Nov/12 5:38 PM When i manually run the /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couchbase/var/lib/couchbase/data -b default --source-vbucket-state=dead --destination-vbucket-state=dead with sudo ahead it, it will work. Otherwise, it doesn't. A security issue?

Farshid Ghods added a comment - 30/Nov/12 5:43 PM one centos 6.0 we get a segmentation fault ( which could be for using any cb* tool ) [ root@ip-10-226-91-111 ~]# rpm -U couchbase-server-community_x86_2.0.0-1969-rel.rpm Stopping couchbase-server ... Stopping couchbase-server Upgrading couchbase-server ...   /opt/couchbase/bin/install/cbupgrade -c /opt/couchbase/var/lib/couchbase/config -a yes Automatic mode: running without interactive questions or confirmations. Upgrading your Couchbase Server to 2.0.0-1969-rel. The upgrade process might take awhile. Analysing... Previous config.dat file is /opt/couchbase/var/lib/couchbase/config/config.dat Target node: ns_1@127.0.0.1 Couchbase should not be running.   Please use: /etc/init.d/couchbase-server stop Database dir: /opt/couchbase/var/lib/couchbase/data Buckets to upgrade: default Checking disk space available for buckets in directory:   /opt/couchbase/var/lib/couchbase/data   Free disk bucket space wanted: 103043072.0   Free disk bucket space available: 3398270976   Free disk space factor: 2.0   Ok. Analysis complete. Copying /opt/couchbase/var/lib/couchbase/config/config.dat     cp /opt/couchbase/var/lib/couchbase/config/config.dat /opt/couchbase/bin/install/../../var/lib/couchbase/config/config.dat Ensuring bucket data directories. Ensuring bucket data directory: /opt/couchbase/var/lib/couchbase/data/default     mkdir -p /opt/couchbase/var/lib/couchbase/data/default Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data     chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data     chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data Upgrading buckets. Upgrading bucket: default     /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couchbase/var/lib/couchbase/data -b default --source-vbucket-state=active --destination-vbucket-state=active cbdbupgrade pid: 1676 cbdbupgrade err: 139 ERROR: upgrade failed for bucket: default Starting couchbase-server[ OK ] You have successfully installed Couchbase Server. Please browse to http://ip-10-226-91-111:8091/ to configure your server. Please refer to http://couchbase.com for additional resources. Please note that you have to update your firewall configuration to allow connections to the following ports: 11211, 11210, 11209, 4369, 8091 and from 21100 to 21299. By using this software you agree to the End User License Agreement. See /opt/couchbase/LICENSE.txt. Stopping couchbase-serverNOTE: shutdown failed {badrpc,nodedown} [ root@ip-10-226-91-111 ~]# /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couchbase/var/lib/couchbase/data -b default --source-vbucket-state=active --destination-vbucket-state=active Segmentation fault (core dumped) [ root@ip-10-226-91-111 ~]# [ root@ip-10-226-91-111 ~]# /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couchbase/var/lib/couchbase/data -b default --source-vbucket-state=active --destination-vbucket-state=active Segmentation fault (core dumped) [ root@ip-10-226-91-111 ~]# /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couchbase/var/lib/couchbase/data -b default --source-vbucket-state=active --destination-vbucket-state=active^C [ root@ip-10-226-91-111 ~]# ls couchbase-server-community_x86_2.0.0-1969-rel.rpm couchbase-server-enterprise_x86_64_1.8.1.rpm couchbase-server-enterprise_x86_1.8.1.rpm [ root@ip-10-226-91-111 ~]# ls /tn ls: cannot access /tn: No such file or directory [ root@ip-10-226-91-111 ~]# /opt/couchbase/bin/cbenable_core_dumps.sh /tmp/ [ root@ip-10-226-91-111 ~]# /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couchbase/var/lib/couchbase/data -b default --source-vbucket-state=active --destination-vbucket-state=active Segmentation fault

Farshid Ghods added a comment - 30/Nov/12 5:53 PM ok i confirmed that cbbackup and cbrestore work fine on centos 6 32-bit so the seg fault is only coming from cbupgrade.

Steve Yen added a comment - 30/Nov/12 8:47 PM root@domU-12-31-39-09-A0-B2 :~# INSTALL_DONT_AUTO_UPGRADE=1 INSTALL_DONT_START_SERVER=1 dpkg -i /tmp/couchbase-server-community_x86_2.0.0-1969-rel.deb (Reading database ... 31755 files and directories currently installed.) Preparing to replace couchbase-server 1.8.1 (using .../couchbase-server-community_x86_2.0.0-1969-rel.deb) ... =INFO REPORT==== 1-Dec-2012::02:41:02 === Initiated server shutdown** at node ns_1@127.0.0.1 ** =INFO REPORT==== 1-Dec-2012::02:41:09 === Stopped ns_server application** at node ns_1@127.0.0.1 **  * Stopped couchbase-server libssl0.9.8 is installed. Continue installing Upgrading previous couchbase ... (1.8.1) Stopping previous couchbase ... (1.8.1)  * Failed to stop couchbase-server Saving previous couchbase config.dat ... Cleaning symlinks ... Unpacking replacement couchbase-server ... Setting up couchbase-server (2.0.0) ... Upgrading couchbase-server ...   /opt/couchbase/bin/install/cbupgrade -c /opt/couchbase/var/lib/couchbase/config -a yes Skipping cbupgrade due to INSTALL_DONT_AUTO_UPGRADE ... Skipping server start due to INSTALL_DONT_START_SERVER ... You have successfully installed Couchbase Server. Please browse to http://domU-12-31-39-09-A0-B2:8091/ to configure your server. Please refer to http://couchbase.com for additional resources. Please note that you have to update your firewall configuration to allow connections to the following ports: 11211, 11210, 11209, 4369, 8091 and from 21100 to 21299. By using this software you agree to the End User License Agreement. See /opt/couchbase/LICENSE.txt. Processing triggers for ureadahead ... root@domU-12-31-39-09-A0-B2 :~# cd /opt/couchbase/ root@domU-12-31-39-09-A0-B2 :/opt/couchbase# ls bin include licenses.tgz manifest.txt samples var etc lib LICENSE.txt manifest.xml share VERSION.txt root@domU-12-31-39-09-A0-B2 :/opt/couchbase# find var var var/log var/log/couchdb var/lib var/lib/couchbase var/lib/couchbase/data var/lib/couchbase/data/default-data var/lib/couchbase/data/default-data/default-0.mb var/lib/couchbase/data/default-data/default-3.mb var/lib/couchbase/data/default-data/default var/lib/couchbase/data/default-data/default-2.mb var/lib/couchbase/data/default-data/default-1.mb var/lib/couchbase/data/isasl.pw var/lib/couchbase/data/ns_log var/lib/couchbase/mnesia var/lib/couchbase/mnesia/stats_archiver-default-hour.DCL var/lib/couchbase/mnesia/stats_archiver-@system-minute.DCD var/lib/couchbase/mnesia/stats_archiver-@system-week.DCD var/lib/couchbase/mnesia/stats_archiver-default-month.DCD var/lib/couchbase/mnesia/stats_archiver-@system-minute.DCL var/lib/couchbase/mnesia/LATEST.LOG var/lib/couchbase/mnesia/stats_archiver-@system-week.DCL var/lib/couchbase/mnesia/stats_archiver-default-week.DCD var/lib/couchbase/mnesia/DECISION_TAB.LOG var/lib/couchbase/mnesia/local_config.DCD var/lib/couchbase/mnesia/stats_archiver-@system-day.DCD var/lib/couchbase/mnesia/stats_archiver-@system-day.DCL var/lib/couchbase/mnesia/stats_archiver-default-hour.DCD var/lib/couchbase/mnesia/stats_archiver-@system-year.DCD var/lib/couchbase/mnesia/stats_archiver-default-year.DCD var/lib/couchbase/mnesia/cluster.DCD var/lib/couchbase/mnesia/stats_archiver-@system-month.DCD var/lib/couchbase/mnesia/schema.DAT var/lib/couchbase/mnesia/stats_archiver-@system-hour.DCD var/lib/couchbase/mnesia/stats_archiver-default-minute.DCL var/lib/couchbase/mnesia/stats_archiver-default-minute.DCD var/lib/couchbase/mnesia/stats_archiver-default-day.DCD var/lib/couchbase/mnesia/stats_archiver-default-day.DCL var/lib/couchbase/mnesia/stats_archiver-@system-hour.DCL var/lib/couchbase/ip.debsave var/lib/couchbase/logs var/lib/couchbase/logs/info.idx var/lib/couchbase/logs/error.idx var/lib/couchbase/logs/error.siz var/lib/couchbase/logs/info.1 var/lib/couchbase/logs/debug.1 var/lib/couchbase/logs/debug.idx var/lib/couchbase/logs/info.siz var/lib/couchbase/logs/debug.siz var/lib/couchbase/logs/error.1 var/lib/couchbase/ip var/lib/couchbase/config var/lib/couchbase/config/config.dat.debsave var/lib/couchbase/config/config.dat var/lib/couchbase/tmp var/lib/moxi var/lib/couchdb var/run var/run/couchdb root@domU-12-31-39-09-A0-B2 :/opt/couchbase# ls -al var/lib/couchbase/data/default-data total 3352 drwxrwxr-x 2 couchbase couchbase 4096 Dec 1 02:41 . drwxrwxr-x 3 couchbase couchbase 4096 Dec 1 02:39 .. -rw-r--r-- 1 couchbase couchbase 52224 Dec 1 02:41 default -rw-r--r-- 1 couchbase couchbase 854016 Dec 1 02:41 default-0.mb -rw-r--r-- 1 couchbase couchbase 842752 Dec 1 02:41 default-1.mb -rw-r--r-- 1 couchbase couchbase 824320 Dec 1 02:41 default-2.mb -rw-r--r-- 1 couchbase couchbase 843776 Dec 1 02:41 default-3.mb

Steve Yen added a comment - 30/Nov/12 8:51 PM after running cbupgrade manually, running as root (note that it generates a bunch of *.mb-shm and *.mb-wal files owned by root)... root@domU-12-31-39-09-A0-B2 :/opt/couchbase# /opt/couchbase/bin/install/cbupgrade -c /opt/couchbase/var/lib/couchbase/config -a yes Automatic mode: running without interactive questions or confirmations. Upgrading your Couchbase Server to 2.0.0-1969-rel. The upgrade process might take awhile. Analysing... Previous config.dat file is /opt/couchbase/var/lib/couchbase/config/config.dat.debsave Target node: ns_1@127.0.0.1 Couchbase should not be running.   Please use: /etc/init.d/couchbase-server stop Database dir: /opt/couchbase/var/lib/couchbase/data Buckets to upgrade: default Checking disk space available for buckets in directory:   /opt/couchbase/var/lib/couchbase/data   Free disk bucket space wanted: 6834176.0   Free disk bucket space available: 6648143872   Free disk space factor: 2.0   Ok. Analysis complete. Copying /opt/couchbase/var/lib/couchbase/config/config.dat.debsave     cp /opt/couchbase/var/lib/couchbase/config/config.dat.debsave /opt/couchbase/bin/install/../../var/lib/couchbase/config/config.dat Ensuring bucket data directories. Ensuring bucket data directory: /opt/couchbase/var/lib/couchbase/data/default     mkdir -p /opt/couchbase/var/lib/couchbase/data/default Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data     chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data     chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data Upgrading buckets. Upgrading bucket: default     /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couchbase/var/lib/couchbase/data -b default --source-vbucket-state=active --destination-vbucket-state=active cbdbupgrade pid: 5789 cbdbupgrade err: 139 ERROR: upgrade failed for bucket: default root@domU-12-31-39-09-A0-B2 :/opt/couchbase# root@domU-12-31-39-09-A0-B2 :/opt/couchbase# root@domU-12-31-39-09-A0-B2 :/opt/couchbase# root@domU-12-31-39-09-A0-B2 :/opt/couchbase# root@domU-12-31-39-09-A0-B2 :/opt/couchbase# find var var var/log var/log/couchdb var/lib var/lib/couchbase var/lib/couchbase/data var/lib/couchbase/data/default-data var/lib/couchbase/data/default-data/default-0.mb var/lib/couchbase/data/default-data/default-wal var/lib/couchbase/data/default-data/default-3.mb-wal var/lib/couchbase/data/default-data/default-3.mb var/lib/couchbase/data/default-data/default-2.mb-wal var/lib/couchbase/data/default-data/default-2.mb-shm var/lib/couchbase/data/default-data/default var/lib/couchbase/data/default-data/default-1.mb-shm var/lib/couchbase/data/default-data/default-0.mb-wal var/lib/couchbase/data/default-data/default-2.mb var/lib/couchbase/data/default-data/default-0.mb-shm var/lib/couchbase/data/default-data/default-1.mb-wal var/lib/couchbase/data/default-data/default-1.mb var/lib/couchbase/data/default-data/default-3.mb-shm var/lib/couchbase/data/default-data/default-shm var/lib/couchbase/data/default var/lib/couchbase/data/default/1020.couch.1 var/lib/couchbase/data/isasl.pw var/lib/couchbase/data/ns_log var/lib/couchbase/mnesia var/lib/couchbase/mnesia/stats_archiver-default-hour.DCL var/lib/couchbase/mnesia/stats_archiver-@system-minute.DCD var/lib/couchbase/mnesia/stats_archiver-@system-week.DCD var/lib/couchbase/mnesia/stats_archiver-default-month.DCD var/lib/couchbase/mnesia/stats_archiver-@system-minute.DCL var/lib/couchbase/mnesia/LATEST.LOG var/lib/couchbase/mnesia/stats_archiver-@system-week.DCL var/lib/couchbase/mnesia/stats_archiver-default-week.DCD var/lib/couchbase/mnesia/DECISION_TAB.LOG var/lib/couchbase/mnesia/local_config.DCD var/lib/couchbase/mnesia/stats_archiver-@system-day.DCD var/lib/couchbase/mnesia/stats_archiver-@system-day.DCL var/lib/couchbase/mnesia/stats_archiver-default-hour.DCD var/lib/couchbase/mnesia/stats_archiver-@system-year.DCD var/lib/couchbase/mnesia/stats_archiver-default-year.DCD var/lib/couchbase/mnesia/cluster.DCD var/lib/couchbase/mnesia/stats_archiver-@system-month.DCD var/lib/couchbase/mnesia/schema.DAT var/lib/couchbase/mnesia/stats_archiver-@system-hour.DCD var/lib/couchbase/mnesia/stats_archiver-default-minute.DCL var/lib/couchbase/mnesia/stats_archiver-default-minute.DCD var/lib/couchbase/mnesia/stats_archiver-default-day.DCD var/lib/couchbase/mnesia/stats_archiver-default-day.DCL var/lib/couchbase/mnesia/stats_archiver-@system-hour.DCL var/lib/couchbase/ip.debsave var/lib/couchbase/logs var/lib/couchbase/logs/info.idx var/lib/couchbase/logs/error.idx var/lib/couchbase/logs/error.siz var/lib/couchbase/logs/info.1 var/lib/couchbase/logs/debug.1 var/lib/couchbase/logs/debug.idx var/lib/couchbase/logs/info.siz var/lib/couchbase/logs/debug.siz var/lib/couchbase/logs/error.1 var/lib/couchbase/ip var/lib/couchbase/config var/lib/couchbase/config/config.dat.debsave var/lib/couchbase/config/config.dat var/lib/couchbase/tmp var/lib/moxi var/lib/couchdb var/run var/run/couchdb root@domU-12-31-39-09-A0-B2 :/opt/couchbase# ls -al var/lib/couchbase/data total 24 drwxrwxr-x 4 couchbase couchbase 4096 Dec 1 02:48 . drwxr-xr-x 7 couchbase couchbase 4096 Dec 1 02:41 .. drwxr-xr-x 2 couchbase couchbase 4096 Dec 1 02:48 default drwxrwxr-x 2 couchbase couchbase 4096 Dec 1 02:48 default-data -rw-rw-r-- 1 couchbase couchbase 23 Dec 1 02:39 isasl.pw -rw-rw-r-- 1 couchbase couchbase 544 Dec 1 02:41 ns_log root@domU-12-31-39-09-A0-B2 :/opt/couchbase# ls -al var/lib/couchbase/data/default total 8 drwxr-xr-x 2 couchbase couchbase 4096 Dec 1 02:48 . drwxrwxr-x 4 couchbase couchbase 4096 Dec 1 02:48 .. -rw-r--r-- 1 root root 0 Dec 1 02:48 1020.couch.1 root@domU-12-31-39-09-A0-B2 :/opt/couchbase# ls -al var/lib/couchbase/data/default-data total 3372 drwxrwxr-x 2 couchbase couchbase 4096 Dec 1 02:48 . drwxrwxr-x 4 couchbase couchbase 4096 Dec 1 02:48 .. -rw-r--r-- 1 couchbase couchbase 52224 Dec 1 02:41 default -rw-r--r-- 1 couchbase couchbase 854016 Dec 1 02:41 default-0.mb -rw-r--r-- 1 root root 32768 Dec 1 02:48 default-0.mb-shm -rw-r--r-- 1 root root 0 Dec 1 02:48 default-0.mb-wal -rw-r--r-- 1 couchbase couchbase 842752 Dec 1 02:41 default-1.mb -rw-r--r-- 1 root root 32768 Dec 1 02:48 default-1.mb-shm -rw-r--r-- 1 root root 0 Dec 1 02:48 default-1.mb-wal -rw-r--r-- 1 couchbase couchbase 824320 Dec 1 02:41 default-2.mb -rw-r--r-- 1 root root 32768 Dec 1 02:48 default-2.mb-shm -rw-r--r-- 1 root root 0 Dec 1 02:48 default-2.mb-wal -rw-r--r-- 1 couchbase couchbase 843776 Dec 1 02:41 default-3.mb -rw-r--r-- 1 root root 32768 Dec 1 02:48 default-3.mb-shm -rw-r--r-- 1 root root 0 Dec 1 02:48 default-3.mb-wal -rw-r--r-- 1 root root 32768 Dec 1 02:48 default-shm -rw-r--r-- 1 root root 0 Dec 1 02:48 default-wal root@domU-12-31-39-09-A0-B2 :/opt/couchbase#

Steve Yen added a comment - 30/Nov/12 8:57 PM Aside... running the cbupgrade manually a 2nd time (as root) seemingly succeeds with no error messages, but the items are not there... # /opt/couchbase/bin/install/cbupgrade -c /opt/couchbase/var/lib/couchbase/config -a yes Automatic mode: running without interactive questions or confirmations. Upgrading your Couchbase Server to 2.0.0-1969-rel. The upgrade process might take awhile. Analysing... Previous config.dat file is /opt/couchbase/var/lib/couchbase/config/config.dat Target node: ns_1@127.0.0.1 Couchbase should not be running.   Please use: /etc/init.d/couchbase-server stop Database dir: /opt/couchbase/var/lib/couchbase/data Buckets to upgrade: default Checking disk space available for buckets in directory:   /opt/couchbase/var/lib/couchbase/data   Free disk bucket space wanted: 6834176.0   Free disk bucket space available: 6647779328   Free disk space factor: 2.0   Ok. Analysis complete. Copying /opt/couchbase/var/lib/couchbase/config/config.dat     cp /opt/couchbase/var/lib/couchbase/config/config.dat /opt/couchbase/bin/install/../../var/lib/couchbase/config/config.dat Ensuring bucket data directories. Ensuring bucket data directory: /opt/couchbase/var/lib/couchbase/data/default     mkdir -p /opt/couchbase/var/lib/couchbase/data/default Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data     chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data     chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data Upgrading buckets. Skipping already converted bucket: /opt/couchbase/var/lib/couchbase/data/default Done. After that, there are 1025 couchstore files, though, created in /opt/couchbase/var/lib/couchbase/data/default (vbucket files (123.couch.1) plus the one master.couch.1 file). And the default-data dir looks back to "normal" with the *.mb-shm and *.mb-wal files now gone... root@domU-12-31-39-09-A0-B2 :/opt/couchbase# ls -al var/lib/couchbase/data/default-data total 3352 drwxrwxr-x 2 couchbase couchbase 4096 Dec 1 02:51 . drwxrwxr-x 5 couchbase couchbase 4096 Dec 1 02:52 .. -rw-r--r-- 1 couchbase couchbase 52224 Dec 1 02:41 default -rw-r--r-- 1 couchbase couchbase 854016 Dec 1 02:41 default-0.mb -rw-r--r-- 1 couchbase couchbase 842752 Dec 1 02:41 default-1.mb -rw-r--r-- 1 couchbase couchbase 824320 Dec 1 02:41 default-2.mb -rw-r--r-- 1 couchbase couchbase 843776 Dec 1 02:41 default-3.mb root@domU-12-31-39-09-A0-B2 :/opt/couchbase#

Steve Yen added a comment - 30/Nov/12 9:01 PM At this point, running cbtransfer against the 1.8.1 sqlite files directly does emit items...    /opt/couchbase/bin/cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default stdout: -b default    /opt/couchbase/bin/cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default stdout: -b default --source-vbucket-state=active Both the above 2 commands emit lots of item output.

Steve Yen added a comment - 01/Dec/12 1:32 AM Saw this one time...      root@domU-12-31-39-09-A0-B2 :~# /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/defaultase/var/lib/couchbase/data -b default --source-vbucket-state=active --destination-vbucket-state=active python: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. Aborted (core dumped)

Steve Yen added a comment - 01/Dec/12 12:40 PM - edited (updated this comment) It's dying in various places with the first calls down to couchstore python-ctypes wrapper, depending so it seems like a general ctypes issue. After adding some print()'s... root@domU-12-31-39-09-A0-B2 :/opt/couchbase/lib# rm -rf /tmp/foo/default/ root@domU-12-31-39-09-A0-B2 :/opt/couchbase/lib# PYTHONPATH=/opt/couchbase/lib/python PATH=/opt/couchbase/lib:$PATH LD_LIBRARY_PATH=/opt/couchbase/lib:$LD_LIBRARY_PATH /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///tmp/foo -b default open_store(vbucket_id) 1020 Segmentation fault root@domU-12-31-39-09-A0-B2 :/opt/couchbase/lib# PYTHONPATH=/opt/couchbase/lib/python PATH=/opt/couchbase/lib:$PATH LD_LIBRARY_PATH=/opt/couchbase/lib:$LD_LIBRARY_PATH /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///tmp/foo -b default open_store(vbucket_id) 1020 open_store(vbucket_id) 1020 0 CouchStore(/tmp/foo/default/1020.couch.1) /tmp/foo/default/1020.couch.1 couchstore.DocumentInfo(str(key)) 745 python: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. couchstore.DocumentInfo(str(key)) 745 Aborted root@domU-12-31-39-09-A0-B2 :/opt/couchbase/lib# PYTHONPATH=/opt/couchbase/lib/python PATH=/opt/couchbase/lib:$PATH LD_LIBRARY_PATH=/opt/couchbase/lib:$LD_LIBRARY_PATH /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///tmp/foo -b default open_store(vbucket_id) 1020 open_store(vbucket_id) 1020 0 CouchStore(/tmp/foo/default/1020.couch.1) /tmp/foo/default/1020.couch.1 couchstore.DocumentInfo(str(key)) 745 python: malloc.c:2451: sYSMALLOc: Assertion `(old_top == (((mbinptr) (((char *) &((av)->bins[((1) - 1) * 2])) - __builtin_offsetof (struct malloc_chunk, fd)))) && old_size == 0) || ((unsigned long) (old_size) >= (unsigned long)((((__builtin_offsetof (struct malloc_chunk, fd_nextsize))+((2 * (sizeof(size_t))) - 1)) & ~((2 * (sizeof(size_t))) - 1))) && ((old_top)->size & 0x1) && ((unsigned long)old_end & pagemask) == 0)' failed. couchstore.DocumentInfo(str(key)) 745 Aborted

Steve Yen added a comment - 01/Dec/12 4:52 PM Dug into couchstore wrapper code and one suspect is that couchstore C api uses things like size_t (which may differs between 32-bit and 64-bit platforms) in its structures. But, the python wrapper always uses c_ulonglong (64-bits). Quickie fix attempt, though, is still crashing, so still looking.

Steve Yen added a comment - 01/Dec/12 7:26 PM Pushed up a fix to gerrit...     http://review.couchbase.org/#/c/22969/ Here's a capture of verifying that the upgrade works manually (using the INSTALL_DONT_AUTO_UPGRADE feature)... root@domU-12-31-39-09-A0-B2 :/opt/couchbase/lib/python# dpkg -i /tmp/couchbase-server-enterprise_x86_1.8.1.deb Setting up couchbase-server (1.8.1) ...  * Started couchbase-server You have successfully installed Couchbase Server. Please browse to http://domU-12-31-39-09-A0-B2:8091/ to configure your server. Please refer to http://couchbase.com for additional resources. Please note that you have to update your firewall configuration to allow connections to the following ports: 11211, 11210, 11209, 4369, 8091 and from 21100 to 21299. By using this software you agree to the End User License Agreement. See /opt/couchbase/LICENSE.txt. Processing triggers for ureadahead ... root@domU-12-31-39-09-A0-B2 :~# /opt/couchbase/bin/memcachetest -i 1000 Average with 1 threads Get operations:      #of ops. min max avg max90th max95th max99th          6716 102 us 70 ms 473 us 164 us 171 us 14 ms Set operations:      #of ops. min max avg max90th max95th max99th          3284 111 us1266874885 5617157 s 156 us 161 us 10 ms Usr: 0.104006 Sys: 0.680042 Tot: 5.033724 Server time: Usr: 0.000000 Sys: 1.000000 Total gets: 0 Total sets: 1000 root@domU-12-31-39-09-A0-B2 :~# INSTALL_DONT_AUTO_UPGRADE=1 INSTALL_DONT_START_SERVER=1 dpkg -i /tmp/couchbase-server-community_x86_2.0.0-1969-rel.deb (Reading database ... 39239 files and directories currently installed.) Preparing to replace couchbase-server 1.8.1 (using .../couchbase-server-community_x86_2.0.0-1969-rel.deb) ... =INFO REPORT==== 2-Dec-2012::00:59:33 === Initiated server shutdown** at node ns_1@127.0.0.1 ** =INFO REPORT==== 2-Dec-2012::00:59:40 === Stopped ns_server application** at node ns_1@127.0.0.1 **  * Stopped couchbase-server libssl0.9.8 is installed. Continue installing Upgrading previous couchbase ... (1.8.1) Stopping previous couchbase ... (1.8.1)  * Failed to stop couchbase-server Saving previous couchbase config.dat ... Cleaning symlinks ... Unpacking replacement couchbase-server ... Setting up couchbase-server (2.0.0) ... Upgrading couchbase-server ...   /opt/couchbase/bin/install/cbupgrade -c /opt/couchbase/var/lib/couchbase/config -a yes Skipping cbupgrade due to INSTALL_DONT_AUTO_UPGRADE ... Skipping server start due to INSTALL_DONT_START_SERVER ... You have successfully installed Couchbase Server. Please browse to http://domU-12-31-39-09-A0-B2:8091/ to configure your server. Please refer to http://couchbase.com for additional resources. Please note that you have to update your firewall configuration to allow connections to the following ports: 11211, 11210, 11209, 4369, 8091 and from 21100 to 21299. By using this software you agree to the End User License Agreement. See /opt/couchbase/LICENSE.txt. Processing triggers for ureadahead ... root@domU-12-31-39-09-A0-B2 :~# diff /tmp/couchstore.py /opt/couchbase/lib/python/couchstore.py 68c68 < _fields_ = [("buf", POINTER(c_char)), ("size", c_size_t)] --- > _fields_ = [("buf", POINTER(c_char)), ("size", c_ulonglong)] 93c93 < ("size", c_size_t) ] --- > ("size", c_ulonglong) ] 106c106 < ("header_position", c_size_t) ] --- > ("header_position", c_ulonglong) ] 226c226 < _check(_lib.couchstore_open_db(path, c_uint64(flags), byref(db))) --- > _check(_lib.couchstore_open_db(path, flags, byref(db))) root@domU-12-31-39-09-A0-B2 :~# cp /tmp/couchstore.py /opt/couchbase/lib/python/couchstore.py root@domU-12-31-39-09-A0-B2 :~# /opt/couchbase/bin/install/cbupgrade -c /opt/couchbase/var/lib/couchbase/config -a yes Automatic mode: running without interactive questions or confirmations. Upgrading your Couchbase Server to 2.0.0-1969-rel. The upgrade process might take awhile. Analysing... Previous config.dat file is /opt/couchbase/var/lib/couchbase/config/config.dat.debsave Target node: ns_1@127.0.0.1 Couchbase should not be running.   Please use: /etc/init.d/couchbase-server stop Database dir: /opt/couchbase/var/lib/couchbase/data Buckets to upgrade: default Checking disk space available for buckets in directory:   /opt/couchbase/var/lib/couchbase/data   Free disk bucket space wanted: 6834176.0   Free disk bucket space available: 6290337792   Free disk space factor: 2.0   Ok. Analysis complete. Copying /opt/couchbase/var/lib/couchbase/config/config.dat.debsave     cp /opt/couchbase/var/lib/couchbase/config/config.dat.debsave /opt/couchbase/bin/install/../../var/lib/couchbase/config/config.dat Ensuring bucket data directories. Ensuring bucket data directory: /opt/couchbase/var/lib/couchbase/data/default     mkdir -p /opt/couchbase/var/lib/couchbase/data/default Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data     chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data     chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data Upgrading buckets. Upgrading bucket: default     /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couchbase/var/lib/couchbase/data -b default --source-vbucket-state=active --destination-vbucket-state=active cbdbupgrade pid: 16221   [####################] 100.0% (1000/1000 msgs) bucket: default, msgs transferred...        : total | last | per sec  batch : 1025 | 1025 | 152.2  byte : 2098645 | 2098645 | 311698.6  msg : 1000 | 1000 | 148.5 done cbdbupgrade err: 0 Upgrading bucket: default     /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couchbase/var/lib/couchbase/data -b default --source-vbucket-state=replica --destination-vbucket-state=replica cbdbupgrade pid: 16230 .............................................................................................................................................................................................................. bucket: default, msgs transferred...        : total | last | per sec  batch : 1025 | 1025 | 1388.7  byte : 0 | 0 | 0.0  msg : 0 | 0 | 0.0 done cbdbupgrade err: 0 Upgrading bucket: default     /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couchbase/var/lib/couchbase/data -b default --source-vbucket-state=pending --destination-vbucket-state=pending cbdbupgrade pid: 16239 .............................................................................................................................................................................................................. bucket: default, msgs transferred...        : total | last | per sec  batch : 1025 | 1025 | 1408.2  byte : 0 | 0 | 0.0  msg : 0 | 0 | 0.0 done cbdbupgrade err: 0 Upgrading bucket: default     /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couchbase/var/lib/couchbase/data -b default --source-vbucket-state=dead --destination-vbucket-state=dead cbdbupgrade pid: 16248 .............................................................................................................................................................................................................. bucket: default, msgs transferred...        : total | last | per sec  batch : 1025 | 1025 | 1540.9  byte : 0 | 0 | 0.0  msg : 0 | 0 | 0.0 done cbdbupgrade err: 0 Ensuring bucket owner/group: /opt/couchbase/var/lib/couchbase/data/default     chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data/default Done. At this point the offline upgrade node running 2.0.0 has 1000 items.

Steve Yen added a comment - 01/Dec/12 7:27 PM Also, did the same (INSTALL_DONT_AUTO_UPGRADE) verification steps on 64-bit ubuntu and it worked.

Steve Yen added a comment - 01/Dec/12 7:43 PM (reopening - i forgot the policy is to not resolve until after it makes it past gerrit)