upgraded a 2 node 1.8.1 cluster to 2.0 and cbupgrade succeeded but buckets are in pending state after upgrade with errors in reading converted couch file

Farshid Ghods added a comment - 02/Dec/12 4:45 PM i see these errors in the error.* file ** Last message in was {'EXIT',<0.220.0>,killed} ** When Server state == {db,<0.230.0>,<0.231.0>,nil,<<"1354487511899126">>,                             <0.227.0>,<0.232.0>,                             {db_header,11,81,                                 <<0,0,0,2,129,51,0,0,0,0,10,161,0,0,0,0,81,0,0,                                   0,0,0,0,0,0,2,118,149>>,                                 <<0,0,0,2,140,23,0,0,0,0,10,186,0,0,0,0,81>>,                                 <<0,0,0,0,0,34,0,0,0,0,0,95>>,                                 0,nil,nil},                             81,                             {btree,<0.227.0>,                                 {164147,                                  <<0,0,0,0,81,0,0,0,0,0,0,0,0,2,118,149>>,                                  2721},                                 #Fun<couch_db_updater.7.89001503>,                                 #Fun<couch_db_updater.8.75953275>,                                 #Fun<couch_btree.5.72034400>,                                 #Fun<couch_db_updater.9.14108461>,1279,2558,                                 true},                             {btree,<0.227.0>,                                 {166935,<<0,0,0,0,81>>,2746},                                 #Fun<couch_db_updater.10.50603258>,                                 #Fun<couch_db_updater.11.85949495>,                                 #Fun<couch_db_updater.6.41937156>,                                 #Fun<couch_db_updater.12.107260449>,1279,2558,                                 true},                             {btree,<0.227.0>,                                 {34,<<>>,95},                                 #Fun<couch_btree.3.59827385>,                                 #Fun<couch_btree.4.7841881>,                                 #Fun<couch_btree.5.72034400>,nil,1279,2558,                                 true},                             81,<<"default/1">>,                             "/opt/couchbase/var/lib/couchbase/data/default/1.couch.1",                             [],nil,                             {user_ctx,null,[],undefined},                             nil,                             [before_header,after_header,on_file_open],                             []} ** Reason for termination ==

Farshid Ghods added a comment - 02/Dec/12 4:46 PM i zipped entire /opt/couchbase/var/lib since cbcolect_info gets stuck if memcached.log fie does not exist.

Farshid Ghods added a comment - 02/Dec/12 4:47 PM [ root@ip-10-226-91-111 ~]# rpm -U couchbase-server-community_x86_2.0.0-1971-rel.rpm Stopping couchbase-server ... Stopping couchbase-server =INFO REPORT==== 2-Dec-2012::17:29:13 === Initiated server shutdown** at node ns_1@10.226.91.111 ** =INFO REPORT==== 2-Dec-2012::17:29:20 === Stopped ns_server application** at node ns_1@10.226.91.111 ** Upgrading couchbase-server ...   /opt/couchbase/bin/install/cbupgrade -c /opt/couchbase/var/lib/couchbase/config -a yes Automatic mode: running without interactive questions or confirmations. Upgrading your Couchbase Server to 2.0.0-1971-rel. The upgrade process might take awhile. Analysing... Previous config.dat file is /opt/couchbase/var/lib/couchbase/config/config.dat Target node: ns_1@10.226.91.111 Couchbase should not be running.   Please use: /etc/init.d/couchbase-server stop Database dir: /opt/couchbase/var/lib/couchbase/data Buckets to upgrade: default,second Checking disk space available for buckets in directory:   /opt/couchbase/var/lib/couchbase/data   Free disk bucket space wanted: 815069184.0   Free disk bucket space available: 2642554880   Free disk space factor: 2.0   Ok. Analysis complete. Copying /opt/couchbase/var/lib/couchbase/config/config.dat     cp /opt/couchbase/var/lib/couchbase/config/config.dat /opt/couchbase/bin/install/../../var/lib/couchbase/config/config.dat Copying /opt/couchbase/var/lib/couchbase/ip     cp /opt/couchbase/var/lib/couchbase/ip /opt/couchbase/bin/install/../../var/lib/couchbase/ip Ensuring bucket data directories. Ensuring bucket data directory: /opt/couchbase/var/lib/couchbase/data/default     mkdir -p /opt/couchbase/var/lib/couchbase/data/default Ensuring bucket data directory: /opt/couchbase/var/lib/couchbase/data/second     mkdir -p /opt/couchbase/var/lib/couchbase/data/second Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data     chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data     chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data Upgrading buckets. Upgrading bucket: default     /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couchbase/var/lib/couchbase/data -b default --source-vbucket-state=active --destination-vbucket-state=active cbdbupgrade pid: 5556   [####################] 100.0% (43637/43637 msgs) bucket: default, msgs transferred...        : total | last | per sec  batch : 1025 | 1025 | 43.7  byte : 79347311 | 79347311 | 3384811.7  msg : 43637 | 43637 | 1861.5 done cbdbupgrade err: 0 Upgrading bucket: default     /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couchbase/var/lib/couchbase/data -b default --source-vbucket-state=replica --destination-vbucket-state=replica cbdbupgrade pid: 5569   [####################] 100.0% (43636/43636 msgs) bucket: default, msgs transferred...        : total | last | per sec  batch : 1025 | 1025 | 42.1  byte : 79617887 | 79617887 | 3271665.1  msg : 43636 | 43636 | 1793.1 done cbdbupgrade err: 0 Upgrading bucket: default     /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couchbase/var/lib/couchbase/data -b default --source-vbucket-state=pending --destination-vbucket-state=pending cbdbupgrade pid: 5579 .............................................................................................................................................................................................................. bucket: default, msgs transferred...        : total | last | per sec  batch : 1025 | 1025 | 494.9  byte : 0 | 0 | 0.0  msg : 0 | 0 | 0.0 done cbdbupgrade err: 0 Upgrading bucket: default     /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couchbase/var/lib/couchbase/data -b default --source-vbucket-state=dead --destination-vbucket-state=dead cbdbupgrade pid: 5588 .............................................................................................................................................................................................................. bucket: default, msgs transferred...        : total | last | per sec  batch : 1025 | 1025 | 479.6  byte : 0 | 0 | 0.0  msg : 0 | 0 | 0.0 done cbdbupgrade err: 0 Ensuring bucket owner/group: /opt/couchbase/var/lib/couchbase/data/default     chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data/default Upgrading bucket: second     /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/second-data/second couchstore-files:///opt/couchbase/var/lib/couchbase/data -b second --source-vbucket-state=active --destination-vbucket-state=active cbdbupgrade pid: 5598   [####################] 100.0% (55000/55000 msgs) bucket: second, msgs transferred...        : total | last | per sec  batch : 1025 | 1025 | 35.1  byte : 102870970 | 102870970 | 3517741.1  msg : 55000 | 55000 | 1880.8 done cbdbupgrade err: 0 Upgrading bucket: second     /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/second-data/second couchstore-files:///opt/couchbase/var/lib/couchbase/data -b second --source-vbucket-state=replica --destination-vbucket-state=replica cbdbupgrade pid: 5608   [####################] 100.0% (55000/55000 msgs) bucket: second, msgs transferred...        : total | last | per sec  batch : 1025 | 1025 | 35.7  byte : 102547276 | 102547276 | 3572509.8  msg : 55000 | 55000 | 1916.1 done cbdbupgrade err: 0 Upgrading bucket: second     /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/second-data/second couchstore-files:///opt/couchbase/var/lib/couchbase/data -b second --source-vbucket-state=pending --destination-vbucket-state=pending cbdbupgrade pid: 5618 .............................................................................................................................................................................................................. bucket: second, msgs transferred...        : total | last | per sec  batch : 1025 | 1025 | 493.0  byte : 0 | 0 | 0.0  msg : 0 | 0 | 0.0 done cbdbupgrade err: 0 Upgrading bucket: second     /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/second-data/second couchstore-files:///opt/couchbase/var/lib/couchbase/data -b second --source-vbucket-state=dead --destination-vbucket-state=dead cbdbupgrade pid: 5627 .............................................................................................................................................................................................................. bucket: second, msgs transferred...        : total | last | per sec  batch : 1025 | 1025 | 517.4  byte : 0 | 0 | 0.0  msg : 0 | 0 | 0.0 done cbdbupgrade err: 0 Ensuring bucket owner/group: /opt/couchbase/var/lib/couchbase/data/second     chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data/second Done. Starting couchbase-server[ OK ] You have successfully installed Couchbase Server. Please browse to http://ip-10-226-91-111:8091/ to configure your server. Please refer to http://couchbase.com for additional resources. Please note that you have to update your firewall configuration to allow connections to the following ports: 11211, 11210, 11209, 4369, 8091 and from 21100 to 21299. By using this software you agree to the End User License Agreement. See /opt/couchbase/LICENSE.txt.

Farshid Ghods added a comment - 02/Dec/12 9:36 PM i just reran same scenario on 64-bit and did not see this issue.

Farshid Ghods added a comment - 02/Dec/12 9:37 PM [ root@ip-10-202-103-209 testrunner]# uname -m x86_64 [ root@ip-10-202-103-209 testrunner]# cat /etc/issue CentOS release 5.4 (Final) Kernel \r on an \m [ root@ip-10-202-103-209 testrunner]# uname -a Linux ip-10-202-103-209 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:34:28 EST 2008 x86_64 x86_64 x86_64 GNU/Linux

Steve Yen added a comment - 02/Dec/12 10:01 PM logging onto node: root@ec2-23-22-192-207.compute-1.amazonaws.com I see that the _local/vbstate docs have the vbuckets in 'dead' state...     {"state": "dead", "checkpoint_id": "0", "max_deleted_seqno": "0"} This is the case for the both the default bucket and the second bucket.

Steve Yen added a comment - 03/Dec/12 9:35 PM http://review.couchbase.org/23022

Steve Yen added a comment - 03/Dec/12 11:46 PM Did a verification on centos, 32-bit with that proposed fix... [ root@ip-10-144-64-55 ~]# /opt/couchbase/bin/memcachetest -h 127.0.0.1 -i 5000 [ root@ip-10-144-64-55 ~]# /etc/init.d/couchbase-server stop Stopping couchbase-server =INFO REPORT==== 4-Dec-2012::00:39:14 === Initiated server shutdown** at node ns_1@10.144.64.55 ** =INFO REPORT==== 4-Dec-2012::00:39:20 === Stopped ns_server application** at node ns_1@10.144.64.55 ** [ root@ip-10-144-64-55 ~]# INSTALL_DONT_AUTO_UPGRADE=1 INSTALL_DONT_START_SERVER=1 rpm -U couchbase-server-enterprise_x86_2.0.0-1971-rel.rpm Stopping couchbase-server ... Stopping couchbase-server Upgrading couchbase-server ...   /opt/couchbase/bin/install/cbupgrade -c /opt/couchbase/var/lib/couchbase/config -a yes Skipping cbupgrade due to INSTALL_DONT_AUTO_UPGRADE ... Skipping server start due to INSTALL_DONT_START_SERVER ... You have successfully installed Couchbase Server. Please browse to http://ip-10-144-64-55:8091/ to configure your server. Please refer to http://couchbase.com for additional resources. Please note that you have to update your firewall configuration to allow connections to the following ports: 11211, 11210, 11209, 4369, 8091 and from 21100 to 21299. By using this software you agree to the End User License Agreement. See /opt/couchbase/LICENSE.txt. Stopping couchbase-server [ root@ip-10-144-64-55 ~]# diff /tmp/pump_mbf.py /opt/couchbase/lib/python/pump_mbf.py 92c92 < for db_file in [f for f in db_files]: --- > for db_file in [f for f in db_files if f.endswith(".mb")]: 98c98 < state = str(row[2]) --- > state = row[2] [ root@ip-10-144-64-55 ~]# cp /tmp/pump_mbf.py /opt/couchbase/lib/python/pump_mbf.py [ root@ip-10-144-64-55 ~]# /opt/couchbase/bin/install/cbupgrade -c /opt/couchbase/var/lib/couchbase/config -a yes Automatic mode: running without interactive questions or confirmations. Upgrading your Couchbase Server to 2.0.0-1971-rel. The upgrade process might take awhile. Analysing... Previous config.dat file is /opt/couchbase/var/lib/couchbase/config/config.dat Target node: ns_1@10.144.64.55 Couchbase should not be running.   Please use: /etc/init.d/couchbase-server stop Database dir: /opt/couchbase/var/lib/couchbase/data Buckets to upgrade: default Checking disk space available for buckets in directory:   /opt/couchbase/var/lib/couchbase/data   Free disk bucket space wanted: 25542656.0   Free disk bucket space available: 21539762176   Free disk space factor: 2.0   Ok. Analysis complete. Copying /opt/couchbase/var/lib/couchbase/config/config.dat     cp /opt/couchbase/var/lib/couchbase/config/config.dat /opt/couchbase/bin/install/../../var/lib/couchbase/config/config.dat Copying /opt/couchbase/var/lib/couchbase/ip     cp /opt/couchbase/var/lib/couchbase/ip /opt/couchbase/bin/install/../../var/lib/couchbase/ip Ensuring bucket data directories. Ensuring bucket data directory: /opt/couchbase/var/lib/couchbase/data/default     mkdir -p /opt/couchbase/var/lib/couchbase/data/default Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data     chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data     chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data Upgrading buckets. Upgrading bucket: default     /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couchbase/var/lib/couchbase/data -b default --source-vbucket-state=active --destination-vbucket-state=active cbdbupgrade pid: 7822 .............................................................................................................................................................................................................. bucket: default, msgs transferred...        : total | last | per sec  batch : 1025 | 1025 | 130.8  byte : 5190453 | 5190453 | 662547.6  msg : 2500 | 2500 | 319.1 done cbdbupgrade err: 0 Upgrading bucket: default     /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couchbase/var/lib/couchbase/data -b default --source-vbucket-state=replica --destination-vbucket-state=replica cbdbupgrade pid: 7832 .............................................................................................................................................................................................................. bucket: default, msgs transferred...        : total | last | per sec  batch : 1025 | 1025 | 124.3  byte : 5176669 | 5176669 | 627583.7  msg : 2500 | 2500 | 303.1 done cbdbupgrade err: 0 Upgrading bucket: default     /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couchbase/var/lib/couchbase/data -b default --source-vbucket-state=pending --destination-vbucket-state=pending cbdbupgrade pid: 7841 .............................................................................................................................................................................................................. bucket: default, msgs transferred...        : total | last | per sec  batch : 1025 | 1025 | 1208.4  byte : 0 | 0 | 0.0  msg : 0 | 0 | 0.0 done cbdbupgrade err: 0 Upgrading bucket: default     /opt/couchbase/bin/install/../cbtransfer /opt/couchbase/var/lib/couchbase/data/default-data/default couchstore-files:///opt/couchbase/var/lib/couchbase/data -b default --source-vbucket-state=dead --destination-vbucket-state=dead cbdbupgrade pid: 7850 .............................................................................................................................................................................................................. bucket: default, msgs transferred...        : total | last | per sec  batch : 1025 | 1025 | 1117.7  byte : 0 | 0 | 0.0  msg : 0 | 0 | 0.0 done cbdbupgrade err: 0 Ensuring bucket owner/group: /opt/couchbase/var/lib/couchbase/data/default     chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data/default Done. [ root@ip-10-144-64-55 ~]# /etc/init.d/couchbase-server start Starting couchbase-server [ OK ] [ root@ip-10-144-64-55 ~]# After starting the 2nd node, the data got converted, but the cluster looks like it still has both nodes as pending.

Steve Yen added a comment - 04/Dec/12 12:09 AM More info on the cbupgrade fix verification... After upgrading data (with the fix), but before starting the servers, the _local/vbstate metadata seems clean. On one node, vbucket 0 is active... [ root@ip-10-144-64-55 ~]# PYTHONPATH=/opt/couchbase/lib/python LD_LIBRARY_PATH=/opt/couchbase/lib python Python 2.4.3 (#1, Jun 18 2012, 08:55:31) [GCC 4.1.2 20080704 (Red Hat 4.1.2-52)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import couchstore >>> c = couchstore.CouchStore("/opt/couchbase/var/lib/couchbase/data/default/0.couch.1", 'r') >>> c.localDocs['_local/vbstate'] '{"state": "active", "max_deleted_seqno": "0", "checkpoint_id": "0"}' >>> On the other node, vbucket 0 is replica... [ root@ip-10-226-91-111 ~]# PYTHONPATH=/opt/couchbase/lib/python LD_LIBRARY_PATH=/opt/couchbase/lib python Python 2.6.6 (r266:84292, May 1 2012, 13:52:42) [GCC 4.4.6 20110731 (Red Hat 4.4.6-3)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import couchstore >>> c = couchstore.CouchStore("/opt/couchbase/var/lib/couchbase/data/default/0.couch.1", 'r') >>> c.localDocs['_local/vbstate'] '{"state": "replica", "max_deleted_seqno": "0", "checkpoint_id": "0"}' >>> At this point, after starting both nodes, the nodes are in PEND state and the log in the U/I shows this event: "Changed cluster compat mode from undefined to [2,0]"

Steve Yen added a comment - 04/Dec/12 11:34 AM double-checked the 10.1.4.10 node (with 1.8.1 installed) that Farshid provided. The vbucket_states table is only in the master db file, not in the shard files.

Steve Yen added a comment - 04/Dec/12 2:08 PM Alk, I'm blocked by this, where nodes are coming up pending. Steve

Steve Yen added a comment - 05/Dec/12 2:35 PM alk need logs / cbcollect_info

Steve Yen added a comment - 05/Dec/12 7:20 PM - edited just a warning msg that cbupgrade could take awhile, asked by Farshid: http://review.couchbase.org/23112 (update - ignore this comment - that fix 23112 was for a different bug)

Steve Yen added a comment - 05/Dec/12 8:52 PM I've attached one cbcollect-info output for one node, where I had to create an empty memcached.log to let it succeed. The other node's cbcollect-info hung try to get cbstats... couchbase config ('/opt/couchbase/bin'/escript '/opt/couchbase/bin'/cbdump-config '/opt/couchbase/var/lib/couchbase/config/config.dat') - OK couchbase logs (debug) (cbbrowse_logs) - OK couchbase logs (info) (cbbrowse_logs info) - OK couchbase logs (error) (cbbrowse_logs error) - OK couchbase logs (couchdb) (cbbrowse_logs couchdb) - OK couchbase logs (xdcr) (cbbrowse_logs xdcr) - OK couchbase logs (xdcr_errors) (cbbrowse_logs xdcr_errors) - OK couchbase logs (views) (cbbrowse_logs views) - OK couchbase logs (mapreduce errors) (cbbrowse_logs mapreduce_errors) - OK couchbase logs (stats) (cbbrowse_logs stats) - OK memcached stats all (cbstats -a 127.0.0.1:11210 all -b _admin -p _admin) -

Aleksey Kondratenko added a comment - 06/Dec/12 2:17 PM I'd need full cluster logs. Other node is master, particularly. Tried upgrading .rpm myself with two nodes and it worked fine.