http://www.couchbase.com/issues This file is an XML representation of an issue en-us 5.2.4 845 26-12-2012 http://www.couchbase.com/issues/browse/MB-7341 Couchbase Server Have a 4 nodes cluster (<a href="http://172.23.96.12:8091">http://172.23.96.12:8091</a>). 1 default buckets 40G RAM on each node for the bucket. 200M items (920 G) in bucket. Resident is as low as 8.3%. <br/> Try to do an offline upgrade from 181 to 2.0 build 1971: <br/> <br/> [<a href='mailto:root@thor05'>root@thor05</a> ~]# rpm -U couchbase-server-enterprise_x86_64_2.0.0-1971-rel.rpm <br/> Stopping couchbase-server ... <br/> Stopping couchbase-server <br/> Upgrading couchbase-server ... <br/> &nbsp;&nbsp;/opt/couchbase/bin/install/cbupgrade -c /opt/couchbase/var/lib/couchbase/config -a yes <br/> Automatic mode: running without interactive questions or confirmations. <br/> Upgrading your Couchbase Server to 2.0.0-1971-rel. <br/> The upgrade process might take awhile. <br/> Analysing... <br/> Previous config.dat file is /opt/couchbase/var/lib/couchbase/config/config.dat <br/> Target node: <a href='mailto:ns_1@172.23.96.14'>ns_1@172.23.96.14</a> <br/> <br/> Couchbase should not be running. <br/> &nbsp;&nbsp;Please use: /etc/init.d/couchbase-server stop <br/> <br/> Database dir: /data2 <br/> <br/> Buckets to upgrade: default <br/> <br/> Checking disk space available for buckets in directory: <br/> &nbsp;&nbsp;/data2 <br/> &nbsp;&nbsp;Free disk bucket space wanted: 494982651904.0 <br/> &nbsp;&nbsp;Free disk bucket space available: 570048225280 <br/> &nbsp;&nbsp;Free disk space factor: 2.0 <br/> &nbsp;&nbsp;Ok. <br/> <br/> Analysis complete. <br/> <br/> Copying /opt/couchbase/var/lib/couchbase/config/config.dat <br/> &nbsp;&nbsp;&nbsp;&nbsp;cp /opt/couchbase/var/lib/couchbase/config/config.dat /opt/couchbase/bin/install/../../var/lib/couchbase/config/config.dat <br/> Copying /opt/couchbase/var/lib/couchbase/ip <br/> &nbsp;&nbsp;&nbsp;&nbsp;cp /opt/couchbase/var/lib/couchbase/ip /opt/couchbase/bin/install/../../var/lib/couchbase/ip <br/> Ensuring bucket data directories. <br/> Ensuring bucket data directory: /data2/default <br/> &nbsp;&nbsp;&nbsp;&nbsp;mkdir -p /data2/default <br/> Ensuring dbdir owner/group: /data2 <br/> &nbsp;&nbsp;&nbsp;&nbsp;chown -R couchbase:couchbase /data2 <br/> Ensuring dbdir owner/group: /opt/couchbase/var/lib/couchbase/data <br/> &nbsp;&nbsp;&nbsp;&nbsp;chown -R couchbase:couchbase /opt/couchbase/var/lib/couchbase/data <br/> Upgrading buckets. <br/> Upgrading bucket: default <br/> &nbsp;&nbsp;&nbsp;&nbsp;/opt/couchbase/bin/install/../cbtransfer /data2/default-data/default <a href="couchstore-files:///data2">couchstore-files:///data2</a> -b default --source-vbucket-state=active --destination-vbucket-state=active <br/> cbdbupgrade pid: 14416 <br/> &nbsp;&nbsp;[ ] 0.0% (0/49997137 msgs) <br/> <br/> <br/> <br/> Offline upgrade hangs Linux version 2.6.18-308.el5 (<a href='mailto:mockbuild@builder10.centos.org'>mockbuild@builder10.centos.org</a>) (gcc version 4.1.2 20080704 (Red Hat 4.1.2-52)) MB-7341

[2.0-hotfix candidate] Offline upgrade hangs at cbtransfer converting sqlite file to couchstorefile

Bug Blocker Resolved Fixed Public Bin Cui Chisheng Hong 2.0-hotfix 2.0-release-notes Mon, 3 Dec 2012 20:39:51 -0600 Mon, 28 Jan 2013 23:44:40 -0600 Tue, 18 Dec 2012 16:45:05 -0600 2.0 2.0.1 installer 0 0 moved to 2.0.1 per bug-scrub. <br/> <br/> And, also recommend users do online upgrade, especially on huge datasizes and highly fragmented sqlite. <a href="http://review.couchbase.org/#/c/23073/">http://review.couchbase.org/#/c/23073/</a> <br/> <br/> Further observation: <br/> 1. cbtransfter was not stucked but just slow to retrieve items from sqlite files. Since cbtransfer will retrieve items vbucket by vbucket and in reverse vbucket id order, and all vbucket 512 to 937 are all replica vbuckets, it will retrieve none of active items. And it is quite time consuming to execute sql statement against fragmented sqlite files. <br/> 2. We can optimize sqlite access process to simply skip the whole kv table based on vbucket id and vbucket state. If we want to retrieve active items, we can easily jump to the first table that contains active items. <br/> <br/> <a href="http://review.couchbase.org/#/c/23073/">http://review.couchbase.org/#/c/23073/</a> <br/> <br/> Further observation: <br/> 1. cbtransfter was not stucked but just slow to retrieve items from sqlite files. Since cbtransfer will retrieve items vbucket by vbucket and in reverse vbucket id order, and all vbucket 512 to 937 are all replica vbuckets, it will retrieve none of active items. And it is quite time consuming to execute sql statement against fragmented sqlite files. <br/> 2. We can optimize sqlite access process to simply skip the whole kv table based on vbucket id and vbucket state. If we want to retrieve active items, we can easily jump to the first table that contains active items. <br/> <br/> Added to RN as: Performing an upgrade from Couchbase 1.8.1 to Couchbase 2.0.0 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;GA build freezes while transfering data from SQLite files <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;to Couchstore files using &lt;command&gt;cbtransfer&lt;/command&gt;. Hmmm, looks like I put the wrong commit msg on this change, but it went in anyways... <br/> <br/> &nbsp;&nbsp;just a warning msg that cbupgrade could take awhile, asked by Farshid: <a href="http://review.couchbase.org/23112">http://review.couchbase.org/23112</a> One one setup to upgrade 50m items, it takes: <br/> <br/> [<a href='mailto:root@thor08'>root@thor08</a> data]# time /opt/couchbase/bin/install/../cbtransfer /data/thor05/default-data/default <a href="couchstore-files:///data/test">couchstore-files:///data/test</a> <br/> da default --source-vbucket-state=active --destination-vbucket-state=active <br/> &nbsp;&nbsp;[####### ] 33.6% (16797698/49997137 msgs)) <br/> &nbsp;&nbsp;[####################] 100.0% (49997137/49997137 msgs) <br/> bucket: default, msgs transferred... <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;: total | last | per sec <br/> &nbsp;batch : 294023 | 294023 | 9.9 <br/> &nbsp;byte : 117679410715 | 117679410715 | 3971062.9 <br/> &nbsp;msg : 49997137 | 49997137 | 1687.1 <br/> done <br/> <br/> real 493m54.341s <br/> user 177m43.942s <br/> sys 30m50.346s <br/> Attached hotfix zipfile: couchbase-server-2.0.0-HOTFIX-<a href="http://www.couchbase.com/issues/browse/MB-7341" title="[2.0-hotfix candidate] Offline upgrade hangs at cbtransfer converting sqlite file to couchstorefile"><strike>MB-7341</strike></a>.zip Duplicate Date of First Response Tue, 4 Dec 2012 13:59:47 -0600 Rank 3468 Time In Status