[RN 2.0.2] 2.0.0 to 2.0.1 upgrade didn't replace couchdb's file2.beam with latest version (was: [centos 32] views are broken after upgrade 2.0.0->2.0.1)

Filipe Manana added a comment - 19/Feb/13 11:14 AM This is an installer issue. The file2 module in 2.0.0 doesn't have the function ensure_dir/1, it was added in 2.0.1. This means after upgrade, a file2.beam from 2.0.0 is still being used.

Farshid Ghods added a comment - 19/Feb/13 11:22 AM Andrei, can you compare such file between fresh 2.0.1 and 2.0 installation to rule out that this could be a buildbot issues also wonder if this is reproducible with 64-bit

Andrei Baranouski added a comment - 20/Feb/13 8:15 AM As I see file2.beam was replaced after upgrade and file_sorter_2.beam was added file2.beam from 2.0.0_1976 build [ root@localhost tmp]# ls -la total 366172 drwxrwxrwt 3 root root 4096 Feb 20 05:45 . drwxr-xr-x 23 root root 4096 Jan 20 20:43 .. -rwxr-xr-x 1 root root 2608 Feb 20 05:45 file2.beam_1976 after upgrade on 2.0.1_161 [ root@localhost tmp]# cp /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-8e96188-git/ebin/file file2.beam_161 file2.beam file_sorter_2.beam [ root@localhost tmp]# cd /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-8e96188-git/ebin/ [ root@localhost ebin]# ls -la total 1124 drwxr-xr-x 2 bin bin 4096 Feb 20 05:59 . drwxr-xr-x 5 bin bin 4096 Feb 20 05:59 .. -rwxr-xr-x 1 bin bin 21484 Feb 18 23:16 couch_api_wrap.beam -rwxr-xr-x 1 bin bin 15872 Feb 18 23:16 couch_api_wrap_httpc.beam -rwxr-xr-x 1 bin bin 1863 Feb 18 23:16 couch.app -rwxr-xr-x 1 bin bin 5752 Feb 18 23:16 couch_app.beam -rwxr-xr-x 1 bin bin 19820 Feb 18 23:16 couch_auth_cache.beam -rwxr-xr-x 1 bin bin 1664 Feb 18 23:16 couch.beam -rwxr-xr-x 1 bin bin 47376 Feb 18 23:16 couch_btree.beam -rwxr-xr-x 1 bin bin 16676 Feb 18 23:16 couch_btree_copy.beam -rwxr-xr-x 1 bin bin 10932 Feb 18 23:16 couch_btree_stats.beam -rwxr-xr-x 1 bin bin 22072 Feb 18 23:16 couch_changes.beam -rwxr-xr-x 1 bin bin 21180 Feb 18 23:16 couch_compaction_daemon.beam -rwxr-xr-x 1 bin bin 3764 Feb 18 23:16 couch_compress.beam -rwxr-xr-x 1 bin bin 12844 Feb 18 23:16 couch_config.beam -rwxr-xr-x 1 bin bin 3444 Feb 18 23:16 couch_config_writer.beam -rwxr-xr-x 1 bin bin 32724 Feb 18 23:16 couch_db.beam -rwxr-xr-x 1 bin bin 9184 Feb 18 23:16 couch_db_consistency_check.beam -rwxr-xr-x 1 bin bin 8480 Feb 18 23:16 couch_db_frontend.beam -rwxr-xr-x 1 bin bin 5564 Feb 18 23:16 couch_db_update_notifier.beam -rwxr-xr-x 1 bin bin 2324 Feb 18 23:16 couch_db_update_notifier_sup.beam -rwxr-xr-x 1 bin bin 47568 Feb 18 23:16 couch_db_updater.beam -rwxr-xr-x 1 bin bin 14628 Feb 18 23:16 couch_doc.beam -rwxr-xr-x 1 bin bin 5248 Feb 18 23:16 couch_drv.beam -rwxr-xr-x 1 bin bin 4536 Feb 18 23:16 couch_ejson_compare.beam -rwxr-xr-x 1 bin bin 4776 Feb 18 23:16 couch_event_sup.beam -rwxr-xr-x 1 bin bin 27876 Feb 18 23:16 couch_file.beam -rwxr-xr-x 1 bin bin 7056 Feb 18 23:16 couch_file_write_guard.beam -rwxr-xr-x 1 bin bin 18224 Feb 18 23:16 couch_httpd_auth.beam -rwxr-xr-x 1 bin bin 40444 Feb 18 23:16 couch_httpd.beam -rwxr-xr-x 1 bin bin 32648 Feb 18 23:16 couch_httpd_db.beam -rwxr-xr-x 1 bin bin 10408 Feb 18 23:16 couch_httpd_external.beam -rwxr-xr-x 1 bin bin 13700 Feb 18 23:16 couch_httpd_misc_handlers.beam -rwxr-xr-x 1 bin bin 10820 Feb 18 23:16 couch_httpd_oauth.beam -rwxr-xr-x 1 bin bin 5152 Feb 18 23:16 couch_httpd_replicator.beam -rwxr-xr-x 1 bin bin 35132 Feb 18 23:16 couch_httpd_view.beam -rwxr-xr-x 1 bin bin 7848 Feb 18 23:16 couch_log.beam -rwxr-xr-x 1 bin bin 20376 Feb 18 23:16 couch_native_process.beam -rwxr-xr-x 1 bin bin 13636 Feb 18 23:16 couch_os_process.beam -rwxr-xr-x 1 bin bin 2432 Feb 18 23:16 couch_primary_sup.beam -rwxr-xr-x 1 bin bin 32032 Feb 18 23:16 couch_query_servers.beam -rwxr-xr-x 1 bin bin 4872 Feb 18 23:16 couch_ref_counter.beam -rwxr-xr-x 1 bin bin 28464 Feb 18 23:16 couch_replication_manager.beam -rwxr-xr-x 1 bin bin 4676 Feb 18 23:16 couch_replication_notifier.beam -rwxr-xr-x 1 bin bin 41112 Feb 18 23:16 couch_replicator.beam -rwxr-xr-x 1 bin bin 22512 Feb 18 23:16 couch_replicator_utils.beam -rwxr-xr-x 1 bin bin 18868 Feb 18 23:16 couch_replicator_worker.beam -rwxr-xr-x 1 bin bin 3828 Feb 18 23:16 couch_rep_sup.beam -rwxr-xr-x 1 bin bin 2148 Feb 18 23:16 couch_secondary_sup.beam -rwxr-xr-x 1 bin bin 16904 Feb 18 23:16 couch_server.beam -rwxr-xr-x 1 bin bin 8948 Feb 18 23:16 couch_server_sup.beam -rwxr-xr-x 1 bin bin 8472 Feb 18 23:16 couch_task_status.beam -rwxr-xr-x 1 bin bin 20384 Feb 18 23:16 couch_util.beam -rwxr-xr-x 1 bin bin 6968 Feb 18 23:16 couch_uuids.beam -rwxr-xr-x 1 bin bin 25104 Feb 18 23:16 couch_view.beam -rwxr-xr-x 1 bin bin 9528 Feb 18 23:16 couch_view_compactor.beam -rwxr-xr-x 1 bin bin 33088 Feb 18 23:16 couch_view_group.beam -rwxr-xr-x 1 bin bin 17084 Feb 18 23:16 couch_view_mapreduce.beam -rwxr-xr-x 1 bin bin 18972 Feb 18 23:16 couch_view_updater.beam -rwxr-xr-x 1 bin bin 9664 Feb 18 23:16 couch_work_queue.beam -rwxr-xr-x 1 bin bin 4280 Feb 18 23:16 erl_diag.beam -rwxr-xr-x 1 bin bin 2948 Feb 18 23:16 file2.beam -rwxr-xr-x 1 bin bin 66240 Feb 18 23:16 file_sorter_2.beam -rwxr-xr-x 1 bin bin 14876 Feb 18 23:16 json_stream_parse.beam test to reproduce: python testrunner -i centos-32-2.0-upgrade.ini -t newupgradetests.MultiNodesUpgradeTests.offline_cluster_upgrade,initial_version=2.0.0-1976-rel,nodes_init=3,ddocs-num=3,upgrade_version=2.0.1-161-rel it is reproducible only with centos 32 failed against 161 build as well http://qa.hq.northscale.net/view/2.0.1/job/centos-32-2.0-upgrade/33/consoleFull

Andrei Baranouski added a comment - 20/Feb/13 8:19 AM Filipe, could you add thoughts to my comments and assign to Steve, if that's installer problem Thanks

Filipe Manana added a comment - 20/Feb/13 8:34 AM I don't understand your analysis here. Erlang VM loads files ending in .beam, so I don't get why the instaler/upgrader or whatever are producing files ending with .beam_xxx. From the last big ls output shown, I don't see how you conclude it's the right or wrong file2.beam module. I would grab file2.beam from 2.0.0, calculate it's md5 (with md5sum) for example, do the upgrade, then check after the upgrade that there's only one subdirectory with a file2.beam and that that file2.beam as a different md5 checksum. In case of multiple file2.beam (in different subdirectories), the Erlang VM can pick the wrong file (from 2.0.0). The Erlang stack trace pasted before clearly (for anyone familiar with Erlang) shows that the loaded file2.beam module doesn't have the function ensure_dir/1 - that means the VM loaded a module version from 2.0.0: error: undef stacktrace: [{file2,ensure_dir,                     ["/opt/couchbase/var/lib/couchbase/data/@indexes/default/tmp_d3bab589b07b8f398f39483b2e415d07_main/e9e434b58f3fac3310504249a00254f9.sort"]},              {couch_set_view_util,do_new_sort_file_path,2},              {couch_set_view_updater,'-maybe_flush_merge_buffers/2-fun-1-',7},              {lists,foldl,3},              {couch_set_view_updater,flush_writes,1},              {couch_set_view_updater,'-update/8-fun-1-',15}]

Andrei Baranouski added a comment - 20/Feb/13 8:56 AM I used format .beam_xxx to save beam file before upgrade and then compare with newer version. and their size are different okay, with md5sum before upgrade( build 1976) [ root@localhost lib]# md5sum /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-be4fa61-git/ebin/file2.beam 4ff9530f20673fb459bb7b7daa0d8bef /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-be4fa61-git/ebin/file2.beam after upgrade(build 161) [ root@localhost lib]# md5sum /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-8e96188-git/ebin/file file2.beam file_sorter_2.beam [ root@localhost lib]# md5sum /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-8e96188-git/ebin/file2.beam 88d98eec0ef83a6f3e8eef5a2b2f7ce4 /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-8e96188-git/ebin/file2.beam there is only one 1 file2.beam after upgrade: [ root@localhost lib]# find /opt/couchbase/ -name file2.beam /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-8e96188-git/ebin/file2.beam [ root@localhost lib]#

Filipe Manana added a comment - 20/Feb/13 9:05 AM - edited That doesn't match the erlang stack trace, which clearly says the module doesn't have the endure_dir/1 function. Maybe the Erlang VM was not restarted, which would prevent reloading the module's code in the VM. Check with the people responsible for the installer / upgrade. There's nothing I can do here, unless the same thing would happen on a clean 2.0.1 install, in which case it would be a problem on my side. Nothing changed on index file formats, file names, etc, everything's fully compatible.

Farshid Ghods added a comment - 21/Feb/13 3:34 PM Steve, do you need more information from Andrei ?

Steve Yen added a comment - 21/Feb/13 4:13 PM > Maybe the Erlang VM was not restarted, which would prevent reloading the module's code in the VM There might be something to that. Looking at the logs from: http://qa.hq.northscale.net/view/2.0.1/job/centos-32-2.0-upgrade/33/consoleFull I see many "shutdown failed" reports, like...   [2013-02-19 05:35:06,094] - [remote_util:1203] INFO - Stopping couchbase-serverNOTE: shutdown failed   [2013-02-19 05:35:06,094] - [remote_util:1203] INFO - {badrpc,nodedown} Oddly, that trace also has mention of 1.8.1, too. I need to get myself a centos 32-bit VM. In the meanwhile, perhaps we have the wrong consoleFull logs URL (or I'm not reading it right)?

Steve Yen added a comment - 21/Feb/13 5:55 PM Hi Bin, I got a 32-bit centos VM for you to try this: 10.1.4.22 Strangely, even single-node upgrade failed for me with build 161 on that VM. (But, we're up to build 164 already) I had a single default bucket, zero items, with 2 design docs each with 2 views on it... > Done: previous node configuration is empty. ---------------------------------------- [ root@localhost ~]# rpm -U couchbase-server-enterprise_x86_2.0.1-161-rel.rpm Stopping couchbase-server ... Stopping couchbase-server =INFO REPORT==== 21-Feb-2013::15:50:17 === Initiated server shutdown** at node ns_1@127.0.0.1 ** =INFO REPORT==== 21-Feb-2013::15:50:25 === Stopped ns_server application** at node ns_1@127.0.0.1 ** Minimum RAM required : 4 GB System RAM configured : 2457784 kB Minimum number of processors required : 4 cores Number of processors on the system : 6 cores Upgrading couchbase-server ...   /opt/couchbase/bin/install/cbupgrade -c /opt/couchbase/var/lib/couchbase/config -a yes Automatic mode: running without interactive questions or confirmations. Analysing... Previous config.dat file is /opt/couchbase/var/lib/couchbase/config/config.dat Target node: ns_1@10.1.4.22 Done: previous node configuration is empty. Starting couchbase-server[ OK ] You have successfully installed Couchbase Server. Please browse to http://localhost.localdomain:8091/ to configure your server. Please refer to http://couchbase.com for additional resources. Please note that you have to update your firewall configuration to allow connections to the following ports: 11211, 11210, 11209, 4369, 8091, 8092 and from 21100 to 21299. By using this software you agree to the End User License Agreement. See /opt/couchbase/LICENSE.txt. Stopping couchbase-serverNOTE: shutdown failed {badrpc,nodedown}

Bin Cui added a comment - 21/Feb/13 6:46 PM on the same 10.1.4.22 setup, i tried the following three scenarios and i haven't see the problem: 1. with data but no design doc 2. with data and with design doc 3. no data but with design doc

Farshid Ghods added a comment - 21/Feb/13 6:55 PM Bin, after upgrade did you get results from the index ? the bug description says the test expects N rows in the view but it receives 0 rows due to an error

Farshid Ghods added a comment - 21/Feb/13 6:58 PM also this occured when running a test agsinst 2 node cluster. the test is checked in . I or someone from QE can help kick off the test if needed.

Bin Cui added a comment - 21/Feb/13 7:19 PM Hard to point the problem to cluster, but who knows. For single node, here is my test steps: 1. install 2.0.0 1976 version, and populate gamesim samples with design docs and views. Make sure query has right data returned. 2. rpm -U 2.0.1 161 version. query return same result from either UI or using curl command as Andrei mentioned. Maybe needs to try out two node cluster ...

Andrei Baranouski added a comment - 22/Feb/13 3:06 AM Tried it with 1 node http://10.1.4.24/, was reproduced. left it for further investigation see MB-7770 _1node_test.log please note that vms with 2GB of RAM( long ago sent a request to update them) you can try with ./testrunner -i centos-32-2.0-upgrade.ini -t newupgradetests.MultiNodesUpgradeTests.offline_cluster_upgrade,initial_version=2.0.0-1976-rel,nodes_init=1,ddocs-num=3,upgrade_version=2.0.1-164-rel centos-32-2.0-upgrade.ini: [global] username:root password:couchbase port:8091 [servers] 1:10.1.4.24 [membase] rest_username:Administrator rest_password:password

Bin Cui added a comment - 22/Feb/13 12:19 PM On 10.1.4.24, repeat the same steps that i did on 10.1.4.22. I don't see any problems. Manual steps without any automation. Need to pay attention to the test script itself though.

Bin Cui added a comment - 22/Feb/13 1:59 PM Talked with Farshid. It is possible that upgrade happens when index building is still going. Need to verify that is the case.

Bin Cui added a comment - 22/Feb/13 4:12 PM Verified that file2.beam is updated accordingly during upgrade. However, maybe erlang process hold file2.beam during upgrade if indexing process is underway and it may lead to upgrade failure. We need to have better understanding about it.

Andrei Baranouski added a comment - 25/Feb/13 6:36 AM the test doesn't start indexing before upgrade. bucket contains only 1K items. After upgrade test runs queries without any stale param for centos 32 I get the issue and in logs there are no mentioned that index was completed  cat /opt/couchbase/var/lib/couchbase/logs/couchdb.1| grep "updater finished" [ root@localhost couchbase]# also, I tried to create new ddoc/view after upgrade. And still can't get result. I see that index was triggered on UI but it is not even started. see screenshots http://10.1.4.24:8092/_set_view/default/_design/upgrade-test-view0/_get_utilization_stats {"total_indexing_time":0,"useful_indexing_time":0,"wasted_indexing_time":0,"updates":0,"updater_interruptions":0,"compaction_time":0,"compactions":0,"compactor_interruptions":0,"replica_utilization_stats":{"total_indexing_time":0,"useful_indexing_time":0,"wasted_indexing_time":0,"updates":0,"updater_interruptions":0,"compaction_time":0,"compactions":0,"compactor_interruptions":0}} for other OSs I get that index was run and completed after upgrade

Andrei Baranouski added a comment - 25/Feb/13 6:52 AM - edited divided test on 2 separate cases: with/out indexing before upgrade note, test is successful on centos 32 too when index was completed/triggered before upgrade Filipe, maybe this will clarify some things? will keep the cluster alive http://10.1.4.24:8091/ ( root/couchbase)

Filipe Manana added a comment - 25/Feb/13 6:55 AM No. My statements from before remain. This is not a code issue on the server.

Farshid Ghods added a comment - 25/Feb/13 11:09 AM Spoke with Andrei, this issue is NOT reproducible with centos 64-bit , also the test as he says does not start indexing at all until upgrade is completed. i recommend adding this to release note and adding the workaround in case someone hits this issue

Jin Lim added a comment - 25/Feb/13 1:17 PM Start with the installer group for gathering more succinct workaround information, Bin can you provide a clear description of what would be an ideal workaround for this issue? If it has nothing to do with the installer (applies to other team, etc) please assign it back to Jin. Thanks!

Bin Cui added a comment - 25/Feb/13 4:03 PM 1. install fresh 164 build, load gamesim sample. We can get view data correctly 2. install 2.0.0-1976, upgrade to 161. Load gamesim. We CANNOT get any view data back. 3. install 2.0.0-1976. upgrade to 161. restart couchbase server, then load game-sim data. We can get view data as expected. Workaround is: After upgrade to 2.0.1, u need to manually restart couchbase server with the following commands: sudo /etc/init.d/couchbase-server stop sudo /etc/init.d/couchbase-server start

Bin Cui added a comment - 25/Feb/13 4:04 PM We may need to document the workaround method for this problem.

Karen Zeller added a comment - 04/Mar/13 4:23 PM Added to 2.0.1 RN as: When you upgrade from Couchbase Server 2.0.0 to 2.0.1 on Linux the install may not replace the <filename>file2.beam</filename> with the latest version. This will cause indexing and querying to fail. The workaround is install 2.0.1 and then manually restart Couchbase Server with the following commands: </para> </programlisting> sudo /etc/init.d/couchbase-server stop sudo /etc/init.d/couchbase-server start </programlisting>

Karen Zeller added a comment - 04/Mar/13 4:23 PM Added to 2.0.1 RN as: When you upgrade from Couchbase Server 2.0.0 to 2.0.1 on Linux the install may not replace the <filename>file2.beam</filename> with the latest version. This will cause indexing and querying to fail. The workaround is install 2.0.1 and then manually restart Couchbase Server with the following commands: </para> </programlisting> sudo /etc/init.d/couchbase-server stop sudo /etc/init.d/couchbase-server start </programlisting>

Aleksey Kondratenko added a comment - 02/May/13 3:04 PM Similar issue is still biting us and Aliaksey found why and has a good plan to fix. Reopening so that we can mark them as duplicates

Karen Zeller added a comment - 02/May/13 3:49 PM Will this a "known issue" release note candidate for 2.0.2 then?

Aliaksey Artamonau added a comment - 02/May/13 3:51 PM No, the fix is on the way.

Aleksey Kondratenko added a comment - 02/May/13 4:12 PM The fix is on the way and I believe (if understand question correctly) it's worth mentioning in release notes that this is fixed and MB-7770 does not apply anymore and need not be worked around.

Karen Zeller added a comment - 02/May/13 4:20 PM Will add as fix for 2.0.2 RN

Aleksey Kondratenko added a comment - 02/May/13 4:22 PM Also let me note that closing this bug that was not fixed at all but just had documented workaround was wrong. Because without fix we're about to merge we would still have that bug and would still need to document it as known issue for 2.0.2 and 2.1 and etc.

Karen Zeller added a comment - 02/May/13 4:26 PM I think you mean doc'ing and closing it back in March, yes? But now, you have a fix that will merge, so after I document it, I can close, yes?

Aleksey Kondratenko added a comment - 02/May/13 4:45 PM I'm not going to tell you what to do. I just pointed out that whatever was done was not quite right. And my intuition tells me that perhaps mixing closed-because-fixed and closed-because-documented is questionable practice. But I'm not going to argue about that. It's not my business.

Karen Zeller added a comment - 02/May/13 4:51 PM Well we have a new process in place for bugs so that tickets that need to be release-noted or documented can be tagged, doc'd but will remain open for engineering unless noted. Information session this Monday. We have been very messy in the past on handling doc requirements/fixes using any system (but then again this is not your business) ; ) On a separate issue, which IS your business. You say it's worth mentioning we fixed this in 2.0.2 so I will add it to RN 2.0.2 as a fix and reference this ticket.......

Aliaksey Artamonau added a comment - 02/May/13 5:15 PM Fix has been merged: http://review.couchbase.org/26044

Maria McDuff added a comment - 06/May/13 6:07 PM alk a., is this ready for QE to verify? if so, can you pls change the status to 'Resolved - Fixed'? Thanks.

Karen Zeller added a comment - 06/May/13 6:21 PM added and commented out as 2.0.2 RN fix: <para>In the past Couchbase Server 2.0.0 upgrades on Linux the install did not replace the <filename>file2.beam</filename> with the latest version. This will cause indexing and querying to fail. This has been fixed.</para>

Aliaksey Artamonau added a comment - 06/May/13 6:21 PM Yes, it's ready.

Maria McDuff added a comment - 06/May/13 6:27 PM andrei, pls verify / close.

Andrei Baranouski added a comment - 07/May/13 9:26 AM fixed in 2.0.2-786 http://qa.hq.northscale.net/job/centos-32-2.0-upgrade-P0/65/

Filipe Manana added a comment - 21/May/13 5:47 AM Any reason why the component field is not populated anymore?