[MB-7770] [RN 2.0.2] 2.0.0 to 2.0.1 upgrade didn't replace couchdb's file2.beam with latest version (was: [centos 32] views are broken after upgrade 2.0.0->2.0.1) Created: 19/Feb/13  Updated: 03/Jun/13  Resolved: 29/May/13

Status: Closed
Project: Couchbase Server
Component/s: installer
Affects Version/s: 2.0.1
Fix Version/s: 2.1.0
Security Level: Public

Type: Bug Priority: Major
Reporter: Andrei Baranouski Assignee: Andrei Baranouski
Resolution: Fixed Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: build 2.0.1-160-rel

Attachments: Zip Archive 10.1.4.14-2192013-1441-diag.zip     Zip Archive 10.1.4.24-2192013-1439-diag.zip     PNG File index_upgrade_centos32_1.png     PNG File index_upgrade_centos32.png     Text File test.log    
Flagged:
Release Note

 Description   

http://qa.hq.northscale.net/job/centos-32-2.0-upgrade/32/consoleFull
./testrunner -i /tmp/upgrade.ini -t newupgradetests.MultiNodesUpgradeTests.offline_cluster_upgrade,initial_version=2.0.0-1976-rel,ddocs-num=2,upgrade_version=2.0.1-160-rel

steps:
1. cluster with 2*2.0.0-1976 nodes
2. default bucket with 2*2 views
3. after offline( the same is for online tests as well)

[2013-02-18 09:22:36,045] - [rest_client:329] INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view0?connectionTimeout=60000
[2013-02-18 09:22:36,191] - [rest_client:329] INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view1?connectionTimeout=60000
[2013-02-18 09:22:36,249] - [rest_client:329] INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view0?connectionTimeout=60000
[2013-02-18 09:22:36,324] - [task:1323] INFO - (1000 rows) expected, (0 rows) returned
[2013-02-18 09:22:36,354] - [rest_client:329] INFO - index query url: http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view1?connectionTimeout=60000
[2013-02-18 09:22:36,385] - [task:1323] INFO - (1000 rows) expected, (0 rows) returned



perform manually:

andrey@baranouski:~/repository/testrunner$ curl -X GET http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view1?stale=False
{"total_rows":0,"rows":[
],
"errors":[
{"from":"local","reason":"undef"},
{"from":"http://10.1.4.14:8092/_view_merge/?stale=false","reason":"undef"}
]
}
andrey@baranouski:~/repository/testrunner$ curl -X GET http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view1?stale=update_after
{"total_rows":0,"rows":[
]
}
andrey@baranouski:~/repository/testrunner$ curl -X GET http://10.1.4.24:8092/default/_design/upgrade-test-view0/_view/upgrade-test-view1?stale=update_after
{"total_rows":0,"rows":[
]
}



andrey@baranouski:~/repository/testrunner$ python scripts/ssh.py -i centos-32-2.0-upgrade.ini "ls -la /opt/couchbase/var/lib/couchbase/data/@indexes/default"
10.1.4.14
total 32
drwxr-xr-x 2 couchbase couchbase 4096 Feb 19 01:54 .
drwxr-xr-x 3 couchbase couchbase 4096 Feb 19 01:53 ..
-rw-r--r-- 1 couchbase couchbase 10531 Feb 19 01:53 main_d3bab589b07b8f398f39483b2e415d07.view.1
-rw-r--r-- 1 couchbase couchbase 10540 Feb 19 01:53 replica_d3bab589b07b8f398f39483b2e415d07.view.1

10.1.4.24
total 32
drwxr-xr-x 2 couchbase couchbase 4096 Feb 19 01:54 .
drwxr-xr-x 3 couchbase couchbase 4096 Feb 19 01:53 ..
-rw-r--r-- 1 couchbase couchbase 10545 Feb 19 01:53 main_d3bab589b07b8f398f39483b2e415d07.view.1
-rw-r--r-- 1 couchbase couchbase 10528 Feb 19 01:53 replica_d3bab589b07b8f398f39483b2e415d07.view.1





from server logs:

[ns_server:debug,2013-02-19T3:28:03.690,ns_1@10.1.4.24:<0.29104.0>:compaction_daemon:bucket_needs_compaction:966]`default` data size is 95325, disk size is 16840750
[ns_server:debug,2013-02-19T3:28:03.691,ns_1@10.1.4.24:<0.29106.0>:compaction_daemon:view_needs_compaction:1123]`default/_design/upgrade-test-view1/main` data_size is 0, disk_size is 10545
[ns_server:debug,2013-02-19T3:28:03.691,ns_1@10.1.4.24:<0.29107.0>:compaction_daemon:view_needs_compaction:1123]`default/_design/upgrade-test-view1/replica` data_size is 0, disk_size is 10528
[ns_server:debug,2013-02-19T3:28:03.691,ns_1@10.1.4.24:<0.29109.0>:compaction_daemon:view_needs_compaction:1123]`default/_design/upgrade-test-view0/main` data_size is 0, disk_size is 10545
[ns_server:debug,2013-02-19T3:28:03.691,ns_1@10.1.4.24:<0.29110.0>:compaction_daemon:view_needs_compaction:1123]`default/_design/upgrade-test-view0/replica` data_size is 0, disk_size is 10528
[ns_server:debug,2013-02-19T3:28:03.691,ns_1@10.1.4.24:compaction_daemon<0.6699.0>:compaction_daemon:handle_info:457]Finished compaction iteration.
[ns_server:debug,2013-02-19T3:28:03.692,ns_1@10.1.4.24:compaction_daemon<0.6699.0>:compaction_daemon:schedule_next_compaction:1442]Finished compaction too soon. Next run will be in 30s
[couchdb:error,2013-02-19T3:28:20.929,ns_1@10.1.4.24:<0.29182.0>:couch_log:error:42]Set view `default`, main group `_design/upgrade-test-view1`, writer error
error: undef
stacktrace: [{file2,ensure_dir,
                    ["/opt/couchbase/var/lib/couchbase/data/@indexes/default/tmp_d3bab589b07b8f398f39483b2e415d07_main/e9e434b58f3fac3310504249a00254f9.sort"]},
             {couch_set_view_util,do_new_sort_file_path,2},
             {couch_set_view_updater,'-maybe_flush_merge_buffers/2-fun-1-',7},
             {lists,foldl,3},
             {couch_set_view_updater,flush_writes,1},
             {couch_set_view_updater,'-update/8-fun-1-',15}]

[couchdb:error,2013-02-19T3:28:20.929,ns_1@10.1.4.24:<0.6720.0>:couch_log:error:42]Set view `default`, main group `_design/upgrade-test-view1`, received error from updater: undef
[couchdb:error,2013-02-19T3:28:23.505,ns_1@10.1.4.24:<0.29209.0>:couch_log:error:42]Set view `default`, main group `_design/upgrade-test-view1`, writer error
error: undef
stacktrace: [{file2,ensure_dir,
                    ["/opt/couchbase/var/lib/couchbase/data/@indexes/default/tmp_d3bab589b07b8f398f39483b2e415d07_main/e9e434b58f3fac3310504249a0025544.sort"]},
             {couch_set_view_util,do_new_sort_file_path,2},
             {couch_set_view_updater,'-maybe_flush_merge_buffers/2-fun-1-',7},
             {lists,foldl,3},
             {couch_set_view_updater,flush_writes,1},
             {couch_set_view_updater,'-update/8-fun-1-',15}]

[couchdb:error,2013-02-19T3:28:23.506,ns_1@10.1.4.24:<0.6720.0>:couch_log:error:42]Set view `default`, main group `_design/upgrade-test-view1`, received error from updater: undef



 Comments   
Comment by Filipe Manana [ 19/Feb/13 ]
This is an installer issue.

The file2 module in 2.0.0 doesn't have the function ensure_dir/1, it was added in 2.0.1.
This means after upgrade, a file2.beam from 2.0.0 is still being used.
Comment by Farshid Ghods (Inactive) [ 19/Feb/13 ]
Andrei,

can you compare such file between fresh 2.0.1 and 2.0 installation to rule out that this could be a buildbot issues

also wonder if this is reproducible with 64-bit
Comment by Andrei Baranouski [ 20/Feb/13 ]
As I see file2.beam was replaced after upgrade and file_sorter_2.beam was added


file2.beam from 2.0.0_1976 build

[root@localhost tmp]# ls -la
total 366172
drwxrwxrwt 3 root root 4096 Feb 20 05:45 .
drwxr-xr-x 23 root root 4096 Jan 20 20:43 ..
-rwxr-xr-x 1 root root 2608 Feb 20 05:45 file2.beam_1976


after upgrade on 2.0.1_161


[root@localhost tmp]# cp /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-8e96188-git/ebin/file file2.beam_161
file2.beam file_sorter_2.beam


[root@localhost tmp]# cd /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-8e96188-git/ebin/
[root@localhost ebin]# ls -la
total 1124
drwxr-xr-x 2 bin bin 4096 Feb 20 05:59 .
drwxr-xr-x 5 bin bin 4096 Feb 20 05:59 ..
-rwxr-xr-x 1 bin bin 21484 Feb 18 23:16 couch_api_wrap.beam
-rwxr-xr-x 1 bin bin 15872 Feb 18 23:16 couch_api_wrap_httpc.beam
-rwxr-xr-x 1 bin bin 1863 Feb 18 23:16 couch.app
-rwxr-xr-x 1 bin bin 5752 Feb 18 23:16 couch_app.beam
-rwxr-xr-x 1 bin bin 19820 Feb 18 23:16 couch_auth_cache.beam
-rwxr-xr-x 1 bin bin 1664 Feb 18 23:16 couch.beam
-rwxr-xr-x 1 bin bin 47376 Feb 18 23:16 couch_btree.beam
-rwxr-xr-x 1 bin bin 16676 Feb 18 23:16 couch_btree_copy.beam
-rwxr-xr-x 1 bin bin 10932 Feb 18 23:16 couch_btree_stats.beam
-rwxr-xr-x 1 bin bin 22072 Feb 18 23:16 couch_changes.beam
-rwxr-xr-x 1 bin bin 21180 Feb 18 23:16 couch_compaction_daemon.beam
-rwxr-xr-x 1 bin bin 3764 Feb 18 23:16 couch_compress.beam
-rwxr-xr-x 1 bin bin 12844 Feb 18 23:16 couch_config.beam
-rwxr-xr-x 1 bin bin 3444 Feb 18 23:16 couch_config_writer.beam
-rwxr-xr-x 1 bin bin 32724 Feb 18 23:16 couch_db.beam
-rwxr-xr-x 1 bin bin 9184 Feb 18 23:16 couch_db_consistency_check.beam
-rwxr-xr-x 1 bin bin 8480 Feb 18 23:16 couch_db_frontend.beam
-rwxr-xr-x 1 bin bin 5564 Feb 18 23:16 couch_db_update_notifier.beam
-rwxr-xr-x 1 bin bin 2324 Feb 18 23:16 couch_db_update_notifier_sup.beam
-rwxr-xr-x 1 bin bin 47568 Feb 18 23:16 couch_db_updater.beam
-rwxr-xr-x 1 bin bin 14628 Feb 18 23:16 couch_doc.beam
-rwxr-xr-x 1 bin bin 5248 Feb 18 23:16 couch_drv.beam
-rwxr-xr-x 1 bin bin 4536 Feb 18 23:16 couch_ejson_compare.beam
-rwxr-xr-x 1 bin bin 4776 Feb 18 23:16 couch_event_sup.beam
-rwxr-xr-x 1 bin bin 27876 Feb 18 23:16 couch_file.beam
-rwxr-xr-x 1 bin bin 7056 Feb 18 23:16 couch_file_write_guard.beam
-rwxr-xr-x 1 bin bin 18224 Feb 18 23:16 couch_httpd_auth.beam
-rwxr-xr-x 1 bin bin 40444 Feb 18 23:16 couch_httpd.beam
-rwxr-xr-x 1 bin bin 32648 Feb 18 23:16 couch_httpd_db.beam
-rwxr-xr-x 1 bin bin 10408 Feb 18 23:16 couch_httpd_external.beam
-rwxr-xr-x 1 bin bin 13700 Feb 18 23:16 couch_httpd_misc_handlers.beam
-rwxr-xr-x 1 bin bin 10820 Feb 18 23:16 couch_httpd_oauth.beam
-rwxr-xr-x 1 bin bin 5152 Feb 18 23:16 couch_httpd_replicator.beam
-rwxr-xr-x 1 bin bin 35132 Feb 18 23:16 couch_httpd_view.beam
-rwxr-xr-x 1 bin bin 7848 Feb 18 23:16 couch_log.beam
-rwxr-xr-x 1 bin bin 20376 Feb 18 23:16 couch_native_process.beam
-rwxr-xr-x 1 bin bin 13636 Feb 18 23:16 couch_os_process.beam
-rwxr-xr-x 1 bin bin 2432 Feb 18 23:16 couch_primary_sup.beam
-rwxr-xr-x 1 bin bin 32032 Feb 18 23:16 couch_query_servers.beam
-rwxr-xr-x 1 bin bin 4872 Feb 18 23:16 couch_ref_counter.beam
-rwxr-xr-x 1 bin bin 28464 Feb 18 23:16 couch_replication_manager.beam
-rwxr-xr-x 1 bin bin 4676 Feb 18 23:16 couch_replication_notifier.beam
-rwxr-xr-x 1 bin bin 41112 Feb 18 23:16 couch_replicator.beam
-rwxr-xr-x 1 bin bin 22512 Feb 18 23:16 couch_replicator_utils.beam
-rwxr-xr-x 1 bin bin 18868 Feb 18 23:16 couch_replicator_worker.beam
-rwxr-xr-x 1 bin bin 3828 Feb 18 23:16 couch_rep_sup.beam
-rwxr-xr-x 1 bin bin 2148 Feb 18 23:16 couch_secondary_sup.beam
-rwxr-xr-x 1 bin bin 16904 Feb 18 23:16 couch_server.beam
-rwxr-xr-x 1 bin bin 8948 Feb 18 23:16 couch_server_sup.beam
-rwxr-xr-x 1 bin bin 8472 Feb 18 23:16 couch_task_status.beam
-rwxr-xr-x 1 bin bin 20384 Feb 18 23:16 couch_util.beam
-rwxr-xr-x 1 bin bin 6968 Feb 18 23:16 couch_uuids.beam
-rwxr-xr-x 1 bin bin 25104 Feb 18 23:16 couch_view.beam
-rwxr-xr-x 1 bin bin 9528 Feb 18 23:16 couch_view_compactor.beam
-rwxr-xr-x 1 bin bin 33088 Feb 18 23:16 couch_view_group.beam
-rwxr-xr-x 1 bin bin 17084 Feb 18 23:16 couch_view_mapreduce.beam
-rwxr-xr-x 1 bin bin 18972 Feb 18 23:16 couch_view_updater.beam
-rwxr-xr-x 1 bin bin 9664 Feb 18 23:16 couch_work_queue.beam
-rwxr-xr-x 1 bin bin 4280 Feb 18 23:16 erl_diag.beam
-rwxr-xr-x 1 bin bin 2948 Feb 18 23:16 file2.beam
-rwxr-xr-x 1 bin bin 66240 Feb 18 23:16 file_sorter_2.beam
-rwxr-xr-x 1 bin bin 14876 Feb 18 23:16 json_stream_parse.beam


test to reproduce:
python testrunner -i centos-32-2.0-upgrade.ini -t newupgradetests.MultiNodesUpgradeTests.offline_cluster_upgrade,initial_version=2.0.0-1976-rel,nodes_init=3,ddocs-num=3,upgrade_version=2.0.1-161-rel

it is reproducible only with centos 32

failed against 161 build as well
http://qa.hq.northscale.net/view/2.0.1/job/centos-32-2.0-upgrade/33/consoleFull
Comment by Andrei Baranouski [ 20/Feb/13 ]
Filipe, could you add thoughts to my comments and assign to Steve, if that's installer problem

Thanks
Comment by Filipe Manana [ 20/Feb/13 ]
I don't understand your analysis here.
Erlang VM loads files ending in .beam, so I don't get why the instaler/upgrader or whatever are producing files ending with .beam_xxx.

From the last big ls output shown, I don't see how you conclude it's the right or wrong file2.beam module.
I would grab file2.beam from 2.0.0, calculate it's md5 (with md5sum) for example, do the upgrade, then check after the upgrade that there's only one subdirectory with a file2.beam and that that file2.beam as a different md5 checksum. In case of multiple file2.beam (in different subdirectories), the Erlang VM can pick the wrong file (from 2.0.0).

The Erlang stack trace pasted before clearly (for anyone familiar with Erlang) shows that the loaded file2.beam module doesn't have the function ensure_dir/1 - that means the VM loaded a module version from 2.0.0:

error: undef
stacktrace: [{file2,ensure_dir,
                    ["/opt/couchbase/var/lib/couchbase/data/@indexes/default/tmp_d3bab589b07b8f398f39483b2e415d07_main/e9e434b58f3fac3310504249a00254f9.sort"]},
             {couch_set_view_util,do_new_sort_file_path,2},
             {couch_set_view_updater,'-maybe_flush_merge_buffers/2-fun-1-',7},
             {lists,foldl,3},
             {couch_set_view_updater,flush_writes,1},
             {couch_set_view_updater,'-update/8-fun-1-',15}]
Comment by Andrei Baranouski [ 20/Feb/13 ]
I used format .beam_xxx to save beam file before upgrade and then compare with newer version. and their size are different

okay, with md5sum

before upgrade( build 1976)
[root@localhost lib]# md5sum /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-be4fa61-git/ebin/file2.beam
4ff9530f20673fb459bb7b7daa0d8bef /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-be4fa61-git/ebin/file2.beam

after upgrade(build 161)

[root@localhost lib]# md5sum /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-8e96188-git/ebin/file
file2.beam file_sorter_2.beam
[root@localhost lib]# md5sum /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-8e96188-git/ebin/file2.beam
88d98eec0ef83a6f3e8eef5a2b2f7ce4 /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-8e96188-git/ebin/file2.beam

there is only one 1 file2.beam after upgrade:

[root@localhost lib]# find /opt/couchbase/ -name file2.beam
/opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-8e96188-git/ebin/file2.beam
[root@localhost lib]#
Comment by Filipe Manana [ 20/Feb/13 ]
That doesn't match the erlang stack trace, which clearly says the module doesn't have the endure_dir/1 function.
Maybe the Erlang VM was not restarted, which would prevent reloading the module's code in the VM.

Check with the people responsible for the installer / upgrade. There's nothing I can do here, unless the same thing would happen on a clean 2.0.1 install, in which case it would be a problem on my side. Nothing changed on index file formats, file names, etc, everything's fully compatible.
Comment by Farshid Ghods (Inactive) [ 21/Feb/13 ]
Steve,

do you need more information from Andrei ?
Comment by Steve Yen [ 21/Feb/13 ]
> Maybe the Erlang VM was not restarted, which would prevent reloading the module's code in the VM

There might be something to that.

Looking at the logs from: http://qa.hq.northscale.net/view/2.0.1/job/centos-32-2.0-upgrade/33/consoleFull

I see many "shutdown failed" reports, like...

  [2013-02-19 05:35:06,094] - [remote_util:1203] INFO - Stopping couchbase-serverNOTE: shutdown failed
  [2013-02-19 05:35:06,094] - [remote_util:1203] INFO - {badrpc,nodedown}

Oddly, that trace also has mention of 1.8.1, too.

I need to get myself a centos 32-bit VM. In the meanwhile, perhaps we have the wrong consoleFull logs URL (or I'm not reading it right)?
Comment by Steve Yen [ 21/Feb/13 ]
Hi Bin,
I got a 32-bit centos VM for you to try this: 10.1.4.22

Strangely, even single-node upgrade failed for me with build 161 on that VM. (But, we're up to build 164 already) I had a single default bucket, zero items, with 2 design docs each with 2 views on it...

> Done: previous node configuration is empty.

----------------------------------------

[root@localhost ~]# rpm -U couchbase-server-enterprise_x86_2.0.1-161-rel.rpm
Stopping couchbase-server ...
Stopping couchbase-server
=INFO REPORT==== 21-Feb-2013::15:50:17 ===
Initiated server shutdown** at node ns_1@127.0.0.1 **

=INFO REPORT==== 21-Feb-2013::15:50:25 ===
Stopped ns_server application** at node ns_1@127.0.0.1 **

Minimum RAM required : 4 GB
System RAM configured : 2457784 kB

Minimum number of processors required : 4 cores
Number of processors on the system : 6 cores

Upgrading couchbase-server ...
  /opt/couchbase/bin/install/cbupgrade -c /opt/couchbase/var/lib/couchbase/config -a yes
Automatic mode: running without interactive questions or confirmations.
Analysing...
Previous config.dat file is /opt/couchbase/var/lib/couchbase/config/config.dat
Target node: ns_1@10.1.4.22
Done: previous node configuration is empty.
Starting couchbase-server[ OK ]

You have successfully installed Couchbase Server.
Please browse to http://localhost.localdomain:8091/ to configure your server.
Please refer to http://couchbase.com for additional resources.

Please note that you have to update your firewall configuration to
allow connections to the following ports: 11211, 11210, 11209, 4369,
8091, 8092 and from 21100 to 21299.

By using this software you agree to the End User License Agreement.
See /opt/couchbase/LICENSE.txt.

Stopping couchbase-serverNOTE: shutdown failed
{badrpc,nodedown}

Comment by Bin Cui [ 21/Feb/13 ]
on the same 10.1.4.22 setup, i tried the following three scenarios and i haven't see the problem:
1. with data but no design doc
2. with data and with design doc
3. no data but with design doc

Comment by Farshid Ghods (Inactive) [ 21/Feb/13 ]
Bin,

after upgrade did you get results from the index ?

the bug description says the test expects N rows in the view but it receives 0 rows due to an error
Comment by Farshid Ghods (Inactive) [ 21/Feb/13 ]
also this occured when running a test agsinst 2 node cluster.

the test is checked in . I or someone from QE can help kick off the test if needed.
Comment by Bin Cui [ 21/Feb/13 ]
Hard to point the problem to cluster, but who knows.

For single node, here is my test steps:
1. install 2.0.0 1976 version, and populate gamesim samples with design docs and views. Make sure query has right data returned.
2. rpm -U 2.0.1 161 version. query return same result from either UI or using curl command as Andrei mentioned.

Maybe needs to try out two node cluster ...
Comment by Andrei Baranouski [ 22/Feb/13 ]
Tried it with 1 node http://10.1.4.24/, was reproduced. left it for further investigation
see MB-7770_1node_test.log

please note that vms with 2GB of RAM( long ago sent a request to update them)

you can try with ./testrunner -i centos-32-2.0-upgrade.ini -t newupgradetests.MultiNodesUpgradeTests.offline_cluster_upgrade,initial_version=2.0.0-1976-rel,nodes_init=1,ddocs-num=3,upgrade_version=2.0.1-164-rel


centos-32-2.0-upgrade.ini:
[global]
username:root
password:couchbase
port:8091

[servers]
1:10.1.4.24

[membase]
rest_username:Administrator
rest_password:password

Comment by Bin Cui [ 22/Feb/13 ]
On 10.1.4.24, repeat the same steps that i did on 10.1.4.22. I don't see any problems. Manual steps without any automation.

Need to pay attention to the test script itself though.
Comment by Bin Cui [ 22/Feb/13 ]
Talked with Farshid. It is possible that upgrade happens when index building is still going. Need to verify that is the case.
Comment by Bin Cui [ 22/Feb/13 ]
Verified that file2.beam is updated accordingly during upgrade. However, maybe erlang process hold file2.beam during upgrade if indexing process is underway and it may lead to upgrade failure. We need to have better understanding about it.
Comment by Andrei Baranouski [ 25/Feb/13 ]
the test doesn't start indexing before upgrade. bucket contains only 1K items. After upgrade test runs queries without any stale param
for centos 32 I get the issue and in logs there are no mentioned that index was completed
 cat /opt/couchbase/var/lib/couchbase/logs/couchdb.1| grep "updater finished"
[root@localhost couchbase]#

also, I tried to create new ddoc/view after upgrade. And still can't get result. I see that index was triggered on UI but it is not even started. see screenshots
http://10.1.4.24:8092/_set_view/default/_design/upgrade-test-view0/_get_utilization_stats
{"total_indexing_time":0,"useful_indexing_time":0,"wasted_indexing_time":0,"updates":0,"updater_interruptions":0,"compaction_time":0,"compactions":0,"compactor_interruptions":0,"replica_utilization_stats":{"total_indexing_time":0,"useful_indexing_time":0,"wasted_indexing_time":0,"updates":0,"updater_interruptions":0,"compaction_time":0,"compactions":0,"compactor_interruptions":0}}

for other OSs I get that index was run and completed after upgrade


Comment by Andrei Baranouski [ 25/Feb/13 ]
divided test on 2 separate cases: with/out indexing before upgrade

note, test is successful on centos 32 too when index was completed/triggered before upgrade

Filipe, maybe this will clarify some things?

will keep the cluster alive http://10.1.4.24:8091/ ( root/couchbase)



Comment by Filipe Manana [ 25/Feb/13 ]
No.
My statements from before remain. This is not a code issue on the server.
Comment by Farshid Ghods (Inactive) [ 25/Feb/13 ]
Spoke with Andrei,

this issue is NOT reproducible with centos 64-bit ,
also the test as he says does not start indexing at all until upgrade is completed.

i recommend adding this to release note and adding the workaround in case someone hits this issue
Comment by Jin Lim [ 25/Feb/13 ]
Start with the installer group for gathering more succinct workaround information, Bin can you provide a clear description of what would be an ideal workaround for this issue? If it has nothing to do with the installer (applies to other team, etc) please assign it back to Jin. Thanks!
Comment by Bin Cui [ 25/Feb/13 ]
1. install fresh 164 build, load gamesim sample. We can get view data correctly
2. install 2.0.0-1976, upgrade to 161. Load gamesim. We CANNOT get any view data back.
3. install 2.0.0-1976. upgrade to 161. restart couchbase server, then load game-sim data. We can get view data as expected.

Workaround is:
After upgrade to 2.0.1, u need to manually restart couchbase server with the following commands:
sudo /etc/init.d/couchbase-server stop
sudo /etc/init.d/couchbase-server start

Comment by Bin Cui [ 25/Feb/13 ]
We may need to document the workaround method for this problem.
Comment by kzeller [ 04/Mar/13 ]
Added to 2.0.1 RN as:

When you upgrade from Couchbase Server 2.0.0 to 2.0.1 on Linux the install may not replace the
<filename>file2.beam</filename> with the latest version. This will cause indexing and querying to fail.
The workaround is install 2.0.1 and then manually restart Couchbase Server with the following commands:
</para>
</programlisting>
sudo /etc/init.d/couchbase-server stop
sudo /etc/init.d/couchbase-server start
</programlisting>
Comment by kzeller [ 04/Mar/13 ]
Added to 2.0.1 RN as:

When you upgrade from Couchbase Server 2.0.0 to 2.0.1 on Linux the install may not replace the
<filename>file2.beam</filename> with the latest version. This will cause indexing and querying to fail.
The workaround is install 2.0.1 and then manually restart Couchbase Server with the following commands:
</para>
</programlisting>
sudo /etc/init.d/couchbase-server stop
sudo /etc/init.d/couchbase-server start
</programlisting>
Comment by Aleksey Kondratenko [ 02/May/13 ]
Similar issue is still biting us and Aliaksey found why and has a good plan to fix. Reopening so that we can mark them as duplicates
Comment by kzeller [ 02/May/13 ]
Will this a "known issue" release note candidate for 2.0.2 then?
Comment by Aliaksey Artamonau [ 02/May/13 ]
No, the fix is on the way.
Comment by Aleksey Kondratenko [ 02/May/13 ]
The fix is on the way and I believe (if understand question correctly) it's worth mentioning in release notes that this is fixed and MB-7770 does not apply anymore and need not be worked around.
Comment by kzeller [ 02/May/13 ]
Will add as fix for 2.0.2 RN
Comment by Aleksey Kondratenko [ 02/May/13 ]
Also let me note that closing this bug that was not fixed at all but just had documented workaround was wrong. Because without fix we're about to merge we would still have that bug and would still need to document it as known issue for 2.0.2 and 2.1 and etc.
Comment by kzeller [ 02/May/13 ]
I think you mean doc'ing and closing it back in March, yes?

But now, you have a fix that will merge, so after I document it, I can close, yes?

Comment by Aleksey Kondratenko [ 02/May/13 ]
I'm not going to tell you what to do. I just pointed out that whatever was done was not quite right.

And my intuition tells me that perhaps mixing closed-because-fixed and closed-because-documented is questionable practice. But I'm not going to argue about that. It's not my business.
Comment by kzeller [ 02/May/13 ]
Well we have a new process in place for bugs so that tickets that need to be release-noted or documented can be tagged, doc'd but will remain open for engineering unless noted. Information session this Monday.

We have been very messy in the past on handling doc requirements/fixes using any system (but then again this is not your business) ; )

On a separate issue, which IS your business. You say it's worth mentioning we fixed this in 2.0.2 so I will add it to RN 2.0.2 as a fix and reference this ticket.......
Comment by Aliaksey Artamonau [ 02/May/13 ]
Fix has been merged: http://review.couchbase.org/26044
Comment by Maria McDuff (Inactive) [ 06/May/13 ]
alk a., is this ready for QE to verify? if so, can you pls change the status to 'Resolved - Fixed'? Thanks.
Comment by kzeller [ 06/May/13 ]
added and commented out as 2.0.2 RN fix:

<para>In the past Couchbase Server 2.0.0 upgrades on Linux the install did not replace the
<filename>file2.beam</filename> with the latest version. This will cause indexing and querying to fail.
This has been fixed.</para>
Comment by Aliaksey Artamonau [ 06/May/13 ]
Yes, it's ready.
Comment by Maria McDuff (Inactive) [ 06/May/13 ]
andrei, pls verify / close.
Comment by Andrei Baranouski [ 07/May/13 ]
fixed in 2.0.2-786 http://qa.hq.northscale.net/job/centos-32-2.0-upgrade-P0/65/
Comment by Filipe Manana [ 21/May/13 ]
Any reason why the component field is not populated anymore?
Generated at Wed Oct 22 22:48:24 CDT 2014 using JIRA 5.2.4#845-sha1:c9f4cc41abe72fb236945343a1f485c2c844dac9.