some docs are lost during rebalance in/out when queries are performed

Filipe Manana added a comment - 23/May/12 7:10 AM Hi Andrei, I'm not sure what the title means. The whole view-engine doesn't write to databases, so it can't delete documents from them while queries are executed or updating/building indexes. I'm seeing that one of the tests failing is an _all_docs one. This likely indicates that some documents were not persisted to disk, so _all_docs can never see them (even if they are in memcached). Let's wait for Tommie's recent improvements ( http://review.couchbase.org/#change,16332 ) to help diagnose this. Also, next time, we should keep the database files for post-mortem analyzes, namely to use the tools couch_dbdump and couch_dbinfo. Example: $ ./couch_dbdump ~/couchbase2-3/ns_server/couch/0/default/0.couch.1 Total docs: 0 $ ./couch_dbinfo ~/couchbase2-3/ns_server/couch/0/default/0.couch.1 DB Info (/Users/fdmanana/couchbase2-3/ns_server/couch/0/default/0.couch.1)    file format version: 10    update_seq: 0    no documents    B-tree size: 0 bytes    total disk size: 20.0 kB

Filipe Manana added a comment - 11/Jun/12 4:45 AM After Tommie's failure diagnosis code that was added to all view tests, we saw for example in http://qa.hq.northscale.net/job/centos-64-vew-merge-tests/176/ that views are missing rows because some documents were not persisted to disk. Grep bellow, or in the above Jenkins' output, for " \'key_valid\': \'ram_but_not_disk\' ". We saw this a few more times, besides that build. I don't know where the problem is, but it's certainly not in the view engine, as it's not designed to index documents not persisted to disk. See the output below of build 176. Please try to see if anyone in other components like ep-engine or ns_server has an idea of why this is happening. Summary : Total:16 test(s) , Passed: 14 test(s) Failed: 2 test(s) - FAIL: viewquerytests.ViewQueryTests.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in,cluster_name:centos-64-7node-viewquery,spec:py-viewquery,conf_file:py-viewquery.conf,num_nodes:7 - FAIL: viewquerytests.ViewQueryTests.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_out,cluster_name:centos-64-7node-viewquery,spec:py-viewquery,conf_file:py-viewquery.conf,num_nodes:7 - PASS: viewquerytests.ViewQueryTests.viewquerytests.ViewQueryTests.test_all_datasets_all_queries,num-docs:10000,cluster_name:centos-64-7node-viewquery,spec:py-viewquery,conf_file:py-viewquery.conf,num_nodes:7 - PASS: viewquerytests.ViewQueryTests.viewquerytests.ViewQueryTests.test_employee_dataset_all_queries,cluster_name:centos-64-7node-viewquery,spec:py-viewquery,conf_file:py-viewquery.conf,num_nodes:7 - PASS: viewquerytests.ViewQueryTests.viewquerytests.ViewQueryTests.test_employee_dataset_alldocs_queries,cluster_name:centos-64-7node-viewquery,spec:py-viewquery,conf_file:py-viewquery.conf,num_nodes:7 - PASS: viewquerytests.ViewQueryTests.viewquerytests.ViewQueryTests.test_employee_dataset_alldocs_queries_rebalance_in,cluster_name:centos-64-7node-viewquery,spec:py-viewquery,conf_file:py-viewquery.conf,num_nodes:7 - PASS: viewquerytests.ViewQueryTests.viewquerytests.ViewQueryTests.test_employee_dataset_group_queries,cluster_name:centos-64-7node-viewquery,spec:py-viewquery,conf_file:py-viewquery.conf,num_nodes:7 - PASS: viewquerytests.ViewQueryTests.viewquerytests.ViewQueryTests.test_employee_dataset_stale_queries,cluster_name:centos-64-7node-viewquery,spec:py-viewquery,conf_file:py-viewquery.conf,num_nodes:7 - PASS: viewquerytests.ViewQueryTests.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_docid_queries,cluster_name:centos-64-7node-viewquery,spec:py-viewquery,conf_file:py-viewquery.conf,num_nodes:7 - PASS: viewquerytests.ViewQueryTests.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries,cluster_name:centos-64-7node-viewquery,spec:py-viewquery,conf_file:py-viewquery.conf,num_nodes:7 - PASS: viewquerytests.ViewQueryTests.viewquerytests.ViewQueryTests.test_simple_dataset_all_queries,num-docs:10000,cluster_name:centos-64-7node-viewquery,spec:py-viewquery,conf_file:py-viewquery.conf,num_nodes:7 - PASS: viewquerytests.ViewQueryTests.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries,cluster_name:centos-64-7node-viewquery,limit:1,num_docs:100000,spec:py-viewquery,conf_file:py-viewquery.conf,num_nodes:7 - PASS: viewquerytests.ViewQueryTests.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries,cluster_name:centos-64-7node-viewquery,limit:1000,num_docs:100000,spec:py-viewquery,conf_file:py-viewquery.conf,num_nodes:7 - PASS: viewquerytests.ViewQueryTests.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries,cluster_name:centos-64-7node-viewquery,limit:10000,num_docs:100000,spec:py-viewquery,conf_file:py-viewquery.conf,num_nodes:7 - PASS: viewquerytests.ViewQueryTests.viewquerytests.ViewQueryTests.test_simple_dataset_stale_queries,cluster_name:centos-64-7node-viewquery,spec:py-viewquery,conf_file:py-viewquery.conf,num_nodes:7 - PASS: viewquerytests.ViewQueryTests.viewquerytests.ViewQueryTests.test_simple_dataset_startkey_endkey_queries,cluster_name:centos-64-7node-viewquery,spec:py-viewquery,conf_file:py-viewquery.conf,num_nodes:7 BUILD UNSTABLE Build duration: 5 hr 35 min Build URL: http://qa.hq.northscale.net/job/centos-64-vew-merge-tests/176/ 16 2 tests failed. FAILED: viewquerytests.ViewQueryTests.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in,cluster_name:centos-64-7node-viewquery,spec:py-viewquery,conf_file:py-viewquery.conf,num_nodes:7 Error Message: null Stack Trace:                        Traceback (most recent call last):  File "pytests/viewquerytests.py", line 247, in test_employee_dataset_startkey_endkey_queries_rebalance_in    self._query_all_views(data_set.views)  File "pytests/viewquerytests.py", line 365, in _query_all_views    self._check_view_intergrity(views)  File "pytests/viewquerytests.py", line 374, in _check_view_intergrity    [ex[1] for ex in view.results.failures]) AssertionError: ['Traceback (most recent call last):\n File "pytests/viewquerytests.py", line 509, in run_queries\n tc.assertEquals(len(key_failures), 0, msg)\nAssertionError: unable to retrieve expected results: ["query doc_id: admin0132-2008_02_07 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339184343\', \'key_data_age\': \'0\', \'key_cas\': \'7325660991709372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0105-2008_02_20 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339184343\', \'key_data_age\': \'0\', \'key_cas\': \'7325657387933372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0137-2008_02_19 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339184343\', \'key_data_age\': \'0\', \'key_cas\': \'7325645083533372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0139-2008_02_14 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339184343\', \'key_data_age\': \'0\', \'key_cas\': \'7325650432839372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0047-2008_03_01 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339184343\', \'key_data_age\': \'0\', \'key_cas\': \'7325725562583193\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: arch0057-2008_02_16 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339184343\', \'key_data_age\': \'0\', \'key_cas\': \'7325671965506372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0154-2008_01_03 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339184343\', \'key_data_age\': \'0\', \'key_cas\': \'7325641149450372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: arch0008-2008_03_17 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339184343\', \'key_data_age\': \'0\', \'key_cas\': \'7325652766812372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0094-2008_02_03 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339184343\', \'key_data_age\': \'0\', \'key_cas\': \'7325712337514193\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0069-2008_01_06 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339184343\', \'key_data_age\': \'0\', \'key_cas\': \'7325755579211944\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}"]\n', 'Traceback (most recent call last):\n File "pytests/viewquerytests.py", line 509, in run_queries\n tc.assertEquals(len(key_failures), 0, msg)\nAssertionError: unable to retrieve expected results: ["query doc_id: admin0132-2008_02_07 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339185279\', \'key_data_age\': \'0\', \'key_cas\': \'7325660991709372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0105-2008_02_20 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339185279\', \'key_data_age\': \'0\', \'key_cas\': \'7325657387933372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0137-2008_02_19 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339185279\', \'key_data_age\': \'0\', \'key_cas\': \'7325645083533372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0139-2008_02_14 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339185283\', \'key_data_age\': \'0\', \'key_cas\': \'7325650432839372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0047-2008_03_01 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339185283\', \'key_data_age\': \'0\', \'key_cas\': \'7325725562583193\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: arch0057-2008_02_16 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339185287\', \'key_data_age\': \'0\', \'key_cas\': \'7325671965506372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0154-2008_01_03 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339185291\', \'key_data_age\': \'0\', \'key_cas\': \'7325641149450372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: arch0008-2008_03_17 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339185291\', \'key_data_age\': \'0\', \'key_cas\': \'7325652766812372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0094-2008_02_03 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339185291\', \'key_data_age\': \'0\', \'key_cas\': \'7325712337514193\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0069-2008_01_06 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339185291\', \'key_data_age\': \'0\', \'key_cas\': \'7325755579211944\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}"]\n', 'Traceback (most recent call last):\n File "pytests/viewquerytests.py", line 509, in run_queries\n tc.assertEquals(len(key_failures), 0, msg)\nAssertionError: unable to retrieve expected results: ["query doc_id: admin0132-2008_02_07 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339185851\', \'key_data_age\': \'0\', \'key_cas\': \'7325660991709372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0105-2008_02_20 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339185851\', \'key_data_age\': \'0\', \'key_cas\': \'7325657387933372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0137-2008_02_19 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339185851\', \'key_data_age\': \'0\', \'key_cas\': \'7325645083533372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0139-2008_02_14 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339185851\', \'key_data_age\': \'0\', \'key_cas\': \'7325650432839372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0047-2008_03_01 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339185851\', \'key_data_age\': \'0\', \'key_cas\': \'7325725562583193\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: arch0057-2008_02_16 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339185851\', \'key_data_age\': \'0\', \'key_cas\': \'7325671965506372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0154-2008_01_03 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339185851\', \'key_data_age\': \'0\', \'key_cas\': \'7325641149450372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: arch0008-2008_03_17 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339185851\', \'key_data_age\': \'0\', \'key_cas\': \'7325652766812372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0034-2008_03_05 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339185851\', \'key_data_age\': \'0\', \'key_cas\': \'7325642655756372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0102-2008_01_10 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339185851\', \'key_data_age\': \'0\', \'key_cas\': \'7325654650046372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}"]\n', 'Traceback (most recent call last):\n File "pytests/viewquerytests.py", line 509, in run_queries\n tc.assertEquals(len(key_failures), 0, msg)\nAssertionError: unable to retrieve expected results: ["query doc_id: admin0132-2008_02_07 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339186303\', \'key_data_age\': \'0\', \'key_cas\': \'7325660991709372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0105-2008_02_20 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339186303\', \'key_data_age\': \'0\', \'key_cas\': \'7325657387933372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0137-2008_02_19 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339186303\', \'key_data_age\': \'0\', \'key_cas\': \'7325645083533372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0139-2008_02_14 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339186303\', \'key_data_age\': \'0\', \'key_cas\': \'7325650432839372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0047-2008_03_01 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339186303\', \'key_data_age\': \'0\', \'key_cas\': \'7325725562583193\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: arch0057-2008_02_16 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339186303\', \'key_data_age\': \'0\', \'key_cas\': \'7325671965506372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0154-2008_01_03 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339186303\', \'key_data_age\': \'0\', \'key_cas\': \'7325641149450372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: arch0008-2008_03_17 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339186307\', \'key_data_age\': \'0\', \'key_cas\': \'7325652766812372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0034-2008_03_05 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339186307\', \'key_data_age\': \'0\', \'key_cas\': \'7325642655756372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0102-2008_01_10 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339186307\', \'key_data_age\': \'0\', \'key_cas\': \'7325654650046372\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}"]\n'] FAILED: viewquerytests.ViewQueryTests.viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_out,cluster_name:centos-64-7node-viewquery,spec:py-viewquery,conf_file:py-viewquery.conf,num_nodes:7 Error Message: null Stack Trace:                        Traceback (most recent call last):  File "pytests/viewquerytests.py", line 263, in test_employee_dataset_startkey_endkey_queries_rebalance_out    self._query_all_views(data_set.views)  File "pytests/viewquerytests.py", line 365, in _query_all_views    self._check_view_intergrity(views)  File "pytests/viewquerytests.py", line 374, in _check_view_intergrity    [ex[1] for ex in view.results.failures]) AssertionError: ['Traceback (most recent call last):\n File "pytests/viewquerytests.py", line 509, in run_queries\n tc.assertEquals(len(key_failures), 0, msg)\nAssertionError: unable to retrieve expected results: ["query doc_id: ui0128-2008_04_25 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339188916\', \'key_data_age\': \'0\', \'key_cas\': \'7329969465018970\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: ui0093-2008_02_06 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339188916\', \'key_data_age\': \'0\', \'key_cas\': \'7330008602536970\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: arch0068-2008_04_27 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339188920\', \'key_data_age\': \'0\', \'key_cas\': \'7329912965265475\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: arch0017-2008_05_03 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339188920\', \'key_data_age\': \'0\', \'key_cas\': \'7330069544249298\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: ui0100-2008_02_26 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339188924\', \'key_data_age\': \'0\', \'key_cas\': \'7330018106110130\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: arch0057-2008_03_08 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339188924\', \'key_data_age\': \'0\', \'key_cas\': \'7330042329012294\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0160-2008_05_22 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339188924\', \'key_data_age\': \'0\', \'key_cas\': \'7329915664706475\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0138-2008_05_10 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339188924\', \'key_data_age\': \'0\', \'key_cas\': \'7329928785659475\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: ui0193-2008_04_01 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339188928\', \'key_data_age\': \'0\', \'key_cas\': \'7330046756414294\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: arch0022-2008_06_26 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339188928\', \'key_data_age\': \'0\', \'key_cas\': \'7329956949692299\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}"]\n', 'Traceback (most recent call last):\n File "pytests/viewquerytests.py", line 509, in run_queries\n tc.assertEquals(len(key_failures), 0, msg)\nAssertionError: unable to retrieve expected results: ["query doc_id: ui0128-2008_04_25 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339189988\', \'key_data_age\': \'0\', \'key_cas\': \'7329969465018970\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: ui0093-2008_02_06 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339189988\', \'key_data_age\': \'0\', \'key_cas\': \'7330008602536970\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: arch0068-2008_04_27 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339189988\', \'key_data_age\': \'0\', \'key_cas\': \'7329912965265475\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: arch0017-2008_05_03 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339189988\', \'key_data_age\': \'0\', \'key_cas\': \'7330069544249298\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: ui0100-2008_02_26 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339189988\', \'key_data_age\': \'0\', \'key_cas\': \'7330018106110130\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: arch0057-2008_03_08 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339189988\', \'key_data_age\': \'0\', \'key_cas\': \'7330042329012294\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0160-2008_05_22 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339189988\', \'key_data_age\': \'0\', \'key_cas\': \'7329915664706475\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0138-2008_05_10 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339189988\', \'key_data_age\': \'0\', \'key_cas\': \'7329928785659475\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: ui0193-2008_04_01 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339189988\', \'key_data_age\': \'0\', \'key_cas\': \'7330046756414294\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: arch0022-2008_06_26 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339189988\', \'key_data_age\': \'0\', \'key_cas\': \'7329956949692299\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}"]\n', 'Traceback (most recent call last):\n File "pytests/viewquerytests.py", line 509, in run_queries\n tc.assertEquals(len(key_failures), 0, msg)\nAssertionError: unable to retrieve expected results: ["query doc_id: ui0128-2008_04_25 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339190368\', \'key_data_age\': \'0\', \'key_cas\': \'7329969465018970\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: ui0093-2008_02_06 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339190368\', \'key_data_age\': \'0\', \'key_cas\': \'7330008602536970\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: arch0068-2008_04_27 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339190368\', \'key_data_age\': \'0\', \'key_cas\': \'7329912965265475\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: arch0017-2008_05_03 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339190368\', \'key_data_age\': \'0\', \'key_cas\': \'7330069544249298\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: ui0100-2008_02_26 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339190368\', \'key_data_age\': \'0\', \'key_cas\': \'7330018106110130\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: ui0088-2008_03_08 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339190368\', \'key_data_age\': \'0\', \'key_cas\': \'7330042329012294\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0160-2008_05_22 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339190368\', \'key_data_age\': \'0\', \'key_cas\': \'7329915664706475\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0138-2008_05_10 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339190368\', \'key_data_age\': \'0\', \'key_cas\': \'7329928785659475\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: ui0193-2008_04_01 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339190368\', \'key_data_age\': \'0\', \'key_cas\': \'7330046756414294\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: arch0022-2008_06_26 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339190368\', \'key_data_age\': \'0\', \'key_cas\': \'7329956949692299\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}"]\n', 'Traceback (most recent call last):\n File "pytests/viewquerytests.py", line 509, in run_queries\n tc.assertEquals(len(key_failures), 0, msg)\nAssertionError: unable to retrieve expected results: ["query doc_id: ui0128-2008_04_25 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339190752\', \'key_data_age\': \'0\', \'key_cas\': \'7329969465018970\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: ui0093-2008_02_06 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339190752\', \'key_data_age\': \'0\', \'key_cas\': \'7330008602536970\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: arch0068-2008_04_27 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339190752\', \'key_data_age\': \'0\', \'key_cas\': \'7329912965265475\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: arch0017-2008_05_03 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339190752\', \'key_data_age\': \'0\', \'key_cas\': \'7330069544249298\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: ui0100-2008_02_26 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339190752\', \'key_data_age\': \'0\', \'key_cas\': \'7330018106110130\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: ui0088-2008_03_08 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339190752\', \'key_data_age\': \'0\', \'key_cas\': \'7330042329012294\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0160-2008_05_22 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339190752\', \'key_data_age\': \'0\', \'key_cas\': \'7329915664706475\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: admin0138-2008_05_10 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339190752\', \'key_data_age\': \'0\', \'key_cas\': \'7329928785659475\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: ui0193-2008_04_01 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339190752\', \'key_data_age\': \'0\', \'key_cas\': \'7330046756414294\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}", "query doc_id: arch0022-2008_06_26 doesn\'t exist in bucket: default", "Error expected in results for key with invalid state {\'key_vb_state\': \'active\', \'key_last_modification_time\': \'1339190752\', \'key_data_age\': \'0\', \'key_cas\': \'7329956949692299\', \'key_exptime\': \'0\', \'key_is_dirty\': \'0\', \'key_flags\': \'0\', \'key_valid\': \'ram_but_not_disk\'}"]\n']

Filipe Manana added a comment - 11/Jun/12 4:46 AM Farshid, where to take it from here?

Farshid Ghods added a comment - 11/Jun/12 11:05 AM Jin, these tests have been consistently failing for a very long time but we were unsure whether this is related to view-engine or persistence layer. the test now uses stat vkey and all_docs to confirm whether the item is stored on the disk or not and it appears that it is not.

Farshid Ghods added a comment - 12/Jun/12 11:48 PM dupe of http://www.couchbase.com/issues/browse/MB-5507 and http://www.couchbase.com/issues/browse/MB-5482?

Jin Lim added a comment - 13/Jun/12 1:54 PM Duplicate of MB-5482

Jin Lim added a comment - 13/Jun/12 2:18 PM NS Server on replicating node (new master) kept crashing due to port collision between memcached and moxi. We are not sure if this is an environment issue (relate to testrunner) or ns server issue. Assign this to Aliaksey A for initial debugging at NS Server side. The crash logs are below: {ok,{1339563643973550,#Ref<0.0.7.250488>}},                                ["2012-06-12 22:00:45: (cproxy.c.330) ERROR: could not listen on port 12009. Please use -Z port_listen=PORT_NUM to specify a different port number.",                                 "2012-06-12 22:00:43: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (8)"],                                0,false} ** Reason for termination == ** {abnormal,1} [error_logger:error] [2012-06-12 22:00:43] [ n_1@10.17.14.30 :error_logger:ale_error_logger_handler:log_report:72] =========================CRASH REPORT=========================   crasher:     initial call: ns_port_server:init/1     pid: <0.13809.1>     registered_name: []     exception exit: {abnormal,1}       in function gen_server:terminate/6     ancestors: [ns_moxi_sup,ns_server_sup,ns_server_cluster_sup,<0.60.0>]     messages: [{'EXIT',#Port<0.52597>,normal}]     links: [<0.13785.1>]     dictionary: []     trap_exit: true     status: running     heap_size: 2584     stack_size: 24     reductions: 2391   neighbours: [error_logger:error] [2012-06-12 22:00:43] [ n_3@10.17.14.30 :error_logger:ale_error_logger_handler:log_report:72] =========================CRASH REPORT=========================   crasher:     initial call: ns_port_server:init/1     pid: <0.10209.1>     registered_name: []     exception exit: {abnormal,1}       in function gen_server:terminate/6     ancestors: [ns_moxi_sup,ns_server_sup,ns_server_cluster_sup,<0.59.0>]     messages: [{'EXIT',#Port<0.48303>,normal}]     links: [<0.10195.1>]     dictionary: []     trap_exit: true     status: running     heap_size: 2584     stack_size: 24     reductions: 2391   neighbours: In order to reproduce this issue, do the following steps: Copy attached files into ~/testrunner/conf/xdcr-adv.conf & ~testrunner/pytest/xdcr.py accordingly. 1) start 4 node cluster, cluster_run -n4 2) run ./testrunner -i b/resources/dev-4-nodes-xdcr.ini -c conf/xdcr-adv.conf 3) as soon as rebalance starts, check ns_server/logs/n_1 or n_4 for crash logs due to the port collision 4) ps aux | grep moxi & ps aux | grep memcached to see that duplicate port (12003 or 12009) being assigned to memcached and moxi

Jin Lim added a comment - 13/Jun/12 2:20 PM This is the issue we debugged together this morning.

Aliaksey Artamonau added a comment - 13/Jun/12 4:36 PM So the problem is that test creates bucket with dedicated moxi. And it takes the next port after global moxi port. But of course it's occupied by the memcached of the next node started by cluster_run.

Jin Lim added a comment - 13/Jun/12 7:14 PM Looks like this was another test environment issue. Can you please take a look at the last comment from Aliaksey and see to it? Thanks.

Andrei Baranouski added a comment - 14/Jun/12 6:58 PM all tests are passed in 2.0.0-1330-rel

Filipe Manana added a comment - 14/Jun/12 7:02 PM Can I ask what changes were made to fix this? Was it any change in a component, the tests, or the test environment configuration? We've seen this happening some times (not too often however) with very different builds, just want to make sure it's gone for sure.

Tommie McAfee added a comment - 14/Jun/12 8:52 PM Possible this bug is related? MB-5534 encountered in 2.0 build 1334

Filipe Manana added a comment - 20/Jun/12 7:42 AM Duplicate of MB-5534 . Nothing was fixed here, the problem existed until MB-5534 was fixed (ep-engine issue).