http://www.couchbase.com/issues This file is an XML representation of an issue en-us 5.2.4 845 26-12-2012 http://www.couchbase.com/issues/browse/MB-5534 Couchbase Server in viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in <br/> <br/> Load 200k docs <br/> Add+Rebalance 6 nodes to cluster while running queries <br/> <br/> Looks like what happens is the keys being moved to new servers are not always being persisted to disk. <br/> <br/> ...from test log&#39;s we report when key a did not persist and cannot be indexed by view-engine. <br/> <br/> [&quot;query doc_id: admin0150-2008_12_27 doesn\&#39;t exist in bucket: default&quot;, &quot;Error expected in results for key with invalid state {\&#39;key_vb_state\&#39;: \&#39;active\&#39;, \&#39;key_last_modification_time\&#39;: \&#39;1339537159\&#39;, \&#39;key_data_age\&#39;: \&#39;0\&#39;, \&#39;key_cas\&#39;: \&#39;7680066560307527\&#39;, \&#39;key_exptime\&#39;: \&#39;0\&#39;, \&#39;key_is_dirty\&#39;: \&#39;0\&#39;, \&#39;key_flags\&#39;: \&#39;0\&#39;, \&#39;key_valid\&#39;: \&#39;ram_but_not_disk\&#39;}&quot;, <br/> <br/> <br/> There should be plenty of time to allow this as we retry the query several times and no new docs are being loaded/updated during the rebalance. <br/> <br/> diags attached. <br/> &nbsp; <br/> <br/> 2.0.0-1314-rel MB-5534

keys failing to persist to disk with state "ram_but_not_disk" during rebalance

Bug Blocker Closed Fixed Public Jin Lim Tommie McAfee Tue, 12 Jun 2012 18:01:55 -0500 Thu, 10 Jan 2013 01:53:38 -0600 Tue, 19 Jun 2012 00:11:58 -0500 2.0 2.0 couchbase-bucket storage-engine 0 1 Hi Jin, trying to repro for you. I did not see anything in the logs about error&#39;s while persisting keys. Seeing now the state of the servers are in the same condition as <a href="http://www.couchbase.com/issues/browse/MB-5511" title="rebalance hanging in latest 2.0"><strike>MB-5511</strike></a> with rebalance hanging and deadlock in indexer. Probably a side-effect of that bug. <a href="http://www.couchbase.com/issues/browse/MB-5511" title="rebalance hanging in latest 2.0"><strike>MB-5511</strike></a> tracking the root cause of this issue. &nbsp;<a href="http://www.couchbase.com/issues/browse/MB-5511" title="rebalance hanging in latest 2.0"><strike>MB-5511</strike></a> is partially fixed (index deadlock) but rebalance is still hanging and this bug is still reproducible in latest 2.0 Hey Jin, I currently have a cluster in this state were keys are not persisted and am using a script to help get the following info: <br/> <br/> Here state of key(admin0110-2008_10_06) on host 10.2.2.64 says is &#39;ram but not disk&#39;: <br/> <br/> {&#39;key_vb_state&#39;: &#39;replica&#39;, &#39;key_last_modification_time&#39;: &#39;1339724247&#39;, &#39;key_data_age&#39;: &#39;0&#39;, &#39;key_cas&#39;: &#39;7866802812969421&#39;, &#39;key_exptime&#39;: &#39;0&#39;, &#39;key_is_dirty&#39;: &#39;0&#39;, &#39;key_flags&#39;: &#39;0&#39;, &#39;key_valid&#39;: &#39;item_deleted&#39;} <br/> <br/> If I query the database, notice this key is skipped in the results: <br/> <br/> startkey = [2008,10,6] , startkey_docid = admin0109-2008_10_06 <br/> ….. <br/> {&quot;id&quot;:&quot;admin0109-2008_10_06&quot;,&quot;key&quot;:[2008,10,6],&quot;value&quot;:[&quot;employee-109-0522e76&quot;,&quot;<a href='mailto:109@couchbase.com'>109@couchbase.com</a>&quot;]}, <br/> {&quot;id&quot;:&quot;admin0111-2008_10_06&quot;,&quot;key&quot;:[2008,10,6],&quot;value&quot;:[&quot;employee-111-0522e76&quot;,&quot;<a href='mailto:111@couchbase.com'>111@couchbase.com</a>&quot;]}, <br/> {&quot;id&quot;:&quot;admin0112-2008_10_06&quot;,&quot;key&quot;:[2008,10,6],&quot;value&quot;:[&quot;employee-112-0522e76&quot;,&quot;<a href='mailto:112@couchbase.com'>112@couchbase.com</a>&quot;]}, <br/> {&quot;id&quot;:&quot;admin0113-2008_10_06&quot;,&quot;key&quot;:[2008,10,6],&quot;value&quot;:[&quot;employee-113-0522e76&quot;,&quot;<a href='mailto:113@couchbase.com'>113@couchbase.com</a>&quot;]}, <br/> <br/> <br/> This is post rebalance in. <br/> master node is 10.2.2.60 <br/> Would be great if someone could take a look. <br/> build 1334 Unlikely to be because of <a href="http://www.couchbase.com/issues/browse/MB-5511" title="rebalance hanging in latest 2.0"><strike>MB-5511</strike></a> or <a href="http://www.couchbase.com/issues/browse/MB-5541" title="viewtests.ViewRebalanceTests.test_delete_x_docs_incremental_rebalance_in occasionally fails due to deadlock in vbucket deletion"><strike>MB-5541</strike></a>. <br/> <br/> We&#39;ve seen this error &quot;ram_but_not_disk&quot; before the changes that added the occasional deadlock were merged. Also, if the deadlock was triggered, all subsequent operations and tests would just timeout or never finish, and this wasn&#39;t the case in the results of <a href="http://www.couchbase.com/issues/browse/MB-5331" title="some docs are lost during rebalance in/out when queries are performed "><strike>MB-5331</strike></a>. With build 1338, which includes the view-engine dead lock fixes (unrelated to the problem btw, <a href="http://www.couchbase.com/issues/browse/MB-5541" title="viewtests.ViewRebalanceTests.test_delete_x_docs_incremental_rebalance_in occasionally fails due to deadlock in vbucket deletion"><strike>MB-5541</strike></a> <a href="http://www.couchbase.com/issues/browse/MB-5511" title="rebalance hanging in latest 2.0"><strike>MB-5511</strike></a>), we&#39;re still seeing this issue:<br/> <br/> <a href="http://qa.hq.northscale.net/job/centos-64-vew-merge-tests/202/">http://qa.hq.northscale.net/job/centos-64-vew-merge-tests/202/</a><br/> <br/> The test detects, via memcached, that some keys are in RAM but didn&#39;t get persisted to disk. Also, looking at one of the tests output: <br/> <br/> [2012-06-15 09:47:29,432] - [data_helper] [140513919293184] - INFO - key arch0008-2008_09_22 doesn&#39;t exist in memcached <br/> [2012-06-15 09:47:29,511] - [data_helper] [140513919293184] - INFO - key arch0000-2008_10_09 doesn&#39;t exist in memcached <br/> [2012-06-15 09:47:29,581] - [data_helper] [140513919293184] - INFO - key arch0191-2008_01_23 doesn&#39;t exist in memcached <br/> [2012-06-15 09:47:29,659] - [data_helper] [140513919293184] - INFO - key arch0187-2008_05_14 doesn&#39;t exist in memcached <br/> [2012-06-15 09:47:29,739] - [data_helper] [140513919293184] - INFO - key arch0070-2008_09_19 doesn&#39;t exist in memcached <br/> [2012-06-15 09:47:29,815] - [data_helper] [140513919293184] - INFO - key arch0040-2008_07_27 doesn&#39;t exist in memcached <br/> [2012-06-15 09:47:29,894] - [data_helper] [140513919293184] - INFO - key arch0095-2008_10_05 doesn&#39;t exist in memcached <br/> [2012-06-15 09:47:29,979] - [data_helper] [140513919293184] - INFO - key arch0087-2008_03_13 doesn&#39;t exist in memcached <br/> [2012-06-15 09:47:30,058] - [data_helper] [140513919293184] - INFO - key arch0034-2008_04_13 doesn&#39;t exist in memcached <br/> [2012-06-15 09:47:30,139] - [data_helper] [140513919293184] - INFO - key arch0084-2008_04_08 doesn&#39;t exist in memcached <br/> <br/> From: <br/> <br/> <a href="http://qa.hq.northscale.net/job/centos-64-vew-merge-tests/202/artifact/logs/testrunner-12-Jun-15_05-28-10/test_employee_dataset_alldocs_queries_rebalance_in.log">http://qa.hq.northscale.net/job/centos-64-vew-merge-tests/202/artifact/logs/testrunner-12-Jun-15_05-28-10/test_employee_dataset_alldocs_queries_rebalance_in.log</a> <br/> Farshid, can you give this high priority, like trunk green blocker? The db files being used for this failed test are gone by now. Otherwise, we could have first verified whether the missing document has never persisted or has actually persisted on disk but somehow being not known to upper layers. Anyway, while looking at remaining logs, I found a bunch of compactor crash one just like this below. And, I think this needs to be looked at by engineering team - as the compactor crash could have left a db file unknown state including missing docs. <br/> <br/> [ns_server:error] [2012-06-15 10:02:37] [<a href='mailto:ns_1@10.2.2.60'>ns_1@10.2.2.60</a>:compaction_daemon:compaction_daemon:handle_info:211] Compactor &lt;0.10399.2&gt; exited unexpectedly: {timeout, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server,call, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;capi_set_view_manager-default&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;fetch_ddocs]}}. Moving to the next bucket. <br/> [error_logger:error] [2012-06-15 10:02:37] [<a href='mailto:ns_1@10.2.2.60'>ns_1@10.2.2.60</a>:error_logger:ale_error_logger_handler:log_report:72] <br/> =========================CRASH REPORT========================= <br/> &nbsp;&nbsp;crasher: <br/> &nbsp;&nbsp;&nbsp;&nbsp;initial call: compaction_daemon:-spawn_bucket_compactor/3-fun-2-/0 <br/> &nbsp;&nbsp;&nbsp;&nbsp;pid: &lt;0.10399.2&gt; <br/> &nbsp;&nbsp;&nbsp;&nbsp;registered_name: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;exception exit: {timeout, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server,call, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;capi_set_view_manager-default&#39;,fetch_ddocs]}} <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in function gen_server:call/2 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in call from compaction_daemon:ddoc_names/1 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in call from compaction_daemon:&#39;-spawn_bucket_compactor/3-fun-2-&#39;/4 <br/> &nbsp;&nbsp;&nbsp;&nbsp;ancestors: [compaction_daemon,&lt;0.431.0&gt;,ns_server_sup, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ns_server_cluster_sup,&lt;0.59.0&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;messages: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;links: [&lt;0.438.0&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;dictionary: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;trap_exit: false <br/> &nbsp;&nbsp;&nbsp;&nbsp;status: running <br/> &nbsp;&nbsp;&nbsp;&nbsp;heap_size: 6765 <br/> &nbsp;&nbsp;&nbsp;&nbsp;stack_size: 24 <br/> &nbsp;&nbsp;&nbsp;&nbsp;reductions: 174901 <br/> &nbsp;&nbsp;neighbours: <br/> &nbsp; Jin, that doesn&#39;t cause the data loss (either persistence or memory) issues. <br/> I&#39;ll put my hands on fire if it does. <br/> And btw, this is not the compactor, it&#39;s just the compaction scheduler. This failure happened before compaction being triggered - even if it failing during compaction, when compaction is restarted it never reuses an existing partially compacted file - it truncates the existing compact file (if any) to 0 bytes - this applies both to database and view compaction. Thanks for the clarification, Filipe. Using a script from Jin that looks for the document in couchdb we found that the missing key exists in disk on a node different than the one reporting &quot;ram_but_not_disk&quot;. Something may have happened after rebalance to cause this. Below you can see two nodes are returning meta_data for the key (probably the original and new node). However, the first node has state &quot;item_deleted&quot; <br/> <br/> ./get_key_meta.py <br/> <br/> 10.2.2.108 <br/> {&#39;key_vb_state&#39;: &#39;replica&#39;, &#39;key_last_modification_time&#39;: &#39;1339788618&#39;, &#39;key_data_age&#39;: &#39;0&#39;, &#39;key_cas&#39;: &#39;7930769912841273&#39;, &#39;key_exptime&#39;: &#39;0&#39;, &#39;key_is_dirty&#39;: &#39;0&#39;, &#39;key_flags&#39;: &#39;0&#39;, &#39;key_valid&#39;: &#39;item_deleted&#39;} <br/> trying diff memcached.... <br/> trying diff memcached.... <br/> trying diff memcached.... <br/> trying diff memcached.... <br/> 10.2.2.63 <br/> {&#39;key_vb_state&#39;: &#39;active&#39;, &#39;key_last_modification_time&#39;: &#39;1339788619&#39;, &#39;key_data_age&#39;: &#39;0&#39;, &#39;key_cas&#39;: &#39;7930769912841273&#39;, &#39;key_exptime&#39;: &#39;0&#39;, &#39;key_is_dirty&#39;: &#39;0&#39;, &#39;key_flags&#39;: &#39;0&#39;, &#39;key_valid&#39;: &#39;ram_but_not_disk&#39;} <br/> <br/> <br/> --------COUCH SEARCH ON 10.2.2.108--------- <br/> [<a href='mailto:root@localhost'>root@localhost</a> default]# ./finddoc.pl arch0107-2008_11_14 <br/> Unquoted string &quot;break&quot; may clash with future reserved word at ./finddoc.pl line 28. <br/> Useless use of a constant in void context at ./finddoc.pl line 28. <br/> -------------------------------------------------- <br/> finding doc, id: arch0107-2008_11_14, in /opt/couchbase/var/lib/couchbase/data/default, may take a while... <br/> -------------------------------------------------- <br/> skipping master doc, master.couch.1 <br/> found doc, id: arch0107-2008_11_14, in 56.couch.1 <br/> [<a href='mailto:root@localhost'>root@localhost</a> default]# <br/> <br/> <br/> <br/> --------COUCH SEARCH ON 10.2.2.63--------- <br/> [<a href='mailto:root@localhost'>root@localhost</a> default]# ./finddoc.pl arch0107-2008_11_14 <br/> Unquoted string &quot;break&quot; may clash with future reserved word at ./finddoc.pl line 28. <br/> Useless use of a constant in void context at ./finddoc.pl line 28. <br/> -------------------------------------------------- <br/> finding doc, id: arch0107-2008_11_14, in /opt/couchbase/var/lib/couchbase/data/default, may take a while... <br/> -------------------------------------------------- <br/> skipping master doc, master.couch.1 <br/> [<a href='mailto:root@localhost'>root@localhost</a> default]# <br/> <br/> <br/> Tommie what&#39;s the vbucket ID? <br/> <br/> As a side questions, where do those scripts live? &gt;Tommie what&#39;s the vbucket ID? <br/> 56 <br/> <br/> &gt;where do those scripts live? <br/> They were local. I will attach them here Thanks. <br/> <br/> So it seems really like a persistence issue in the first place. <br/> The key should be persisted in node 10.2.2.63, but it is not. <br/> <br/> Hard to tell when exactly the failing queries happened, but I see that at the end of the logs for those 2 nodes the vbucket 56 is not marked as active in the indexes. Looks like a blocker rather then a major issue More debugging tips from Filipe…. discovered update_seq = 0 for db(56) which corresponds to this key on active node. This db also reports &quot;no documents&quot; so no writes have happened here: <br/> <br/> /opt/couchbase/bin/couch_dbinfo 56.couch.1 <br/> DB Info (56.couch.1) <br/> &nbsp;&nbsp;&nbsp;file format version: 10 <br/> &nbsp;&nbsp;&nbsp;update_seq: 0 <br/> &nbsp;&nbsp;&nbsp;no documents <br/> &nbsp;&nbsp;&nbsp;B-tree size: 0 bytes <br/> &nbsp;&nbsp;&nbsp;total disk size: 4.0 kB <br/> <br/> <br/> Information also reflected in index info: <br/> &nbsp;curl -s &#39;<a href="http://10.2.2.63:8092/_set_view/default/_design/dev_test_view-6ffa498/_info&#39;">http://10.2.2.63:8092/_set_view/default/_design/dev_test_view-6ffa498/_info&amp;#39;</a> | json_xs <br/> …. <br/> <br/> &nbsp;&quot;active_partitions&quot; : [ <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;56, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;57, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;58, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;59, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;60, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;61, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;62, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;63, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;64, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;65, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;66, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;67, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;68, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;69, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;70, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;71, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;72, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;73 <br/> &nbsp;&nbsp;&nbsp;], <br/> &nbsp;&nbsp;&nbsp;&quot;pending_transition&quot; : null, <br/> &nbsp;&nbsp;&nbsp;&quot;update_seqs&quot; : { <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;67&quot; : 1560, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;63&quot; : 1571, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;71&quot; : 1585, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;70&quot; : 1581, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;68&quot; : 1558, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;72&quot; : 1585, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;65&quot; : 1580, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;57&quot; : 1573, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;64&quot; : 1581, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;61&quot; : 1584, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;58&quot; : 1585, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;59&quot; : 1590, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;69&quot; : 1563, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;60&quot; : 1125, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;56&quot; : 0, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;73&quot; : 1581, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;66&quot; : 1562, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;62&quot; : 1572 <br/> &nbsp;&nbsp;&nbsp;}, <br/> Here&#39;s the test runner command to reproduce: <br/> python testrunner -i resources/jenkins/centos-64-7node-viewquery.ini -t viewquerytests.ViewQueryTests.test_employee_dataset_startkey_endkey_queries_rebalance_in <br/> <br/> I&#39;ve disabled the job in jenkins so you can run this exact command and it will run the test against our servers if you cannot reproduce in your own vm&#39;s. <br/> <br/> Some final observations in manually trying to reproduce this is that heavy queries causes disk write queue to drain slower, and when items are still in queue before rebalance(in) then we can get into this state. <br/> <br/> Let me know if anything else is needed. <br/> <br/> thanks, The test now passes without any failure after locally build ep engine with the fix, <a href="http://review.couchbase.org/#change,17367">http://review.couchbase.org/#change,17367</a>. <br/> <br/> Note: the number of items being persisted while rebalancing still fluctuates for vbuckets movements, which is perfectly normal under the current design. After the completion of rebalancing one should expect NO data loss at all. <br/> <br/> &nbsp;<a href="http://review.couchbase.org/#change,17367">http://review.couchbase.org/#change,17367</a> Super, thanks Jin! :) Date of First Response Wed, 13 Jun 2012 13:53:17 -0500 Rank 5012 Time In Status