http://www.couchbase.com/issues This file is an XML representation of an issue en-us 5.2.4 845 26-12-2012 http://www.couchbase.com/issues/browse/MB-6711 Couchbase Server 2.0.0-1751-rel <br/> <br/> testrunner -i /tmp/rebalance_regression.ini get-logs=True,disabled_consistent_view=False -t swaprebalance.SwapRebalanceFailedTests.test_failed_swap_rebalance,replica=2,num-buckets=1,num-swap=2,swap-orchestrator=False <br/> <a href="http://qa.hq.northscale.net/job/centos-64-2.0-rebalance-regressions/47/consoleFull">http://qa.hq.northscale.net/job/centos-64-2.0-rebalance-regressions/47/consoleFull</a> <br/> <br/> <br/> <br/> 2012-09-21 06:26:25.276 menelaus_web_alerts_srv:1:info:message(<a href='mailto:ns_1@10.3.121.94'>ns_1@10.3.121.94</a>) - Approaching full disk warning. Usage of disk &quot;/&quot; on node &quot;10.3.121.94&quot; is around 91%. <br/> 2012-09-21 06:43:15.462 supervisor_cushion:1:warning:port exited too soon after restart(<a href='mailto:ns_1@10.3.121.94'>ns_1@10.3.121.94</a>) - Service memcached exited on node &#39;<a href='mailto:ns_1@10.3.121.94'>ns_1@10.3.121.94</a>&#39; in 0.58s <br/> <br/> 2012-09-21 06:44:30.881 ns_vbucket_mover:0:critical:message(<a href='mailto:ns_1@10.3.121.92'>ns_1@10.3.121.92</a>) - &lt;0.17933.33&gt; exited with {exited, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&#39;EXIT&#39;,&lt;0.17965.33&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{replicator_died, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&#39;EXIT&#39;,&lt;19812.318.4&gt;,downstream_closed}}}} <br/> 2012-09-21 06:44:30.892 ns_memcached:2:info:message(<a href='mailto:ns_1@10.3.121.96'>ns_1@10.3.121.96</a>) - Shutting down bucket &quot;bucket-0&quot; on &#39;<a href='mailto:ns_1@10.3.121.96'>ns_1@10.3.121.96</a>&#39; for deletion <br/> 2012-09-21 06:44:31.274 ns_orchestrator:2:info:message(<a href='mailto:ns_1@10.3.121.92'>ns_1@10.3.121.92</a>) - Rebalance exited with reason {exited, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&#39;EXIT&#39;,&lt;0.17965.33&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{replicator_died, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&#39;EXIT&#39;,&lt;19812.318.4&gt;,downstream_closed}}}} <br/> <br/> <br/> logs on vm 10.3.121.94 contain many retries to notify CouchDB of update : vbid=294 rev=1 <br/> <br/> memcached&lt;0.31491.3&gt;: Fri Sep 21 06:12:14.929498 PDT 3: Retry notify CouchDB of update, vbid=294 rev=1 <br/> memcached&lt;0.31491.3&gt;: Fri Sep 21 06:12:14.931793 PDT 3: Retry notify CouchDB of update, vbid=294 rev=1 <br/> <a href='mailto:root@ubuntu1104-64'>root@ubuntu1104-64</a>:/opt/couchbase/var/lib/couchbase/logs# more info.5 <br/> [ns_server:info,2012-09-21T6:06:34.551,<a href='mailto:ns_1@10.3.121.94'>ns_1@10.3.121.94</a>:ns_port_memcached:ns_port_server:log:169]memcached&lt;0.31491.3&gt;: Fri Sep 21 06:06:34.3 <br/> 50714 PDT 3: Retry notify CouchDB of update, vbid=294 rev=1 <br/> memcached&lt;0.31491.3&gt;: Fri Sep 21 06:06:34.352976 PDT 3: Retry notify CouchDB of update, vbid=294 rev=1 <br/> memcached&lt;0.31491.3&gt;: Fri Sep 21 06:06:34.355496 PDT 3: Retry notify CouchDB of update, vbid=294 rev=1 <br/> <br/> <br/> as a result the size of vbucket 294 is 9.3 GB? <br/> <br/> <br/> <a href='mailto:root@ubuntu1104-64'>root@ubuntu1104-64</a>:/opt/couchbase/var/lib/couchbase/data/bucket-2# ls -la <br/> total 9337196 <br/> drwxr-xr-x 2 couchbase couchbase 4096 2012-09-21 06:44 . <br/> drwxr-xr-x 4 couchbase couchbase 4096 2012-09-21 03:54 .. <br/> -rw-r--r-- 1 couchbase couchbase 442459 2012-09-21 04:11 0.couch.16 <br/> -rw-r--r-- 1 couchbase couchbase 397403 2012-09-21 04:11 147.couch.14 <br/> -rw-r--r-- 1 couchbase couchbase 438363 2012-09-21 06:44 148.couch.14 <br/> -rw-r--r-- 1 couchbase couchbase 413787 2012-09-21 04:11 149.couch.14 <br/> -rw-r--r-- 1 couchbase couchbase 417883 2012-09-21 06:44 150.couch.14 <br/> -rw-r--r-- 1 couchbase couchbase 454747 2012-09-21 06:44 151.couch.13 <br/> -rw-r--r-- 1 couchbase couchbase 442459 2012-09-21 04:11 152.couch.13 <br/> -rw-r--r-- 1 couchbase couchbase 430171 2012-09-21 04:11 153.couch.13 <br/> -rw-r--r-- 1 couchbase couchbase 413787 2012-09-21 06:44 154.couch.13 <br/> -rw-r--r-- 1 couchbase couchbase 409691 2012-09-21 04:11 172.couch.12 <br/> -rw-r--r-- 1 couchbase couchbase 413787 2012-09-21 04:11 173.couch.12 <br/> -rw-r--r-- 1 couchbase couchbase 426075 2012-09-21 04:11 174.couch.12 <br/> -rw-r--r-- 1 couchbase couchbase 405595 2012-09-21 04:11 175.couch.12 <br/> -rw-r--r-- 1 couchbase couchbase 397403 2012-09-21 04:11 176.couch.12 <br/> -rw-r--r-- 1 couchbase couchbase 438363 2012-09-21 04:11 177.couch.12 <br/> -rw-r--r-- 1 couchbase couchbase 405595 2012-09-21 04:11 178.couch.11 <br/> -rw-r--r-- 1 couchbase couchbase 426075 2012-09-21 04:11 179.couch.11 <br/> -rw-r--r-- 1 couchbase couchbase 413787 2012-09-21 04:11 1.couch.16 <br/> -rw-r--r-- 1 couchbase couchbase 1355867 2012-09-21 03:54 25.couch.1 <br/> -rw-r--r-- 1 couchbase couchbase 9494917305 2012-09-21 06:43 294.couch.1 <br/> -rw-r--r-- 1 couchbase couchbase 430171 2012-09-21 04:11 294.couch.11 <br/> -rw-r--r-- 1 couchbase couchbase 421979 2012-09-21 04:11 295.couch.11 <br/> -rw-r--r-- 1 couchbase couchbase 405595 2012-09-21 04:11 296.couch.11 <br/> -rw-r--r-- 1 couchbase couchbase 426075 2012-09-21 04:11 297.couch.10 <br/> -rw-r--r-- 1 couchbase couchbase 409691 2012-09-21 04:11 298.couch.10 <br/> -rw-r--r-- 1 couchbase couchbase 426075 2012-09-21 04:11 299.couch.10 <br/> -rw-r--r-- 1 couchbase couchbase 389211 2012-09-21 04:11 2.couch.16 <br/> -rw-r--r-- 1 couchbase couchbase 421979 2012-09-21 04:11 300.couch.10 <br/> -rw-r--r-- 1 couchbase couchbase 405595 2012-09-21 04:11 301.couch.9 <br/> -rw-r--r-- 1 couchbase couchbase 594011 2012-09-21 03:54 33.couch.1 <br/> -rw-r--r-- 1 couchbase couchbase 430171 2012-09-21 04:11 343.couch.9 <br/> -rw-r--r-- 1 couchbase couchbase 426075 2012-09-21 04:11 344.couch.9 <br/> -rw-r--r-- 1 couchbase couchbase 438363 2012-09-21 04:11 345.couch.8 <br/> -rw-r--r-- 1 couchbase couchbase 454747 2012-09-21 04:11 346.couch.8 <br/> -rw-r--r-- 1 couchbase couchbase 405595 2012-09-21 04:11 347.couch.8 <br/> -rw-r--r-- 1 couchbase couchbase 401499 2012-09-21 04:11 348.couch.8 <br/> -rw-r--r-- 1 couchbase couchbase 421979 2012-09-21 04:11 349.couch.7 <br/> -rw-r--r-- 1 couchbase couchbase 426075 2012-09-21 04:11 350.couch.7 <br/> -rw-r--r-- 1 couchbase couchbase 430171 2012-09-21 04:11 351.couch.7 <br/> -rw-r--r-- 1 couchbase couchbase 409691 2012-09-21 04:11 352.couch.7 <br/> -rw-r--r-- 1 couchbase couchbase 405595 2012-09-21 04:11 353.couch.7 <br/> -rw-r--r-- 1 couchbase couchbase 401499 2012-09-21 04:11 354.couch.6 <br/> -rw-r--r-- 1 couchbase couchbase 426075 2012-09-21 04:11 355.couch.6 <br/> -rw-r--r-- 1 couchbase couchbase 397403 2012-09-21 04:11 356.couch.6 <br/> -rw-r--r-- 1 couchbase couchbase 442459 2012-09-21 04:11 357.couch.5 <br/> -rw-r--r-- 1 couchbase couchbase 438363 2012-09-21 04:11 358.couch.5 <br/> -rw-r--r-- 1 couchbase couchbase 450651 2012-09-21 04:11 359.couch.5 <br/> -rw-r--r-- 1 couchbase couchbase 421979 2012-09-21 04:11 360.couch.5 <br/> -rw-r--r-- 1 couchbase couchbase 405595 2012-09-21 04:11 361.couch.5 <br/> -rw-r--r-- 1 couchbase couchbase 421979 2012-09-21 04:11 362.couch.5 <br/> -rw-r--r-- 1 couchbase couchbase 430171 2012-09-21 04:11 363.couch.4 <br/> -rw-r--r-- 1 couchbase couchbase 426075 2012-09-21 04:11 364.couch.4 <br/> -rw-r--r-- 1 couchbase couchbase 430171 2012-09-21 04:11 365.couch.4 <br/> -rw-r--r-- 1 couchbase couchbase 450651 2012-09-21 04:11 366.couch.4 <br/> -rw-r--r-- 1 couchbase couchbase 426075 2012-09-21 04:11 367.couch.4 <br/> -rw-r--r-- 1 couchbase couchbase 430171 2012-09-21 04:11 392.couch.4 <br/> -rw-r--r-- 1 couchbase couchbase 413787 2012-09-21 04:11 393.couch.4 <br/> -rw-r--r-- 1 couchbase couchbase 413787 2012-09-21 04:11 394.couch.4 <br/> -rw-r--r-- 1 couchbase couchbase 413787 2012-09-21 04:11 395.couch.4 <br/> -rw-r--r-- 1 couchbase couchbase 442459 2012-09-21 04:11 396.couch.4 <br/> -rw-r--r-- 1 couchbase couchbase 417883 2012-09-21 04:11 397.couch.4 <br/> -rw-r--r-- 1 couchbase couchbase 430171 2012-09-21 04:11 398.couch.4 <br/> -rw-r--r-- 1 couchbase couchbase 23777465 2012-09-21 06:44 399.couch.1 <br/> -rw-r--r-- 1 couchbase couchbase 421979 2012-09-21 04:11 399.couch.4 <br/> -rw-r--r-- 1 couchbase couchbase 426075 2012-09-21 04:11 3.couch.16 <br/> -rw-r--r-- 1 couchbase couchbase 999515 2012-09-21 03:54 42.couch.1 <br/> -rw-r--r-- 1 couchbase couchbase 450651 2012-09-21 04:11 4.couch.15 <br/> -rw-r--r-- 1 couchbase couchbase 442459 2012-09-21 04:11 594.couch.3 <br/> -rw-r--r-- 1 couchbase couchbase 450651 2012-09-21 04:11 595.couch.3 <br/> -rw-r--r-- 1 couchbase couchbase 426075 2012-09-21 04:11 596.couch.3 <br/> -rw-r--r-- 1 couchbase couchbase 401499 2012-09-21 04:11 597.couch.3 <br/> -rw-r--r-- 1 couchbase couchbase 430171 2012-09-21 04:11 598.couch.3 <br/> -rw-r--r-- 1 couchbase couchbase 421979 2012-09-21 04:11 599.couch.3 <br/> -rw-r--r-- 1 couchbase couchbase 401499 2012-09-21 04:11 5.couch.15 <br/> -rw-r--r-- 1 couchbase couchbase 454747 2012-09-21 04:11 600.couch.3 <br/> -rw-r--r-- 1 couchbase couchbase 413787 2012-09-21 04:11 601.couch.2 <br/> -rw-r--r-- 1 couchbase couchbase 409691 2012-09-21 04:11 634.couch.2 <br/> -rw-r--r-- 1 couchbase couchbase 417883 2012-09-21 04:11 635.couch.2 <br/> -rw-r--r-- 1 couchbase couchbase 405595 2012-09-21 04:11 636.couch.2 <br/> -rw-r--r-- 1 couchbase couchbase 442459 2012-09-21 04:11 637.couch.2 <br/> -rw-r--r-- 1 couchbase couchbase 426075 2012-09-21 04:11 638.couch.2 <br/> -rw-r--r-- 1 couchbase couchbase 421979 2012-09-21 04:11 639.couch.2 <br/> -rw-r--r-- 1 couchbase couchbase 426075 2012-09-21 04:11 640.couch.2 <br/> -rw-r--r-- 1 couchbase couchbase 434267 2012-09-21 04:11 641.couch.2 <br/> -rw-r--r-- 1 couchbase couchbase 413787 2012-09-21 04:11 642.couch.2 <br/> -rw-r--r-- 1 couchbase couchbase 430171 2012-09-21 04:11 643.couch.2 <br/> -rw-r--r-- 1 couchbase couchbase 434267 2012-09-21 04:11 644.couch.2 <br/> -rw-r--r-- 1 couchbase couchbase 401499 2012-09-21 04:11 645.couch.2 <br/> -rw-r--r-- 1 couchbase couchbase 438363 2012-09-21 04:11 646.couch.2 <br/> -rw-r--r-- 1 couchbase couchbase 446555 2012-09-21 04:11 647.couch.2 <br/> -rw-r--r-- 1 couchbase couchbase 413787 2012-09-21 04:11 648.couch.2 <br/> -rw-r--r-- 1 couchbase couchbase 446555 2012-09-21 04:11 649.couch.2 <br/> -rw-r--r-- 1 couchbase couchbase 417883 2012-09-21 04:11 6.couch.15 <br/> -rw-r--r-- 1 couchbase couchbase 426075 2012-09-21 04:11 7.couch.15 <br/> -rw-r--r-- 1 couchbase couchbase 417883 2012-09-21 04:11 8.couch.15 <br/> -rw-r--r-- 1 couchbase couchbase 4130 2012-09-21 04:11 master.couch.16 <br/> -rw-r--r-- 1 couchbase couchbase 18915 2012-09-21 04:29 stats.json <br/> -rw-r--r-- 1 couchbase couchbase 0 2012-09-21 06:44 stats.json.new <br/> -rw-r--r-- 1 couchbase couchbase 18915 2012-09-21 04:28 stats.json.old <br/> <a href='mailto:root@ubuntu1104-64'>root@ubuntu1104-64</a>:/opt/couchbase/var/lib/couchbase/data/bucket-2# ^C <br/> <br/> <br/> <br/> <br/> <br/> <br/> MB-6711

size of one vBucket is 10 GB - Rebalance exited with reason replicator_died( many retries to notify CouchDB of update : vbid=294 rev=1)

Bug Blocker Closed Fixed Public Jin Lim Andrei Baranouski trunk-green-blockers Sun, 23 Sep 2012 01:28:48 -0500 Fri, 26 Oct 2012 15:42:42 -0500 Mon, 24 Sep 2012 17:51:31 -0500 2.0-beta-2 2.0-beta-2 couchbase-bucket ns_server 0 2 Looks like that directory listing race that we fixed in capi view manager but from perspective of ep-engine.<br/> <br/> I.e. you can see that vbucket 294 is at file revision 11 while ep-engine apparently tries to write to 1st.<br/> <br/> My guess is that ep-engine did not see 294.11 due to race in vbucket listing versus end of compaction, and started writing revision 1 assuming there&#39;s no vbucket file. And perhaps then something is broken in retry logic I think this is due to persistence not working on the latest builds Assigning to epengine team set this bug as critical because there are many tests that are failed and this issue causes the opening of new bugs: <br/> <a href="http://www.couchbase.com/issues/browse/MB-6710" title="Deleting bucket on a cluster gives error &quot;Some nodes are still deleting bucket&quot;"><strike>MB-6710</strike></a> Deleting bucket on a cluster gives error &quot;Some nodes are still deleting bucket&quot; <br/> <a href="http://www.couchbase.com/issues/browse/MB-6712" title="buckets still being deleted are not visible anywhere in REST API (was: API pools/default/buckets/ doesn&#39;t return any buckets but attempt to create bucket gives: Bucket with given name still exists)"><strike>MB-6712</strike></a> API pools/default/buckets/ doesn&#39;t return any buckets but attempt to create bucket gives: Bucket with given name still exists <br/> <br/> tests: <br/> 1) <br/> <a href="http://qa.hq.northscale.net/job/centos-64-2.0-view-query-extended-tests/70/consoleFull">http://qa.hq.northscale.net/job/centos-64-2.0-view-query-extended-tests/70/consoleFull</a> <br/> <br/> 2012-09-22 17:39:44,219] - [rest_client:96] INFO - existing buckets : [] <br/> [2012-09-22 17:39:44,226] - [rest_client:1234] INFO - <a href="http://10.2.2.60:8091/pools/default/buckets">http://10.2.2.60:8091/pools/default/buckets</a> with param: proxyPort=11211&amp;bucketType=membase&amp;authType=sasl&amp;replicaIndex=1&amp;name=default&amp;saslPassword=&amp;replicaNumber=1&amp;ramQuotaMB=1456 <br/> [2012-09-22 17:39:44,236] - [rest_client:582] ERROR - <a href="http://10.2.2.60:8091/pools/default/buckets">http://10.2.2.60:8091/pools/default/buckets</a> error 503 reason: unknown {&quot;_&quot;:&quot;Bucket with given name still exists&quot;} <br/> [2012-09-22 17:39:44,246] - [bucket_helper:124] INFO - deleting existing buckets on [ip:10.2.2.60 port:8091 ssh_username:root, ip:10.2.2.108 port:8091 ssh_username:root, ip:10.2.2.63 port:8091 ssh_username:root, ip:10.2.2.64 port:8091 ssh_username:root, ip:10.2.2.65 port:8091 ssh_username:root] <br/> [2012-09-22 17:39:44,321] - [cluster_helper:199] INFO - rebalancing all nodes in order to remove nodes <br/> [2012-09-22 17:39:44,326] - [rest_client:826] INFO - rebalance params : password=password&amp;ejectedNodes=ns_1%4010.2.2.108&amp;user=Administrator&amp;knownNodes=ns_1%4010.2.2.60%2Cns_1%4010.2.2.108 <br/> [2012-09-22 17:39:44,331] - [rest_client:833] INFO - rebalance operation started <br/> [2012-09-22 17:39:44,336] - [rest_client:929] INFO - rebalance percentage : 0 % <br/> [2012-09-22 17:39:46,341] - [rest_client:929] INFO - rebalance percentage : 0 % <br/> [2012-09-22 17:39:48,345] - [rest_client:929] INFO - rebalance percentage : 0 % <br/> [2012-09-22 17:39:50,350] - [rest_client:929] INFO - rebalance percentage : 0 % <br/> [2012-09-22 17:39:52,354] - [rest_client:929] INFO - rebalance percentage : 0 % <br/> [2012-09-22 17:39:54,359] - [rest_client:929] INFO - rebalance percentage : 0 % <br/> [2012-09-22 17:39:56,368] - [rest_client:929] INFO - rebalance percentage : 0 % <br/> [2012-09-22 17:39:58,372] - [rest_client:929] INFO - rebalance percentage : 0 % <br/> [2012-09-22 17:40:00,377] - [rest_client:929] INFO - rebalance percentage : 0 % <br/> [2012-09-22 17:40:02,381] - [rest_client:929] INFO - rebalance percentage : 0 % <br/> [2012-09-22 17:40:04,386] - [rest_client:914] ERROR - {u&#39;status&#39;: u&#39;none&#39;, u&#39;errorMessage&#39;: u&#39;Rebalance failed. See logs for detailed reason. You can try rebalance again.&#39;} - rebalance failed <br/> ERROR <br/> <br/> <br/> this show that user can trigger rebalance ( that will be failed) when &#39;deleted bucket is not deleted&#39; (separate bug?) <br/> <br/> <br/> 2)<a href="http://qa.hq.northscale.net/job/centos-64-2.0-new-rebalance/77/consoleFull">http://qa.hq.northscale.net/job/centos-64-2.0-new-rebalance/77/consoleFull</a> <br/> <br/> rebalance hangs on the same progress, disk size is growing and rebalance will falls due to lack of space <br/> <br/> andrei ~/repository/testrunner $ scripts/ssh.py -i andrei_rebalance.ini &quot;ls -la /opt/couchbase/var/lib/couchbase/data/default&quot; <br/> 10.3.3.91 <br/> total 29165028 <br/> drwxr-xr-x 2 couchbase couchbase 4096 Sep 22 17:25 . <br/> drwxr-xr-x 4 couchbase couchbase 4096 Sep 22 16:55 .. <br/> -rw-r--r-- 1 couchbase couchbase 82011 Sep 22 16:56 56.couch.2 <br/> ... <br/> -rw-r--r-- 1 couchbase couchbase 29833224378 Sep 23 01:30 95.couch.1 <br/> -rw-r--r-- 1 couchbase couchbase 77915 Sep 22 16:56 95.couch.2 <br/> .... <br/> <br/> ls: /opt/couchbase/var/lib/couchbase/data/default: No such file or directory <br/> 10.3.3.99 <br/> total 32114992 <br/> drwxr-xr-x 2 couchbase couchbase 4096 Sep 22 17:24 . <br/> drwxr-xr-x 4 couchbase couchbase 4096 Sep 22 16:54 .. <br/> -rw-r--r-- 1 couchbase couchbase 77915 Sep 22 16:57 103.couch.2 <br/> .... <br/> -rw-r--r-- 1 couchbase couchbase 82011 Sep 22 16:57 69.couch.2 <br/> -rw-r--r-- 1 couchbase couchbase 32847953966 Sep 23 01:30 70.couch.1 <br/> -rw-r--r-- 1 couchbase couchbase 82011 Sep 22 16:57 70.couch.2 <br/> ....... <br/> <br/> <br/> 10.3.3.82 <br/> total 29765164 <br/> drwxr-xr-x 2 couchbase couchbase 4096 Sep 22 17:25 . <br/> drwxr-xr-x 4 couchbase couchbase 4096 Sep 22 16:55 .. <br/> -rw-r--r-- 1 couchbase couchbase 77915 Sep 22 16:56 100.couch.2 <br/> ..... <br/> -rw-r--r-- 1 couchbase couchbase 82011 Sep 22 16:56 94.couch.2 <br/> -rw-r--r-- 1 couchbase couchbase 30447755449 Sep 23 01:30 95.couch.1 <br/> -rw-r--r-- 1 couchbase couchbase 77915 Sep 22 16:56 95.couch.2 <br/> ...... <br/> <br/> <br/> 10.3.3.93 <br/> total 35250148 <br/> drwxr-xr-x 2 couchbase couchbase 4096 Sep 22 16:55 . <br/> drwxr-xr-x 4 couchbase couchbase 4096 Sep 22 16:54 .. <br/> -rw-r--r-- 1 couchbase couchbase 82011 Sep 22 16:55 0.couch.3 <br/> ..... <br/> -rw-r--r-- 1 couchbase couchbase 77915 Sep 22 16:55 5.couch.3 <br/> -rw-r--r-- 1 couchbase couchbase 36052365358 Sep 23 01:30 50.couch.1 <br/> -rw-r--r-- 1 couchbase couchbase 82011 Sep 22 16:54 50.couch.2 <br/> .. <br/> <br/> <br/> even as Blocker Jin,<br/> <br/> I think this is a regression from our recent changes that fixed the windows issue. <a href="http://review.couchbase.org/#/c/21056/">http://review.couchbase.org/#/c/21056/</a> <br/> <br/> fix was merged Integrated in github-ep-engine-2-0 #433 (See [<a href="http://qa.hq.northscale.net/job/github-ep-engine-2-0/433/">http://qa.hq.northscale.net/job/github-ep-engine-2-0/433/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.couchbase.com/issues/browse/MB-6711" title="size of one vBucket is 10 GB - Rebalance exited with reason replicator_died( many retries to notify CouchDB of update : vbid=294 rev=1)"><strike>MB-6711</strike></a> Do not create new db file with old revision number (Revision cbc03de3b0cda2b7d7bb8bbddcdf161e4b6c0f84)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> Jin Lim : <br/> Files : <br/> * src/couch-kvstore/couch-kvstore.cc<br/> Andrei <br/> Please close the issue if it&#39;s resolved in the latest builds yes, it was fixed in 1757 RN: Replication had exited with replicator_died message after <br/> multiple attempts to update. Problem was caused by using <br/> old revision numbers for new database files. Now new database <br/> files use new revision numbers, resolving the problem. Date of First Response Sun, 23 Sep 2012 01:43:50 -0500 Rank 3915 Time In Status