http://www.couchbase.com/issues This file is an XML representation of an issue en-us 5.2.4 845 26-12-2012 http://www.couchbase.com/issues/browse/MB-7262 Couchbase Server Steps to reproduce: <br/> <br/> 1. Create a 4node cluster cl1 with default bucket (1024 vbuckets, 1 replica) <br/> 2. Create a 4node cluster cl2 with default bucket (1024 vbuckets, 1 replica) <br/> 3. Setup bidirectional XDCR for the default bucket on cl1, cl2 <br/> 4. Load 10k items to cl1 default bucket. <br/> 5. Verify both cl1, cl2 have 10k items after a while(xdcr replication) <br/> 6. Delete and recreate bucket default on cl1. <br/> 7. Add 3 documents to cl2 default bucket. <br/> 8. Total number of documents in default bucket on cl1 is 29 and on cl2 is 10003(expected). <br/> <br/> The curr_items on cl1(i.e 29) seems little strange. <br/> <br/> The cluster is available if you want to take a look: <br/> cl1 - 10.3.2.57 <br/> cl2 - 10.3.3.95 build#1965 MB-7262

curr_items after deleting/recreating a bucket on bidirectional xdcr setup seems inconsistent

Bug Critical Closed Duplicate Public Dipti Borkar Deepkaran Salooja 2.0-release-notes Mon, 26 Nov 2012 12:12:31 -0600 Wed, 10 Apr 2013 04:48:40 -0500 Thu, 6 Dec 2012 19:05:54 -0600 2.0 2.1 cross-datacenter-replication 0 3 Abhinav, <br/> <br/> Can you repro this and add information on this?<br/> <br/> On reproducing this: <br/> - After the bucket is deleted, and upon recreation of the bucket, I delete and recreate replication as well, replication worked as expected. <br/> - However I did notice one thing (when I do not recreate the replication), from the original description of the bug, at stage 7: Upon adding 3 items, 27 items were replicated to c1. <br/> - After this, when i delete and recreate replication on c2, the correct number of items were replicated on c1. <br/> - Moving on, added 3 items on c1 (currently 10006 items), the 3 items weren&#39;t replicated to c2 (as after recreation of bucket on c1, replication wasn&#39;t re-setup, replicating just shows as &quot;starting-up&quot;), upon re-setting up the replication on c1, the right number of items were replicated to c2. <br/> - Deleted the bucket on c2 now: No replication errors seen on c1 (for about 15min) <br/> - After recreating the bucket on c2, the replication from c2 stays in &quot;starting-up&quot; state, while replication on c1 says &quot;replicating&quot;, but no items replicated. <br/> - Add 3 items on c1, 37 items somehow replicated to c2, against c1&#39;s now 10009. <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;- 3 XDCR errors on c1: <br/> <br/> 2012-11-26 12:37:01 - Error replicating vbucket 537: {bad_return_value, {db_not_found, &lt;&lt;&quot;<a href="http://Administrator:*****@10.3.121.127:8092/default%2f537%3baabf41c1fa84ae48124892ea6cf855e7/">http://Administrator:*****@10.3.121.127:8092/default%2f537%3baabf41c1fa84ae48124892ea6cf855e7/</a>&quot;&gt;&gt;}} <br/> 2012-11-26 12:35:51 - Error replicating vbucket 246: {bad_return_value, {db_not_found, &lt;&lt;&quot;<a href="http://Administrator:*****@10.1.3.237:8092/default%2f246%3baabf41c1fa84ae48124892ea6cf855e7/">http://Administrator:*****@10.1.3.237:8092/default%2f246%3baabf41c1fa84ae48124892ea6cf855e7/</a>&quot;&gt;&gt;}} <br/> 2012-11-26 12:26:13 - Error replicating vbucket 770: {bad_return_value, {db_not_found, &lt;&lt;&quot;<a href="http://Administrator:*****@10.3.2.55:8092/default%2f770%3baabf41c1fa84ae48124892ea6cf855e7/">http://Administrator:*****@10.3.2.55:8092/default%2f770%3baabf41c1fa84ae48124892ea6cf855e7/</a>&quot;&gt;&gt;}} <br/> <br/> - Recreate replication on c1: items start replicating to c2, and replication completes successfully. <br/> - Replication on c2 however still says &quot;starting-up&quot; and this changes to &quot;replicating&quot; only upon recreating the replication. This is a corner case touching the dark side of bucket identity management in XDCR (probably a generic issue in couchbase server). <br/> <br/> This happens only when <br/> <br/> 1) you delete a bucket during an inbound ongoing replication, without deleting the replication first, and immediately create another one with the same name <br/> and <br/> 2) after you re-create the bucket with the same name, you load some data on the source to wake up some vb replicators <br/> <br/> IMHO, the expected behavior is that no replication should happen after you re-create the bucket, because even with the same name, they are considered completely different buckets and thus no replication should resume. <br/> <br/> Today XDCR does not check the UUID and make sure the remote cluster is still the old one when replicator is initialized. Instead, it is just trying fetch the remote cluster and start replicating. In short, we do not maintain the identity of remote cluster during XDCR, although we do that during replication of single vb replicator. In this test case filed by Deepkaran, data loading at step 7 woke up few vb replicators at C2, and they started replication without checking the bucket identity change at the other side. <br/> <br/> <br/> Some thoughts to fix the issue <br/> <br/> First of all, today we should recommend or requite users to delete any bucket only AFTER the users have deleted all XDCR toward that bucket. It is not clear to me to how to identify an inbound XDCR for a bucket since the incoming traffic from XDCR is just a stream of setMeta/getMeta/deleteWithMeta <br/> <br/> Second, probably we should maintain the remote bucket identity for the whole XDCR, instead of single vb replicator. Say, we can store the UUID of remote bucket when XDCR is created, and each time when vb replicator is initialized we should check to make sure the remote bucket UUID does not change. <br/> <br/> This fix may possibly involve change in both XDCR and remote_cluster_info module. At this time, it does not look a blocker to me, and I would like to defer the fix to 2.0.1 given the limited timing. <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> <br/> Added to RN as: <br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;Be aware that if you are using XDCR for replication to a destination bucket and you remove and create a <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;new bucket with the same name, it has a different UUIDs. Therefore any replication you had established <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;with the deleted bucket will not apply to the new bucket. <br/> <br/> Will add to XDCR chapter. Karen / Dipti, <br/> <br/> What Karen said is what I wanted but today the Couchbase server itself does not have consistent way to identify bucket. Within XDCR, we use UUID to identify a bucket and post docs to that bucket, but this does not apply for normal front-end operations. For example, if you are loading some data to a bucket, in the middle of which you delete it and recreate a new bucket with the same name, you are still able to continue loading your data to the new bucket. That essentially means we still use the name to identify a bucket in some cases. <br/> <br/> Within XDCR, either using UUID or name as bucket identifier is OK with me, but we need to be consistent that the XDCR behavior is consistent with other Couchbase Server components. <br/> <br/> <br/> Dipti, <br/> <br/> I put this bug on hold and hand it over to you. You may want to talk to ns_server team to find out which way to go, then XDCR will fix this issue accordingly. <br/> <br/> Thanks, <br/> <br/> We already have one bug handling bucket identity issue. This one is a duplicate to <a href="http://www.couchbase.com/issues/browse/MB-7266" title="XDCR never starts up after deletion and recreation of bucket (with the same name) in a unidirectional scenario"><strike>MB-7266</strike></a>. <a href="http://www.couchbase.com/issues/browse/MB-7266" title="XDCR never starts up after deletion and recreation of bucket (with the same name) in a unidirectional scenario"><strike>MB-7266</strike></a> Date of First Response Mon, 26 Nov 2012 13:34:24 -0600 Rank 3380 Time In Status