http://www.couchbase.com/issues This file is an XML representation of an issue en-us 5.2.4 845 26-12-2012 http://www.couchbase.com/issues/browse/MB-6041 Couchbase Server - create replication <br/> - upload some data into the source bucket <br/> - remove the replication (replication document is not present in _replicator/_all_docs anymore) <br/> - observe that number of items in the destination bucket keeps growing <br/> <br/> seeing in this on current HEAD MB-6041

XDC replication keeps on replicating even after replication document is removed

Bug Major Resolved Fixed Public Junyi Xie Aliaksey Artamonau Fri, 27 Jul 2012 14:54:58 -0500 Wed, 29 Aug 2012 14:23:19 -0500 Tue, 28 Aug 2012 14:00:02 -0500 2.0 cross-datacenter-replication 0 0 There could be some delay between the time you remove the rep doc and XDCR manager got notified and canceled all replications. Can you please provide the log of source? <br/> It seems that it happens when there are more than one replication (possibly to the same cluster). I initially observed it when I had two replications between two clusters. Then I tried to reproduced with only one replication. It worked flawlessly. Then I tried it with two replications again and again observed the bug. Attaching a diag from a source. Cannot open xz file on MacOS. Can you upload a .gz or .tar package? Thanks Ketaki, can you try this one out before and after Damien&#39;s changes. - Set up a 2:2 unidirectional replication on build 1623. <br/> - Load on source, replication kicks off on destination. <br/> - Deleted the replication on the source side: <br/> &nbsp;&nbsp;- Replication ceases to stop immediately on the destination. <br/> &nbsp;&nbsp;- I expected the replication would stop when the item count reaches the count on the source when I killed the replication. <br/> &nbsp;&nbsp;- However, the count surpasses that check point but does stop at a point much later, with the load on the source still going. Locally I created 1-1 clusters, each with two bucktes, default and default2. Start two concurrent XDCR for default and defult2, and then delete the two replication docs from UI. Both replications stopped within several seconds after I deleted the replication doc. At least at local testing, I do not see any issue. <br/> <br/> Aliaksey, can you please retry the latest code to see if the issue still exists? Thanks. <br/> <br/> <br/> <br/> I was able to reproduce it by creating two replications from the same bucket on the source to two different buckets on destination. Probably it&#39;s not very realistic scenario. But it might uncover an important issue. Will attach diag from the source cluster shortly. Replications stopped finally stopped several minutes after I removed corresponding replication documents. I tried the same setting as yours (1 -&gt; 1 replication, <a href='mailto:default@node1'>default@node1</a> -&gt; <a href='mailto:default@node2'>default@node2</a>, and <a href='mailto:default@node1'>default@node1</a> -&gt; <a href='mailto:default2@node2'>default2@node2</a>), and it seems there is nothing wrong. <br/> <br/> From the log below, XDCR replication manager got notified from ns_server instantly after I deleted the replication doc from UI, it instantly shutdown all ongoing bucket replication process, with no delay. And all XDCR activity stopped at source right after that. However, there could be some activity on destination cluster even after XDCR stopped replication on source side, because it may take a while to persist all items in memory to storage. I am not sure if there is any delay between UI stats and the real activity. Also, if both nodes in your test are on the local machine with 1024 vbuckets, it may take longer to finish. I think the delay should be much shorter if we use VMs to conduct the test. <br/> <br/> At this time I am not sure what to fix. I merged some logs for timing purpose, and will ask Ketaki to do the same test on VM. If it is really an issue, we will reopen this bug and investigate the logs from VM. <br/> <br/> <br/> [couchdb:info,2012-08-28T14:43:47.255,<a href='mailto:n_0@127.0.0.1'>n_0@127.0.0.1</a>:&lt;0.742.0&gt;:couch_log:info:39]127.0.0.1 - - DELETE /_replicator/1d38c26cdc5c5bb0e6be126e8ae272be%2Fdefault%2Fdefault?rev=1-9ee1a1c9 200 <br/> [xdcr:debug,2012-08-28T14:43:47.257,<a href='mailto:n_0@127.0.0.1'>n_0@127.0.0.1</a>:xdc_rep_manager:xdc_rep_manager:process_update:174]replication doc deleted (docId: &lt;&lt;&quot;1d38c26cdc5c5bb0e6be126e8ae272be/default/default&quot;&gt;&gt;), stop all replications <br/> [xdcr:debug,2012-08-28T14:43:47.258,<a href='mailto:n_0@127.0.0.1'>n_0@127.0.0.1</a>:xdc_rep_manager:xdc_replication_sup:stop_replication:49]all replications for DocId &lt;&lt;&quot;1d38c26cdc5c5bb0e6be126e8ae272be/default/default&quot;&gt;&gt; have been stopped <br/> <br/> <br/> [ns_server:debug,2012-08-28T14:43:47.259,<a href='mailto:n_0@127.0.0.1'>n_0@127.0.0.1</a>:&lt;0.2113.0&gt;:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_config_events,&lt;0.2112.0&gt;} exited with reason shutdown <br/> [ns_server:debug,2012-08-28T14:43:47.260,<a href='mailto:n_0@127.0.0.1'>n_0@127.0.0.1</a>:&lt;0.2113.0&gt;:ns_pubsub:do_subscribe_link:149]Deleting {ns_config_events,&lt;0.2112.0&gt;} event handler: ok <br/> [xdcr:debug,2012-08-28T14:43:47.296,<a href='mailto:n_0@127.0.0.1'>n_0@127.0.0.1</a>:&lt;0.11655.0&gt;:xdc_vbucket_rep_worker:find_missing:121]after conflict resolution at target (&quot;<a href="http://Administrator:">http://Administrator:</a><a href='mailto:asdasd@127.0.0.1'>asdasd@127.0.0.1</a>:9501/default%2f87%3b5816\ <br/> f256233b9dffc119c2c32325a512/&quot;), out of all 396 docs the number of docs we need to replicate is: 396 <br/> [couchdb:info,2012-08-28T14:43:47.304,<a href='mailto:n_0@127.0.0.1'>n_0@127.0.0.1</a>:&lt;0.1858.0&gt;:couch_log:info:39]checkpointing view update at seq 5 for _replicator _design/_replicator_info <br/> [couchdb:info,2012-08-28T14:43:47.320,<a href='mailto:n_0@127.0.0.1'>n_0@127.0.0.1</a>:&lt;0.1852.0&gt;:couch_log:info:39]127.0.0.1 - - GET /_replicator/_design/_replicator_info/_view/infos?group_level=1&amp;_=1346179427278 200 <br/> [ns_server:debug,2012-08-28T14:44:00.037,<a href='mailto:n_0@127.0.0.1'>n_0@127.0.0.1</a>:compaction_daemon:compaction_daemon:handle_info:269]Starting compaction for the following buckets: <br/> [&lt;&lt;&quot;default&quot;&gt;&gt;] <br/> [ns_server:info,2012-08-28T14:44:00.074,<a href='mailto:n_0@127.0.0.1'>n_0@127.0.0.1</a>:&lt;0.13612.0&gt;:compaction_daemon:try_to_cleanup_indexes:439]Cleaning up indexes for bucket `default` <br/> [ns_server:info,2012-08-28T14:44:00.164,<a href='mailto:n_0@127.0.0.1'>n_0@127.0.0.1</a>:&lt;0.13612.0&gt;:compaction_daemon:spawn_bucket_compactor:404]Compacting bucket default with config: <br/> [{database_fragmentation_threshold,{30,undefined}}, <br/> <br/> <a href="http://review.couchbase.org/#/c/20196/5">http://review.couchbase.org/#/c/20196/5</a> Integrated in github-ns-server-2-0 #456 (See [<a href="http://qa.hq.northscale.net/job/github-ns-server-2-0/456/">http://qa.hq.northscale.net/job/github-ns-server-2-0/456/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.couchbase.com/issues/browse/MB-6041" title="XDC replication keeps on replicating even after replication document is removed"><strike>MB-6041</strike></a>: add logs to time replication stop (Revision 1b1cf1f99f6e84b0baaa90a9ac2504b46e1d583a)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> Junyi Xie : <br/> Files : <br/> * src/xdc_rep_manager.erl<br/> * src/xdc_replication_sup.erl<br/> Date of First Response Fri, 27 Jul 2012 14:59:57 -0500 Rank 4552 Time In Status