Slow XDCR performance

Marty Schoch added a comment - 16/Nov/12 10:55 AM This is the testcase to reproduce the issue

Marty Schoch added a comment - 16/Nov/12 10:56 AM The attached python file can be placed in testrunner/pytests I'm then using the following ini file: [global] username:Administrator password:password [membase] rest_username:Administrator rest_password:password [servers] 1:10.5.2.12 2:10.5.2.6 [10.5.2.12] ip:10.5.2.12 port:8091 [10.5.2.6] ip:10.5.2.6 port:8091 And the following config: slowxdcr.SlowXDCR.slow_xdcr,nodes_in=0,replicas=1,items=10000,doc_ops=create;update;delete,xdcr_host=10.5.2.6 Note that this test is NOT using the normal XDCR test setup. Both servers are configured in the ini file as if it were one cluster, but they're never actually arranged that way. Also, the address of the second cluster is specified using a config parameter xdcr_host in the conf file. None of this is ideal for long-term testing, but happens to be what this test needs. And its the only thing we have to reproduce this behavior. When I run this test on this cluster, the second cluster will have about 8500 (out of 9999) documents when the active vb reps drops to 0, and it then struggles to slowly replicate the remaining data from that point on. Also, this isn't a true test, it just waits for 1 hour after setting up this scenario so you can watch it happen. After 1 hour, it will delete everything and record a passing test.

Marty Schoch added a comment - 16/Nov/12 10:57 AM While discussing this on a call with Dipti, it came up that the rate of replication drops when the mutations stop. I don't know exactly what the trigger mechanisms are for deciding when replicate a vbucket, but could it be that once the mutations stop, the trigger mechanism falls back to a timer, and that in this mode it takes considerably longer to walk the full vbucket circle?

Mike Wiederhold added a comment - 16/Nov/12 6:52 PM Farshid, Is this something we want to address for 2.0?

Marty Schoch added a comment - 19/Nov/12 9:52 AM From Junyi: I tried your script. The issue is your test started replication "too quickly" after it created the bucket "default" on remote cluster. The bucket was not fully created when you started the replication. Source saw a bunch of "db_not_found" error (see attachment screenshot) and wait for a while to restart vb replicator, as a result, you see slow XDCR rate. You could see such errors on XDCR tab, click "last 10 errors". I modify your test script by simply sleeping 60 seconds after you create the remote bucket, and start replication. 15K items was replicated flawlessly. Please see the attached script junyi-slowxdcr.py for my change. Note I modified it to make it working on my VMs, but it doe not change the test semantics.

Marty Schoch added a comment - 19/Nov/12 9:54 AM Confirmed Junyi's observations. I then went back to my original ElasticSearch tests and confirmed the same thing was happening. If I started XDCR too soon after creating the ElasticSearch index, it would return bucket_not found for many of the vbuckets initially. This then causes those vbuckets to not be retried for some period of time. By waiting a sufficient amount of time after creating bucket/index before starting the XDCR transfer this problem is avoided. Replication keeps up at the expected pace.

Marty Schoch added a comment - 19/Nov/12 9:55 AM Also, I believe this should be closed as not an issue (I don't have permission)

Dipti Borkar added a comment - 19/Nov/12 10:37 AM Bucket creation is much better than it used to be, but it is not 100% synchronous. The ns_server timeout is 30seconds. If disk is slow, creation of all vBuckets may not complete. We need an example of how to check if the bucket creation is done and add it to the "Create a bucket" section here : http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-admin-restapi-creating-buckets.html With a pointer to this from the XDCR note saying "Make sure the bucket on the remote cluster is created" assigning to Karen for documenting.

Karen Zeller added a comment - 20/Nov/12 12:57 PM Ok, I added this to the XDCR section and will cross reference the REST API. Here is the XDCR warning/note: " Before you set up replication via XDCR,          you should make sure that a destination bucket already exists.          If this bucket does not exist, replication          via XDCR may not find some shards on the destination cluster;          this will result in replication of only some data from          the source bucket. This would then require you to retry replication          multiple times to get a source bucket          to be fully replicated to a destination. Therefore, do make sure          that you check that a destination bucket exists."

Karen Zeller added a comment - 26/Nov/12 1:05 PM Added warning about ensuring bucket exists prior to replicating. Added workarounds for determine if bucket successfully created across nodes via cbstats or SDK.