http://www.couchbase.com/issues This file is an XML representation of an issue en-us 5.2.4 845 26-12-2012 http://www.couchbase.com/issues/browse/MB-7321 Couchbase Server 2 &lt;-&gt; 2 nodes, 2 buckets per cluster, unidir replication <br/> 4K ops/sec/cluster, 50/50 gets/sets), no views VMs, Windows 64-bit, 24GB, 4 cores <br/> build 1969 MB-7321

XDCR: constant crashes/time-outs/mb_master restarts during perf. tests on Windows

Bug Major Resolved Cannot Reproduce Public Pavel Paulau Pavel Paulau Mon, 3 Dec 2012 06:40:53 -0600 Thu, 24 Jan 2013 22:42:57 -0600 Thu, 24 Jan 2013 22:42:57 -0600 2.0 2.0.1 cross-datacenter-replication ns_server 0 2 diags:<br/> <br/> <a href="https://s3.amazonaws.com/bugdb/jira/MB-7321/d052ea5a/192.168.162.30-1222012-849-diag.zip">https://s3.amazonaws.com/bugdb/jira/MB-7321/d052ea5a/192.168.162.30-1222012-849-diag.zip</a><br/> <br/> <a href="https://s3.amazonaws.com/bugdb/jira/MB-7321/d052ea5a/192.168.162.31-1222012-855-diag.zip">https://s3.amazonaws.com/bugdb/jira/MB-7321/d052ea5a/192.168.162.31-1222012-855-diag.zip</a><br/> <br/> <a href="https://s3.amazonaws.com/bugdb/jira/MB-7321/d052ea5a/192.168.162.32-1222012-852-diag.zip">https://s3.amazonaws.com/bugdb/jira/MB-7321/d052ea5a/192.168.162.32-1222012-852-diag.zip</a><br/> <br/> <a href="https://s3.amazonaws.com/bugdb/jira/MB-7321/d052ea5a/192.168.162.33-1222012-858-diag.zip">https://s3.amazonaws.com/bugdb/jira/MB-7321/d052ea5a/192.168.162.33-1222012-858-diag.zip</a> Pavel, <br/> <br/> to understand the severity of this issue <br/> <br/> does this impact rate of replication ? is the cluster usable ? <br/> does this problem go away when you reduce the load on the cluster ? It does impact but slightly and for short period of time, for given light workload it results in small queue spikes. <br/> <br/> Symptoms are similar to issues with scheduler threads whereas _async_ threads were enabled in recent builds. Do you see these timeouts right at the start when you just set up the replication? <br/> For e.g: there would be xdcr errors if replication is set up immediately after creating a replication reference and you would be seeing &quot;Failures in grabbing vbucket stats&quot; .. <br/> <br/> I tried reproducing your scenario, but in my case i started replication a couple of minutes after i set up the replication reference, and i noticed no crashes or drop in the replication rate at any point until the finish, I had a load with the similar sets and gets ratio as well. <br/> <br/> So a fixed value for this timeout that we need to give the cluster between setting up the replication reference and actually starting the replication, is something that I am not sure of, but if we do give it a couple of minutes I am pretty sure that we shouldn&#39;t be seeing any xdcr errors in grabbing vbucket stats. No, it happens after 4-5 hours of test run time. <br/> <br/> Last run was the most troubling so far, 2 nodes were marked as down even after test. There are logs here if you are curious: <br/> <a href="http://qa.hq.northscale.net/job/xperf-win/32/">http://qa.hq.northscale.net/job/xperf-win/32/</a> XDCR saw a lot of timeout from underlying ns_server and ep_engine, and therefore, a lot of vb replicators <br/> crashed as expected. Such timeout are not expected to see for this scale of test. <br/> <br/> Pavel mentioned this happened for a short period of time, what happened to ns_server or ep_engine during that time making it so slow and even too busy to serve xdcr request? <br/> <br/> <br/> I am not sure what I can fix on the side of XDCR. Looks to me ep_engine or ns_server team need to triage the issue. <br/> <br/> <br/> <br/> [xdcr:error,2012-12-02T8:16:20.578,<a href='mailto:ns_1@10.2.3.33'>ns_1@10.2.3.33</a>:&lt;0.15416.0&gt;:xdc_vbucket_rep:terminate:298]Replication `41d1101e89da1d590261faef5067a4e8/bucket-1/bucket-1` (`bucket-1/412` -&gt; `<a href="http://Administrator:">http://Administrator:</a><a href='mailto:password@10.2'>password@10.2</a>.\ <br/> 3.31:8092/bucket-1%2f412%3b2b6f9272ff82c9cc0dfcdd22ce77b9d6`) failed: {timeout,{gen_server,call,[ns_config,get]}} <br/> <br/> [xdcr:error,2012-12-02T8:20:01.874,<a href='mailto:ns_1@10.2.3.33'>ns_1@10.2.3.33</a>:&lt;0.1527.128&gt;:capi_replication:update_replicated_docs:100][Bucket:&quot;bucket-0&quot;, Vb:505]: update 170 docs takes too long to finish!(total time spent: 189 secs, defaul\ <br/> t connection time out: 180 secs) <br/> [xdcr:error,2012-12-02T8:20:11.406,<a href='mailto:ns_1@10.2.3.33'>ns_1@10.2.3.33</a>:&lt;0.1531.128&gt;:capi_replication:update_replicated_docs:100][Bucket:&quot;bucket-0&quot;, Vb:432]: update 172 docs takes too long to finish!(total time spent: 199 secs, defaul\ <br/> t connection time out: 180 secs) <br/> <br/> <br/> [xdcr:error,2012-12-02T8:33:13.109,<a href='mailto:ns_1@10.2.3.33'>ns_1@10.2.3.33</a>:&lt;0.14775.128&gt;:xdc_vbucket_rep:terminate:284]Shutting xdcr vb replicator ({init_state, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{rep, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;41d1101e89da1d590261faef5067a4e8/bucket-1/bucket-1&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;bucket-1&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;/remoteClusters/41d1101e89da1d590261faef5067a4e8/buckets/bucket-1&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{connection_timeout,180000}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{continuous,true}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{http_connections,20}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{retries,2}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{socket_options, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{keepalive,true},{nodelay,false}]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{worker_batch_size,500}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{worker_processes,4}]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;488,&lt;0.15362.0&gt;,&lt;0.15363.0&gt;,&lt;0.15357.0&gt;}) down without ever successfully initializing: {badmatch, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{error, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;all_nodes_failed, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;Failed to grab remote bucket info from any of known nodes&quot;&gt;&gt;}} <br/> <br/> <br/> <br/> <br/> From the log above, XDCR saw lots of timeout at different stages of replicator, e.g., fetching ns_config parameter during initialization, post data during replication, and even fetch remote vbucket map. <br/> Please, be more specific about what exactly you want me to help with. From Junyi:<br/> <br/> &quot;Looks to me ep_engine or ns_server team need to triage the issue.&quot;<br/> <br/> Both xdcr and ns_server team gave the runaround. You are last candidate. per bug scrub <br/> please rerun with the latest 2.0.1 build per bug scrub <br/> <br/> Ronnie, <br/> <br/> are there results available from xdcr performance testing <br/> <br/> I don&#39;t think so. Reassign to pavel. <br/> <br/> Hi Pavel,<br/> <br/> Is there a place we summarize xdcr results?<br/> <br/> Thanks,<br/> Ronnie Not reproduced in 2.0.1 so far. Date of First Response Mon, 3 Dec 2012 11:00:02 -0600 Rank 3402 Time In Status