http://www.couchbase.com/issues This file is an XML representation of an issue en-us 5.2.4 845 26-12-2012 http://www.couchbase.com/issues/browse/MB-7147 Couchbase Server ____________________ <br/> From: Ronnie Sun <br/> Sent: Friday, November 09, 2012 6:34 PM <br/> To: Steve Yen <br/> Subject: Re: swap rebalance perf? <br/> <br/> Hi Steve, <br/> <br/> Got some preliminary results showing 2.0 is 10x slower, which falls into the same area we&#39;ve seen for reb-litmus tests. <br/> <br/> 1.8.1-938: 203 seconds <br/> 2.0.0-1939: 2983 seconds <br/> <br/> Spec: (reb-litmus-swap-2) 3M mixed, start with 2 nodes, swap 1 node with 9 client firing 1k ops each. <br/> <br/> Current jenkins automation for this has some problems, so I don&#39;t have comparison graphs. <br/> <br/> Thanks, <br/> Ronnie <br/> MB-7147

swap rebalance slowness

Bug Blocker Resolved Fixed Public Ronnie Sun Ronnie Sun Fri, 9 Nov 2012 20:49:01 -0600 Thu, 3 Jan 2013 12:21:26 -0600 Wed, 21 Nov 2012 13:25:22 -0600 2.0-beta-2 2.0 ns_server 0 3 adding affected version and fix version values so this will show up in the right filters for 2.0 bug scrub. <br/> <br/> also, changing from major, and if it&#39;s Blocker, we&#39;ll raise it in the next bug scrub mtg. Ronnie,<br/> <br/> I need cb_collect_info from the nodes that are newly added. chiyoung,<br/> <br/> I repeated the test on vms, this time reb took 1 hr.<br/> <br/> Left the cluster for you, 10.2.2.168.<br/> <br/> Note the ip addresses shown on the UI are internal ips.<br/> <br/> Here are the maps fyi:<br/> <br/> 10.2.2.167 : 192.168.0.20 (reb out)<br/> 10.2.2.168 : 192168.0.21 (new node)<br/> 10.2.2.169 : 192.168.0.22 <br/> <br/> btw: info.zip is from 10.2.2.168 The followings are the timings of vbucket checkpoint persistence from the node that is newly added: <br/> <br/> &nbsp;chk_persistence_cmd (1336 total) <br/> &nbsp;&nbsp;&nbsp;&nbsp;16ms - 32ms : ( 0.07%) 1 <br/> &nbsp;&nbsp;&nbsp;&nbsp;32ms - 65ms : ( 0.15%) 1 <br/> &nbsp;&nbsp;&nbsp;&nbsp;65ms - 131ms : ( 0.37%) 3 <br/> &nbsp;&nbsp;&nbsp;&nbsp;131ms - 262ms : ( 0.97%) 8 <br/> &nbsp;&nbsp;&nbsp;&nbsp;262ms - 524ms : ( 13.70%) 170 ################# <br/> &nbsp;&nbsp;&nbsp;&nbsp;524ms - 1s : ( 34.66%) 280 ############################# <br/> &nbsp;&nbsp;&nbsp;&nbsp;1s - 2s : ( 53.22%) 248 ########################## <br/> &nbsp;&nbsp;&nbsp;&nbsp;2s - 4s : ( 68.49%) 204 ##################### <br/> &nbsp;&nbsp;&nbsp;&nbsp;4s - 8s : ( 78.89%) 139 ############## <br/> &nbsp;&nbsp;&nbsp;&nbsp;8s - 16s : ( 95.66%) 224 ####################### <br/> &nbsp;&nbsp;&nbsp;&nbsp;16s - 33s : (100.00%) 58 ###### <br/> <br/> From the above timings, we can easily see they took more than 50minutes overall. <br/> bug-scrub: chiyoung still diagnosing underlying issue Alk,<br/> <br/> I have a quick question on checkpoint persistence. I saw that ep-engine sometimes receives two checkpoint_persistence commands for the same vbucket even though it doesn&#39;t send any timeout tmp_fail response to the ebucketmigrator.<br/> <br/> For example, this is the timings from the node that was newly added for swap rebalance in two node cluster:<br/> <br/> cbstats 10.2.2.168:11210 timings<br/> &nbsp;chk_persistence_cmd (1354 total)<br/> &nbsp;&nbsp;&nbsp;&nbsp;32ms - 65ms : ( 0.22%) 3 <br/> &nbsp;&nbsp;&nbsp;&nbsp;65ms - 131ms : ( 0.59%) 5 <br/> &nbsp;&nbsp;&nbsp;&nbsp;131ms - 262ms : ( 1.92%) 18 #<br/> &nbsp;&nbsp;&nbsp;&nbsp;262ms - 524ms : ( 14.03%) 164 ##############<br/> &nbsp;&nbsp;&nbsp;&nbsp;524ms - 1s : ( 35.08%) 285 #########################<br/> &nbsp;&nbsp;&nbsp;&nbsp;1s - 2s : ( 50.89%) 214 ###################<br/> &nbsp;&nbsp;&nbsp;&nbsp;2s - 4s : ( 71.34%) 277 #########################<br/> &nbsp;&nbsp;&nbsp;&nbsp;4s - 8s : ( 88.04%) 226 ####################<br/> &nbsp;&nbsp;&nbsp;&nbsp;8s - 16s : ( 97.71%) 131 ###########<br/> &nbsp;&nbsp;&nbsp;&nbsp;16s - 33s : (100.00%) 31 ##<br/> <br/> <br/> There were no timeout responses to the ebucketmigrator. In this case, I expect that there would be 1024 checkpoint_persistence commands (512 for active and 512 for replica vbuckets).<br/> <br/> Please let me know if this is still fine and reassign it back to me.<br/> An example of such duplicate checkpoint_persistence requests is for vbucket 37 on 10.2.2.168: <br/> <br/> memcached&lt;0.4538.0&gt;: Mon Nov 12 12:52:59.092677 PST 3: Notified the completion of checkpoint persistence for vbucket 37, cookie 0x5c62000 <br/> ... <br/> memcached&lt;0.4538.0&gt;: Mon Nov 12 12:53:01.096198 PST 3: Notified the completion of checkpoint persistence for vbucket 37, cookie 0x5c63600 collect info for chiyoung&#39;s comments. Reassigned it to me as I was told that it&#39;s an expected behavior. As part of meeting understanding views and rebalance we decided Ronnie will run same rebalance but with no mutations and same rebalance but with rebalance_index_waiting_disabled set to false. To see if that 10x is due to need to persist deltas from ongoing mutations Ronnie just posted on some internal forum that without mutations we&#39;re still 4x slower. I&#39;m still waiting newer diags. <br/> <br/> I also lack information about environment. Looks like this is run on vmware, but any details about this environment are missing Here&#39;s related messages about vbucket 37: <br/> <br/> [rebalance:debug,2012-11-12T12:52:58.562,<a href='mailto:ns_1@192.168.0.21'>ns_1@192.168.0.21</a>:&lt;0.7340.0&gt;:janitor_agent:handle_call:651]Going to wait for persistence of checkpoint 2 in vbucket 37 <br/> [ns_server:info,2012-11-12T12:52:58.572,<a href='mailto:ns_1@192.168.0.21'>ns_1@192.168.0.21</a>:ns_port_memcached&lt;0.4538.0&gt;:ns_port_server:log:171]memcached&lt;0.4538.0&gt;: Mon Nov 12 12:52:58.371396 PST 3: TAP (Consumer) eq_tapq:anon_137 - disconnected <br/> memcached&lt;0.4538.0&gt;: Mon Nov 12 12:52:58.453295 PST 3: TAP (Consumer) eq_tapq:anon_138 - Reset vbucket 37 was completed succecssfully. <br/> <br/> memcached&lt;0.4538.0&gt;: Mon Nov 12 12:52:59.092677 PST 3: Notified the completion of checkpoint persistence for vbucket 37, cookie 0x5c62000 <br/> <br/> I.e. we can see our initial waiting for checkpoint on vbucket 37 completes in about 1 second. That&#39;s were we expect bulk of items to get persisted. After that we create another checkpoint on source and wait for it&#39;s persistence. It&#39;s needed for views consistency. In this case even though we don&#39;t have any views we still do that second checkpoint. But we always expect that second checkpoint to be persisted very quickly because it&#39;ll have only few items to persist. <br/> <br/> Here&#39;s relevant messages: <br/> <br/> [rebalance:debug,2012-11-12T12:53:00.225,<a href='mailto:ns_1@192.168.0.21'>ns_1@192.168.0.21</a>:&lt;0.7363.0&gt;:janitor_agent:handle_call:651]Going to wait for persistence of checkpoint 3 in vbucket 37 <br/> <br/> (hm, we see about 1 seconds delay from ep-engine&#39;s log message to ns_server proceeding to next step, which is potential problem in ns_server or memcached) <br/> <br/> [ns_server:info,2012-11-12T12:53:01.296,<a href='mailto:ns_1@192.168.0.21'>ns_1@192.168.0.21</a>:ns_port_memcached&lt;0.4538.0&gt;:ns_port_server:log:171]memcached&lt;0.4538.0&gt;: Mon Nov 12 12:53:01.096198 PST 3: Notified the completion of checkpoint persistence for vbucket 37, cookie <br/> <br/> We see another 1 second delay. Ronnie mentioned (not on ticket as usual) that load is about 9k ops per second which per vbucket 37 is about 9 items per second. So that&#39;s at most few tens of items that we needed to persist for that second &quot;delta&quot; checkpoint. <br/> Repeated the test (physical cluster), took ~840 seconds,<br/> <br/> cbcollect info stats attached. (including diags)<br/> <br/> Starting rebalance, KeepNodes = [&#39;<a href='mailto:ns_1@10.2.1.63'>ns_1@10.2.1.63</a>&#39;,&#39;<a href='mailto:ns_1@10.2.1.58'>ns_1@10.2.1.58</a>&#39;], EjectNodes = [&#39;<a href='mailto:ns_1@10.2.1.61'>ns_1@10.2.1.61</a>&#39;] I think this is a design defect on the consistence view. I don&#39;t want to adapt the ep-engine flusher anymore at this time. I think we really should revisit the overall design of consistent view. <br/> <br/> For a safer and more reliable workaround, we should disable the consistence view during rebalance if there are no views defined. And test with index_waiting disabled took 2580 seconds. <br/> <br/> Actually there is some something interesting and non-flusher related in in timestamps from latest diags (without ongoing mutations). <br/> <br/> E.g.: <br/> <br/> [rebalance:debug,2012-11-14T11:10:17.702,<a href='mailto:ns_1@10.2.1.58'>ns_1@10.2.1.58</a>:&lt;0.17937.1&gt;:janitor_agent:handle_call:651]Going to wait for persistence of checkpoint 2 in vbucket 511 <br/> [ns_server:info,2012-11-14T11:10:17.899,<a href='mailto:ns_1@10.2.1.58'>ns_1@10.2.1.58</a>:ns_port_memcached&lt;0.2008.0&gt;:ns_port_server:log:171]memcached&lt;0.2008.0&gt;: Wed Nov 14 11:10:17.698884 PST 3: TAP (Consumer) eq_tapq:anon_1537 - Reset vbucket 511 was completed succecssfully. <br/> <br/> [ns_server:debug,2012-11-14T11:10:17.951,<a href='mailto:ns_1@10.2.1.58'>ns_1@10.2.1.58</a>:janitor_agent-default&lt;0.2070.0&gt;:janitor_agent:handle_info:682]Got done message from subprocess: &lt;0.17937.1&gt; (ok) <br/> [ns_server:info,2012-11-14T11:10:18.151,<a href='mailto:ns_1@10.2.1.58'>ns_1@10.2.1.58</a>:ns_port_memcached&lt;0.2008.0&gt;:ns_port_server:log:171]memcached&lt;0.2008.0&gt;: Wed Nov 14 11:10:17.951680 PST 3: Notified the completion of checkpoint persistence for vbucket 511, cookie 0x13fef080 <br/> <br/> [ns_server:info,2012-11-14T11:10:18.656,<a href='mailto:ns_1@10.2.1.58'>ns_1@10.2.1.58</a>:ns_port_memcached&lt;0.2008.0&gt;:ns_port_server:log:171]memcached&lt;0.2008.0&gt;: Wed Nov 14 11:10:18.455993 PST 3: Schedule cleanup of &quot;eq_tapq:anon_1536&quot; <br/> <br/> [rebalance:debug,2012-11-14T11:10:19.365,<a href='mailto:ns_1@10.2.1.58'>ns_1@10.2.1.58</a>:&lt;0.17944.1&gt;:janitor_agent:handle_call:651]Going to wait for persistence of checkpoint 2 in vbucket 511 <br/> [ns_server:debug,2012-11-14T11:10:19.366,<a href='mailto:ns_1@10.2.1.58'>ns_1@10.2.1.58</a>:janitor_agent-default&lt;0.2070.0&gt;:janitor_agent:handle_info:682]Got done message from subprocess: &lt;0.17944.1&gt; (ok) <br/> <br/> We see that we actually get reply quite quickly from ep-engine in both invokations. Second is especially quick as it&#39;s a NOP in this case. But there&#39;s some weird 1 second delay in between those two calls. Only significant activity in between there only stats request for &quot;checkpoint &lt;vbucket-id&gt;&quot; and possible create_new_checkpoint request. <br/> Steve,<br/> <br/> From now, I wouldn&#39;t work on any rebalance slowness issues due to consistent view. As I mentioned, depending on the persistence for view updates is NOT scalable approach.<br/> <br/> Initially, I suggested to stream incoming mutations or takeover items to the indexer through TAP without waiting for persistence, but heard from someone that it won&#39;t make any difference compared with the current approach. However, I recently showed that the current approach is way slow due to the disk slowness and fsync per vbucket.<br/> <br/> I don&#39;t expect that we will change the current architecture in post 2.0, and am not interested in giving any suggestions to people who are so stubborn.<br/> <br/> Again, my suggestion is to disable the consistent view if there are no views defined.<br/> <br/> Please assign it to people who is willing to work on this issue.<br/> <br/> In post 2.0, I will remove this prioritization because it&#39;s not a good approach, but instead workaround. I will send a notice to the cluster and XDCR teams after 2.0.<br/> Chiyoung made more changes in ep-engine flusher priority with toy-build. 4x faster. But more agressive flusher can likely lead to more starvation of writes. Ronnie, reassigning to you to please rerun tests with the Chiyoung&#39;s toy-build, and help cover Pavel while he&#39;s traveling.<br/> <br/> From Chiyoung&#39;s email...<br/> <br/> Ronnie, Pavel,<br/> <br/> I made more changes in ep-engine, so that the flusher can work on persisting high priority vbuckets much more aggressively. In my tests, I observed ~ 4X faster rebalance.<br/> <br/> However, my main concern on this new change is that it might cause the starvation on flushing regular vbuckets, and consequently grow the number of dirty items on those regular vbuckets (i.e., disk write queue size might grow a lot especially in XDCR tests because we prioritize flushing 32 vbuckets once every 30 minutes)<br/> <br/> You can download the toy build from<br/> <br/> <a href="http://builds.hq.northscale.net/latestbuilds/couchbase-server-community_toy-chiyoung-x86_64_2.0.0-2002-toy.rpm">http://builds.hq.northscale.net/latestbuilds/couchbase-server-community_toy-chiyoung-x86_64_2.0.0-2002-toy.rpm</a><br/> <br/> Ronnie, can you please test it with the swap rebalance?<br/> <br/> Pavel, can you test it with the XDCR?<br/> <br/> Thanks,<br/> Chiyoung priority to critical, per bug-scrub, but still want more larger scale test runs (more nodes &amp; more items). <br/> <br/> Ronnie, <br/> please send a test case to Tony for larger scale retest, and please reassign. <br/> thanks, <br/> steve <br/> test 1:<br/> <br/> - 8 nodes, swap 1<br/> - 20M items. <br/> - 30 clients with 500 ops per second (set:get = 50:50)<br/> <br/> test 2:<br/> - 8 nodes, swap 3<br/> - 20M items.<br/> - 30 clients with 500 ops per second See also: <a href="http://www.couchbase.com/issues/browse/CBD-682">http://www.couchbase.com/issues/browse/CBD-682</a> [global] <br/> #username:root <br/> username:root <br/> password:couchbase <br/> port:8091 <br/> <br/> data_path=/data <br/> <br/> [servers] <br/> 1:10.6.2.37 <br/> 2:10.6.2.38 <br/> 3:10.6.2.39 <br/> 4:10.6.2.40 <br/> 5:10.6.2.42 <br/> 6:10.6.2.43 <br/> 7:10.6.2.44 <br/> 8:10.6.2.45 <br/> <br/> #9:10.3.121.24 <br/> #10:10.3.121.25 <br/> #11:10.3.121.26 <br/> #12:10.3.121.27 <br/> <br/> [membase] <br/> rest_username:Administrator <br/> rest_password:password <br/> <br/> <br/> This is the cluster information for orange cluster. Ronnie you can run the test on them. due to the cluster limitations: <br/> <br/> change test plan to: <br/> <br/> 1) 4 node swap 3. <br/> <br/> 2) 6 nodes swap 1. <br/> <br/> <br/> <br/> <br/> bug-scrub - moved to 2.0.1 Early results for larger number of nodes swap rebalance (swap 3 nodes from a 4 node cluster). <br/> <br/> No foreground load case looks promising. <br/> <br/> 1.8.1-938: 988.58 seconds. <br/> 2.0.0- 1954: 974.67 seconds. <br/> 4-3 swap rebalance: <br/> <br/> without foreground load (1x identical): <br/> <br/> 1.8.1-938: 988.58 seconds. <br/> 2.0.0- 1954: 974.67 seconds. <br/> <br/> with ~10k foreground (7x slower): <br/> <br/> 1.8.1-938: 222 sec <br/> 2.0.0- 1954: 1723 sec <br/> <br/> 6-1 swap rebalance: <br/> <br/> with ~10k foreground (1x identical): <br/> 1.8.1-938: 639.52 sec <br/> 2.0.0- 1954: 654.22 sec <br/> Based on 6-1 results (started with 6 nodes, swap 1 node):<br/> <br/> 2.0 has similar reb time as 1.8.1, while latencies are ~30% worse.<br/> <br/> <br/> <a href="https://www.yammer.com/couchbase.com/#/Threads/show?threadId=236527848">https://www.yammer.com/couchbase.com/#/Threads/show?threadId=236527848</a> <br/> <br/> 4-3 swap rebalance: <br/> <br/> without foreground load (1x identical): <br/> <br/> 1.8.1-938: 988.58 seconds. <br/> 2.0.0- 1954: 974.67 seconds. <br/> <br/> with ~10k foreground (7x slower): <br/> <br/> 1.8.1-938: 222 sec <br/> 2.0.0- 1954: 1723 sec <br/> <br/> 6-1 swap rebalance: <br/> <br/> with ~10k foreground (1x identical): <br/> 1.8.1-938: 639.52 sec <br/> 2.0.0- 1954: 654.22 sec <br/> Like &#xB7; Reply &#xB7; Share &#xB7; More &#xB7; 13 minutes ago <br/> &nbsp; <br/> Ronnie Sun: Based on 6-1 results (started with 6 nodes, swap 1 node): <br/> <br/> 2.0 has similar reb time as 1.8.1, while latencies are ~30% worse. <br/> <br/> reb-swap-6-1.loop_1.8.1-938-rel-enterprise_2.0.0-1954-rel-enterprise_orange_Nov-21-2012_10-07-08 <br/> &nbsp; <br/> Steve Yen: @Aliaksey Kandratsenka, @Chiyoung Seo <br/> <br/> &quot;4-3 swap rebalance&quot; == 4 nodes initially and swap 3 of them. <br/> <br/> &quot;6-1 swap rebalance&quot; == 6 nodes initially and swap 1 of them. <br/> <br/> @Ronnie Sun also reports latencies were worse in 2.0 during swap rebalance. <br/> <br/> This seems to indicate that we don&#39;t need @Chiyoung Seo&#39;s patch that prioritizes the vbucket takeover even more. <br/> <br/> This was on system test cluster (xen VM&#39;s) with SSD, key-value mixed workload (no views), with consistent views enabled (the @Chiyoung Seo . No views defined, so updating component list. No views defined, so updating component list. Date of First Response Fri, 9 Nov 2012 20:56:01 -0600 Rank 3563 Time In Status