http://www.couchbase.com/issues This file is an XML representation of an issue en-us 5.2.4 845 26-12-2012 http://www.couchbase.com/issues/browse/MB-7587 Couchbase Server Setup a cluster as above. <br/> Create 2 views per bucket. <br/> Run time : 10 hour+ <br/> <br/> Seeing uneven compaction/ high doc fragmentation across the nodes. ( Screenshot attached below) <br/> <br/> Seeing very high swap on these nodes as well ( Screenshot below) <br/> <br/> * Indexing / Compaction is running continuosly, but never completes. <br/> <br/> Adding logs. <br/> <br/> * Does this high doc fragmentation(70-80 percent) indicate that compaction did not work as expected? <br/> Is this something we ve seen before/ know of? <br/> <br/> Please let me know if you need any other system information on this. <br/> <br/> 2.0.1-140 build <br/> <br/> 1024 vbuckets <br/> &nbsp;7 node ssd cluster, <br/> default and sasl buckets with 150M and 53M items. <br/> 2 views per bucket, 1 ddoc each <br/> <br/> * Node are in DGM( 30 percent, 83 percent) an, and heavy swap. <br/> <br/> Link to the cluster <a href="http://10.6.2.38:8091/">http://10.6.2.38:8091/</a> MB-7587

System Testing : Uneven compaction + Very high swap, over a cluster, some nodes have much higher (70 percent ) fragmentation than others.

Bug Major Resolved Fixed Public Ketaki Gangal Ketaki Gangal Wed, 23 Jan 2013 11:05:36 -0600 Mon, 28 Jan 2013 13:48:04 -0600 Mon, 28 Jan 2013 13:37:20 -0600 2.0.1 2.0.1 test-execution 0 2 Atop from node : 10.6.2.37 shows memcached is in 2G swap. I can look at the logs once they&#39;re available but keep in mind: <br/> <br/> 1) memcached is totally unrelated to view engine; <br/> <br/> 2) doc fragmentation (vbuckets), is unrelated to view engine as well. <br/> <br/> So I think it&#39;s premature to classify the view engine as the source of the problem. Further, it&#39;s continuous indexing and compaction is not necessarily bad. If there&#39;s data being loaded continuously and queries, continuous indexing is expected. The progress bar you see in the UI might not be related to the same indexer run. For very small indexing iterations, the UI might give an idea of staleness. <br/> <br/> Anyway, waiting for the logs to tell what&#39;s going on. yes. assigning this back to QE as there is not enough information available <br/> <br/> also doc fragmentation is related to compaction engine Thanks for the input Filipe. <br/> <br/> Yes - I agree this may not be a view issue. But i ve seen some system unusual behavior after I added views :) and hence the tagging. <br/> <br/> I ve currently assigned this bug to myself. And will post the logs once I am on a slightly faster n/w ( @MV). <br/> <br/> memcached in higher swap is something we are aware of. [ seen after the change on &#39;mlock_all of beam pages&#39;] <br/> Compaction not catching up on some nodes needs to be looked at, and this would be at a couchstore layer. <br/> Well it depends, I would need the logs to see if there&#39;s any problem with view indexing and/or compaction on that specific node (from what I understand this is being a problem in a single node right?). <br/> <br/> To my knowledge there hasn&#39;t been any for some a good period of time. <br/> <br/> thanks Ketaki cbcollect_info from node 10.6.242 ( high doc fragmentation on this node) Logs from all the nodes <a href="https://s3.amazonaws.com/bugdb/7587/7587.tar">https://s3.amazonaws.com/bugdb/7587/7587.tar</a> Looking at log from node .42, the one with largest total file size: <br/> <br/> <br/> [ns_server:info,2013-01-23T0:10:57.058,<a href='mailto:ns_1@10.6.2.42'>ns_1@10.6.2.42</a>:&lt;0.16991.0&gt;:compaction_daemon:spawn_dbs_compactor:690]Compacting databases for bucket default <br/> [ns_server:info,2013-01-23T0:10:57.059,<a href='mailto:ns_1@10.6.2.42'>ns_1@10.6.2.42</a>:&lt;0.16992.0&gt;:compaction_daemon:spawn_vbucket_compactor:793]Compacting &lt;&lt;&quot;default/master&quot;&gt;&gt; <br/> [couchdb:error,2013-01-23T0:10:57.085,<a href='mailto:ns_1@10.6.2.42'>ns_1@10.6.2.42</a>:&lt;0.16993.0&gt;:couch_log:error:42]Native compact for &quot;default/master&quot; failed due to error {exit_status,139}. Falling back to erlang. <br/> [error_logger:error,2013-01-23T0:10:57.087,<a href='mailto:ns_1@10.6.2.42'>ns_1@10.6.2.42</a>:error_logger&lt;0.6.0&gt;:ale_error_logger_handler:log_msg:76]Error in process &lt;0.16993.0&gt; on node &#39;<a href='mailto:ns_1@10.6.2.42'>ns_1@10.6.2.42</a>&#39; with exit value: {{badmatch,{error,no_valid_header}},[{couch_db_updater,start_copy_compact,2}]} <br/> <br/> <br/> [ns_server:warn,2013-01-23T0:10:57.087,<a href='mailto:ns_1@10.6.2.42'>ns_1@10.6.2.42</a>:&lt;0.16991.0&gt;:compaction_daemon:do_chain_compactors:665]Compactor for vbucket `default/master` (pid [{type,vbucket}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{name,&lt;&lt;&quot;default/master&quot;&gt;&gt;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{important,false}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{fa, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{#Fun&lt;compaction_daemon.17.36422344&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&lt;&lt;&quot;default&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{master,&lt;&lt;&quot;default/master&quot;&gt;&gt;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[]]}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{on_success, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{#Fun&lt;compaction_daemon.18.44611183&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[1,293]}}]) terminated unexpectedly (ignoring this): {{badmatch, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{error, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;no_valid_header}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{couch_db_updater, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;start_copy_compact, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;2}]} <br/> [ns_server:info,2013-01-23T0:10:57.088,<a href='mailto:ns_1@10.6.2.42'>ns_1@10.6.2.42</a>:&lt;0.16991.0&gt;:compaction_daemon:spawn_dbs_compactor:730]Finished compaction of databases for bucket default <br/> [error_logger:error,2013-01-23T0:10:57.088,<a href='mailto:ns_1@10.6.2.42'>ns_1@10.6.2.42</a>:error_logger&lt;0.6.0&gt;:ale_error_logger_handler:log_report:72] <br/> =========================CRASH REPORT========================= <br/> &nbsp;&nbsp;crasher: <br/> &nbsp;&nbsp;&nbsp;&nbsp;initial call: compaction_daemon:-spawn_vbucket_compactor/3-fun-0-/0 <br/> &nbsp;&nbsp;&nbsp;&nbsp;pid: &lt;0.16992.0&gt; <br/> &nbsp;&nbsp;&nbsp;&nbsp;registered_name: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;exception exit: {{badmatch,{error,no_valid_header}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{couch_db_updater,start_copy_compact,2}]} <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in function compaction_daemon:&#39;-spawn_vbucket_compactor/3-fun-0-&#39;/5 <br/> &nbsp;&nbsp;&nbsp;&nbsp;ancestors: [&lt;0.16991.0&gt;,&lt;0.16989.0&gt;,&lt;0.16988.0&gt;,compaction_daemon, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;0.2488.0&gt;,ns_server_sup,ns_server_cluster_sup,&lt;0.58.0&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;messages: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;links: [&lt;0.16991.0&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;dictionary: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;trap_exit: true <br/> &nbsp;&nbsp;&nbsp;&nbsp;status: running <br/> &nbsp;&nbsp;&nbsp;&nbsp;heap_size: 2584 <br/> &nbsp;&nbsp;&nbsp;&nbsp;stack_size: 24 <br/> &nbsp;&nbsp;&nbsp;&nbsp;reductions: 1084 <br/> &nbsp;&nbsp;neighbours: <br/> <br/> [ns_server:debug,2013-01-23T0:10:57.088,<a href='mailto:ns_1@10.6.2.42'>ns_1@10.6.2.42</a>:&lt;0.16999.0&gt;:compaction_daemon:file_needs_compaction:985]Estimated size for `default/_design/dev_ddoc1/main`: data 0, file 0 <br/> [error_logger:error,2013-01-23T0:10:57.088,<a href='mailto:ns_1@10.6.2.42'>ns_1@10.6.2.42</a>:error_logger&lt;0.6.0&gt;:ale_error_logger_handler:log_msg:76]** Generic server &lt;0.16994.0&gt; terminating <br/> ** Last message in was {&#39;EXIT&#39;,&lt;0.16993.0&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{{badmatch,{error,no_valid_header}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{couch_db_updater,start_copy_compact,2}]}} <br/> ** When Server state == {file,&lt;0.16995.0&gt;,&lt;0.16996.0&gt;,0} <br/> ** Reason for termination == <br/> ** {{badmatch,{error,no_valid_header}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;[{couch_db_updater,start_copy_compact,2}]} <br/> <br/> <br/> <br/> There&#39;s indeed a problem with database compaction. <br/> Couchstore&#39;s vbucket compactor exited with status 139 (seg fault). <br/> <br/> Check with Aaron. And that results in the compaction daemon (ns_server) to fail on each compaction iteration (not proceeding to other vbuckets and indexes). Each iteration fails always with the same error when attempting to compact default/master vbucket. Looking at the logs of all nodes, neither indexing nor view compaction got any errors or got stuck. <br/> <br/> In the past (before beta), view compaction retry phase had many iterations when there was load in parallel (sometimes hundreds of iterations in the extreme). It&#39;s not the case anymore, and confirmed in the logs: <br/> <br/> fdmanana 21:24:59 ~/jira/<a href="http://www.couchbase.com/issues/browse/MB-7587" title="System Testing : Uneven compaction + Very high swap, over a cluster, some nodes have much higher (70 percent ) fragmentation than others."><strike>MB-7587</strike></a>/7587 &gt; egrep &#39;retry number&#39; * | cut -d &#39; &#39; -f 16 | tr -d &#39;)&#39; | sort -n | uniq -c <br/> &nbsp;&nbsp;&nbsp;&nbsp;473 1 <br/> &nbsp;&nbsp;&nbsp;&nbsp;180 2 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;43 3 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;17 4 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;12 5 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;7 6 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6 7 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6 8 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4 9 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;3 10 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1 11 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1 12 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1 13 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1 14 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;1 15 <br/> fdmanana 21:29:36 ~/jira/<a href="http://www.couchbase.com/issues/browse/MB-7587" title="System Testing : Uneven compaction + Very high swap, over a cluster, some nodes have much higher (70 percent ) fragmentation than others."><strike>MB-7587</strike></a>/7587 &gt; <br/> <br/> <br/> Indexing compaction times histogram (for all nodes, all ddocs), values in seconds: <br/> <br/> fdmanana 21:30:01 ~/jira/<a href="http://www.couchbase.com/issues/browse/MB-7587" title="System Testing : Uneven compaction + Very high swap, over a cluster, some nodes have much higher (70 percent ) fragmentation than others."><strike>MB-7587</strike></a>/7587 &gt; egrep &#39;compaction complete in&#39; * | cut -d &#39; &#39; -f 10 | perl -MStatistics::Histogram -e &#39;@data = &lt;&gt;; chomp @data; print get_histogram(\@data);&#39; <br/> Count: 684 <br/> Range: 0.746 - 6905.309; Mean: 183.588; Median: 50.588; Stddev: 523.667 <br/> Percentiles: 90th: 332.315; 95th: 533.129; 99th: 2635.922 <br/> &nbsp;&nbsp;&nbsp;0.746 - 2.997: 39 ############### <br/> &nbsp;&nbsp;&nbsp;2.997 - 8.151: 72 ########################### <br/> &nbsp;&nbsp;&nbsp;8.151 - 19.951: 100 ##################################### <br/> &nbsp;&nbsp;19.951 - 46.965: 124 ############################################## <br/> &nbsp;&nbsp;46.965 - 108.811: 108 ######################################## <br/> &nbsp;108.811 - 250.400: 142 ##################################################### <br/> &nbsp;250.400 - 574.553: 65 ######################## <br/> &nbsp;574.553 - 1316.667: 15 ###### <br/> 1316.667 - 3015.656: 14 ##### <br/> 3015.656 - 6905.309: 5 ## <br/> fdmanana 21:30:05 ~/jira/<a href="http://www.couchbase.com/issues/browse/MB-7587" title="System Testing : Uneven compaction + Very high swap, over a cluster, some nodes have much higher (70 percent ) fragmentation than others."><strike>MB-7587</strike></a>/7587 &gt; <br/> <br/> <br/> Indexing times histogram (again, all nodes and all ddocs), values in seconds: <br/> <br/> fdmanana 21:29:50 ~/jira/<a href="http://www.couchbase.com/issues/browse/MB-7587" title="System Testing : Uneven compaction + Very high swap, over a cluster, some nodes have much higher (70 percent ) fragmentation than others."><strike>MB-7587</strike></a>/7587 &gt; egrep &#39;Indexing time:&#39; * | cut -d &#39; &#39; -f 3 | perl -MStatistics::Histogram -e &#39;@data = &lt;&gt;; chomp @data; print get_histogram(\@data);&#39; <br/> Count: 2242 <br/> Range: 0.566 - 9082.209; Mean: 70.312; Median: 21.408; Stddev: 255.755 <br/> Percentiles: 90th: 160.939; 95th: 282.844; 99th: 695.317 <br/> &nbsp;&nbsp;&nbsp;0.566 - 2.725: 13 # <br/> &nbsp;&nbsp;&nbsp;2.725 - 7.861: 499 ############################################## <br/> &nbsp;&nbsp;&nbsp;7.861 - 20.078: 552 ################################################## <br/> &nbsp;&nbsp;20.078 - 49.138: 581 ##################################################### <br/> &nbsp;&nbsp;49.138 - 118.266: 304 ############################ <br/> &nbsp;118.266 - 282.701: 180 ################ <br/> &nbsp;282.701 - 673.847: 88 ######## <br/> &nbsp;673.847 - 1604.276: 19 ## <br/> 1604.276 - 3817.515: 5 | <br/> 3817.515 - 9082.209: 1 | <br/> fdmanana 21:30:42 ~/jira/<a href="http://www.couchbase.com/issues/browse/MB-7587" title="System Testing : Uneven compaction + Very high swap, over a cluster, some nodes have much higher (70 percent ) fragmentation than others."><strike>MB-7587</strike></a>/7587 &gt; <br/> <br/> <br/> The fact that you see indexing happening all the time is indeed due to queries happening all the time, each triggering an index update that processes a small number of changes, whence likely hard to know via the UI if the progress bar relates to different indexer runs or the same - there were 2242 index updates. See compaction crash in windows cluster with build 2.0.1-140 <br/> <br/> Environment: <br/> * 9 windows 2008 R2 64bit. <br/> * Each server has 4 CPU, 8GB RAM and SSD disk <br/> * Cluster has 2 buckets, default and sasl bucket with consistent view enable. <br/> * Load 26 million items to default bucket and 16 million items to sasl bucket. Each key has size from 128 to 512 bytes <br/> * Each bucket has one doc and 2 views for each doc. <br/> <br/> - Rebalance out 2 nodes 10.3.121.173 and 10.3.121.243 <br/> <br/> Starting rebalance, KeepNodes = [&#39;<a href='mailto:ns_1@10.3.3.181'>ns_1@10.3.3.181</a>&#39;,&#39;<a href='mailto:ns_1@10.3.121.47'>ns_1@10.3.121.47</a>&#39;, <br/> &#39;<a href='mailto:ns_1@10.3.3.214'>ns_1@10.3.3.214</a>&#39;,&#39;<a href='mailto:ns_1@10.3.3.182'>ns_1@10.3.3.182</a>&#39;, <br/> &#39;<a href='mailto:ns_1@10.3.3.180'>ns_1@10.3.3.180</a>&#39;,&#39;<a href='mailto:ns_1@10.3.121.171'>ns_1@10.3.121.171</a>&#39;, <br/> &#39;<a href='mailto:ns_1@10.3.121.169'>ns_1@10.3.121.169</a>&#39;], EjectNodes = [&#39;<a href='mailto:ns_1@10.3.121.173'>ns_1@10.3.121.173</a>&#39;, <br/> &#39;<a href='mailto:ns_1@10.3.121.243'>ns_1@10.3.121.243</a>&#39;] ns_orchestrator004 <a href='mailto:ns_1@10.3.121.169'>ns_1@10.3.121.169</a> 23:26:03 - Tue Jan 22, 2013 <br/> <br/> <br/> <br/> =========================CRASH REPORT========================= <br/> &nbsp;&nbsp;crasher: <br/> &nbsp;&nbsp;&nbsp;&nbsp;initial call: compaction_daemon:-spawn_view_index_compactor/6-fun-0-/0 <br/> &nbsp;&nbsp;&nbsp;&nbsp;pid: &lt;0.29697.56&gt; <br/> &nbsp;&nbsp;&nbsp;&nbsp;registered_name: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;exception exit: {{badmatch, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{error, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{file_error, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;c:/view/@indexes/default/tmp_40ae216d3de3826de065e665ac0f52dd_replica/d8004c67163a37e21f8c542b9ffa9c05.compact&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;enoent}}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{couch_set_view_compactor,merge_files,3}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_set_view_compactor,apply_log,4}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_set_view_compactor,maybe_retry_compact,5}]} <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in function compaction_daemon:do_spawn_view_index_compactor/5 <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in call from compaction_daemon:&#39;-spawn_view_index_compactor/6-fun-0-&#39;/7 <br/> &nbsp;&nbsp;&nbsp;&nbsp;ancestors: [&lt;0.12760.55&gt;,&lt;0.16968.54&gt;,&lt;0.16931.54&gt;,compaction_daemon, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;0.152.1&gt;,ns_server_sup,ns_server_cluster_sup,&lt;0.67.0&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;messages: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;links: [&lt;0.12760.55&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;dictionary: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;trap_exit: true <br/> &nbsp;&nbsp;&nbsp;&nbsp;status: running <br/> &nbsp;&nbsp;&nbsp;&nbsp;heap_size: 4181 <br/> &nbsp;&nbsp;&nbsp;&nbsp;stack_size: 24 <br/> &nbsp;&nbsp;&nbsp;&nbsp;reductions: 2873 <br/> &nbsp;&nbsp;neighbours: <br/> <br/> [error_logger:error,2013-01-23T5:02:44.219,<a href='mailto:ns_1@10.3.121.169'>ns_1@10.3.121.169</a>:error_logger&lt;0.6.0&gt;:ale_error_logger_handler:log_msg:76]Error in process &lt;0.24289.57&gt; on node &#39;<a href='mailto:ns_1@10.3.121.169'>ns_1@10.3.121.169</a>&#39; with exit value: {{badmatch,{error,{file_error,&quot;c:/view/@indexes/default/tmp_40ae216d3de3826de065e665ac0f52dd_replica/d8004c67163a37e21f8c542b9ffa9c05.compact&quot;,enoent}}},[{couch_set_view_compactor,merge_files,3},{couch_set_view_compactor... <br/> <br/> <br/> [error_logger:error,2013-01-23T5:02:44.219,<a href='mailto:ns_1@10.3.121.169'>ns_1@10.3.121.169</a>:error_logger&lt;0.6.0&gt;:ale_error_logger_handler:log_report:72] <br/> =========================CRASH REPORT========================= <br/> &nbsp;&nbsp;crasher: <br/> &nbsp;&nbsp;&nbsp;&nbsp;initial call: compaction_daemon:-spawn_view_compactor/5-fun-1-/0 <br/> &nbsp;&nbsp;&nbsp;&nbsp;pid: &lt;0.12760.55&gt; <br/> &nbsp;&nbsp;&nbsp;&nbsp;registered_name: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;exception exit: {{badmatch, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{error, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{file_error, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;c:/view/@indexes/default/tmp_40ae216d3de3826de065e665ac0f52dd_replica/d8004c67163a37e21f8c542b9ffa9c05.compact&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;enoent}}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{couch_set_view_compactor,merge_files,3}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_set_view_compactor,apply_log,4}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_set_view_compactor,maybe_retry_compact,5}]} <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in function compaction_daemon:do_chain_compactors/2 <br/> &nbsp;&nbsp;&nbsp;&nbsp;ancestors: [&lt;0.16968.54&gt;,&lt;0.16931.54&gt;,compaction_daemon,&lt;0.152.1&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ns_server_sup,ns_server_cluster_sup,&lt;0.67.0&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;messages: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;links: [&lt;0.16968.54&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;dictionary: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;trap_exit: true <br/> &nbsp;&nbsp;&nbsp;&nbsp;status: running <br/> &nbsp;&nbsp;&nbsp;&nbsp;heap_size: 1597 <br/> &nbsp;&nbsp;&nbsp;&nbsp;stack_size: 24 <br/> &nbsp;&nbsp;&nbsp;&nbsp;reductions: 4965 <br/> &nbsp;&nbsp;neighbours: <br/> <br/> <br/> Links to collect info of all nodes <a href="https://s3.amazonaws.com/packages.couchbase/collect_info/2_0_1/201301/9nodes-col-201-140-rebalance-failed-buckets-shutdown-20130123-14-11-14.tgz">https://s3.amazonaws.com/packages.couchbase/collect_info/2_0_1/201301/9nodes-col-201-140-rebalance-failed-buckets-shutdown-20130123-14-11-14.tgz</a> Tony, that&#39;s a totally different issue from Ketaki&#39;s and on a different environment (windows). <br/> Lets not mix everything here. Please create a separate issue for that and assign it to me for now. Sure, I will create the new bug. If we have compaction crashes I&#39;ll at least need stack traces or cores to have any hope of diagnosing them. Found the cores. <br/> <br/> It looks like tmpfile() is returning NULL on this machine, looks like error # 13, permission denied. Could be caused by Couchbase not having permission to create files in /tmp. <br/> Will do a fix to handle this (the compactor will still not work, It&#39;ll just exit with an error that&#39;s not a segfault). <br/> <br/> To make sure this doesn&#39;t happen, the directory tmpfile() creates file in (typically /tmp) needs to be writable by the user couchbase runs as. Per bug-scrub: After bug is merged, QE will re-run Ketaki to rerun the test with build 144+ I don&#39;t think any fix was done here. <br/> Couchstore&#39;s compactor was only changed to exit with a status other than 139 (seg fault). From above, after database compaction failed with couchstore, the Erlang based compactor, which it fallbacks to, also failed: <br/> <br/> [couchdb:error,2013-01-23T0:10:57.085,<a href='mailto:ns_1@10.6.2.42'>ns_1@10.6.2.42</a>:&lt;0.16993.0&gt;:couch_log:error:42]Native compact for &quot;default/master&quot; failed due to error {exit_status,139}. Falling back to erlang. <br/> [error_logger:error,2013-01-23T0:10:57.087,<a href='mailto:ns_1@10.6.2.42'>ns_1@10.6.2.42</a>:error_logger&lt;0.6.0&gt;:ale_error_logger_handler:log_msg:76]Error in process &lt;0.16993.0&gt; on node &#39;<a href='mailto:ns_1@10.6.2.42'>ns_1@10.6.2.42</a>&#39; with exit value: {{badmatch,{error,no_valid_header}},[{couch_db_updater,start_copy_compact,2}]} <br/> <br/> It needs to be analyzed why master vbucket compaction fails (apparently empty file or non-empty but no header written, as it has at least 1 doc). Date of First Response Wed, 23 Jan 2013 11:10:56 -0600 Rank 3321 Sprint Status Time In Status