Couchbase Server 2.0 - Dashboard Bugs(minus docs) (Couchbase) http://www.couchbase.com/issues/secure/IssueNavigator.jspa?requestId=10363 en-us 5.2.4 845 26-12-2012 [MB-8210] Update the incorrect information on the Automated Index Updates http://www.couchbase.com/issues/browse/MB-8210 Couchbase Server We need to fix this page <a href="http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-views-operation-autoupdate.html">http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-views-operation-autoupdate.html</a> ? <br/> <br/> Relevant information about how it actually works. <br/> <br/> &quot;Every updateInterval milliseconds it checks if index file is more than updateMinChanges behind .couch files (which is itself behind in-memory source of truth, potentially for tens of seconds). And true, it triggers view update.&quot; MB-8210 Update the incorrect information on the Automated Index Updates Bug Blocker Open Unresolved Public Karen Zeller Anil Kumar Tue, 7 May 2013 18:20:49 -0500 Tue, 21 May 2013 14:47:03 -0500 2.0 2.0.1 2.0.2 2.0.2 documentation 0 1 Rank 95 Sprint Status [MB-8002] Memcached ops sometimes take an extraordinarily long amount of time http://www.couchbase.com/issues/browse/MB-8002 Couchbase Server We have noticed that memcached ops sometimes take a very long time to complete and can cause ns_server to think that memcached has become unresponsive. We usually see this when a system starts to swap, but we have also occasionally seen this issue even when we&#39;re not in swap. MB-8002 Memcached ops sometimes take an extraordinarily long amount of time Bug Blocker Open Unresolved Public Mike Wiederhold Mike Wiederhold Mon, 1 Apr 2013 19:26:18 -0500 Mon, 1 Apr 2013 19:30:51 -0500 2.0 2.1 couchbase-bucket 0 3 See <a href="http://www.couchbase.com/issues/browse/MB-7111" title="[system test] rebalance failed with error &quot;wait_checkpoint_persisted_failed&quot; due to timeout"><strike>MB-7111</strike></a> which is a duplicate of this issue. Rank 10330 [MB-7282] erlang's global naming facility apparently drops globally registered service with actual service still alive (was: impossible to change settings/autoFailover after rebalance) http://www.couchbase.com/issues/browse/MB-7282 Couchbase Server build <br/> ./testrunner -i resources/jenkins/centos-64-5node-failover.ini get-logs=True -t autofailovertests.AutoFailoverTests.test_enable,replicas=2,keys-count=1000000,num-buckets=2 <br/> <a href="http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/480/consoleFull">http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/480/consoleFull</a> <br/> <br/> test changes settings/autoFailover after rebalance <br/> <br/> test logs: <br/> <br/> 2012-11-27 22:29:42,105 - INFO - MainProcess:MainThread - rest_client:_rebalance_progress - rebalance percentage : 99.5678770149 % <br/> 2012-11-27 22:29:44,114 - INFO - MainProcess:MainThread - rest_client:_rebalance_progress - rebalance percentage : 99.6821896236 % <br/> 2012-11-27 22:29:48,134 - INFO - MainProcess:MainThread - rest_client:monitorRebalance - rebalance progress took 1111.69762397 seconds <br/> 2012-11-27 22:29:48,135 - INFO - MainProcess:MainThread - rest_client:monitorRebalance - sleep for 10 seconds after rebalance... <br/> 2012-11-27 22:29:58,133 - INFO - MainProcess:MainThread - rest_client:update_autofailover_settings - settings/autoFailover params : enabled=true&amp;timeout=30 <br/> 2012-11-27 22:29:58,140 - ERROR - MainProcess:MainThread - rest_client:_http_request - <a href="http://10.1.3.114:8091/settings/autoFailover">http://10.1.3.114:8091/settings/autoFailover</a> error 500 reason: unknown [&quot;Unexpected server error, request logged.&quot;] <br/> <br/> server logs: <br/> <br/> [menelaus:warn,2012-11-27T22:53:28.404,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:&lt;0.21541.23&gt;:menelaus_web:loop:430]Server error during processing: [&quot;web request failed&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{path,&quot;/settings/autoFailover&quot;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{type,exit}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{what, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{noproc, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server,call, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{global,auto_failover}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{enable_auto_failover,30,1}]}}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{trace, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{gen_server,call,2}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{menelaus_web, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;handle_settings_auto_failover_post,1}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{menelaus_web,loop,3}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{mochiweb_http,headers,5}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{proc_lib,init_p_do_apply,3}]}] <br/> <br/> Alk, if we have to handle the error and retry again, please assign the ticket back to me <br/> <br/> MB-7282 erlang's global naming facility apparently drops globally registered service with actual service still alive (was: impossible to change settings/autoFailover after rebalance) Bug Blocker Open Unresolved Public Aleksey Kondratenko Andrei Baranouski Wed, 28 Nov 2012 08:56:06 -0600 Thu, 2 May 2013 17:31:37 -0500 2.0 2.1 ns_server 0 3 <a href="https://s3.amazonaws.com/bugdb/jira/MB-7282/350d5885-93c9-4497-bc64-fd5c65d4c162-10.1.3.114-diag.txt.gz">https://s3.amazonaws.com/bugdb/jira/MB-7282/350d5885-93c9-4497-bc64-fd5c65d4c162-10.1.3.114-diag.txt.gz</a> does it work when you retry ? per bug-scrub, assigning back to andrei.<br/> <br/> please try to reproduce it again (manually), and does it succeed if you retry to operation? I was not able to reproduce it manually, even with constant change autoFailover settings through script during the test. but the same problem was in 2.0.0-1965-rel run <a href="http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/476/consoleFull">http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/476/consoleFull</a> per-bug-scrub - moved to 2.0.1. <br/> <br/> apparently hard to reproduce manually. deferring to 2.1 per bug scrub meeting ( Dipti &amp; Farshid -December 7th ) reproduced again in some test against 2.0.1-103-rel build: <br/> <br/> <a href="http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/491/consoleFull">http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/491/consoleFull</a> Alk, if you have a chance , could you look today on 10.1.3.114 from the test, maybe this will help to quickly identify the problem: <br/> <br/> resetCount( or update settings/autoFailover) started giving an 500 error after a certain point in the tests <br/> <br/> <br/> &nbsp;curl -v --user Administrator:password -X POST <a href="http://10.1.3.114:8091/settings/autoFailover/resetCount">http://10.1.3.114:8091/settings/autoFailover/resetCount</a> <br/> * About to connect() to 10.1.3.114 port 8091 (#0) <br/> * Trying 10.1.3.114... connected <br/> * Connected to 10.1.3.114 (10.1.3.114) port 8091 (#0) <br/> * Server auth using Basic with user &#39;Administrator&#39; <br/> &gt; POST /settings/autoFailover/resetCount HTTP/1.1 <br/> &gt; Authorization: Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA== <br/> &gt; User-Agent: curl/7.21.4 (x86_64-unknown-linux-gnu) libcurl/7.21.4 OpenSSL/1.0.1 zlib/1.2.3.4 <br/> &gt; Host: 10.1.3.114:8091 <br/> &gt; Accept: */* <br/> &gt; <br/> &lt; HTTP/1.1 500 Internal Server Error <br/> &lt; Server: Couchbase Server 2.0.1-103-rel-enterprise <br/> &lt; Pragma: no-cache <br/> &lt; Date: Mon, 17 Dec 2012 10:14:05 GMT <br/> &lt; Content-Type: application/json <br/> &lt; Content-Length: 44 <br/> &lt; Cache-Control: no-cache <br/> &lt; <br/> * Connection #0 to host 10.1.3.114 left intact <br/> * Closing connection #0 <br/> Indeed a bug.<br/> <br/> Looks like you rebalanced out master. And new master needs some time to be elected. Thus autofailover service is indeed not running anywhere. As workaround I suggest you to wait a bit and retry. Or consider not doing this POST at all. retry doesn&#39;t help here May I see: <br/> <br/> * code that does retry <br/> <br/> * logs from that attempt where retry didn&#39;t help <br/> See above now we don&#39;t use retries in our tests for such cases. <br/> results from command POST above were gotten when suites failed with the same error and I tried to check it through curl. Ie was a long time since the server was idle, before I did this. <br/> can&#39;t provide more info now because new tests were triggered. Ok, so there&#39;s possibly two bugs here. But I&#39;ll need logs for that case where it failed even long time after rebalance. Nominating for 2.0.1 RN Nominating ? reproduced again on build 147:<br/> <a href="http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/529/consoleFull">http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/529/consoleFull</a><br/> <br/> all logs:<br/> <a href="https://s3.amazonaws.com/bugdb/jira/MB-7282/529_run_logs.zip">https://s3.amazonaws.com/bugdb/jira/MB-7282/529_run_logs.zip</a><br/> or<br/> <a href="http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/529/artifact/*zip*/archive.zip">http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/529/artifact/*zip*/archive.zip</a> build 159 <a href="http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/546/consoleFull">http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/546/consoleFull</a> build 164 too Indeed a problem. <br/> <br/> Appears to be something with global name service because autofailover service _is_ running we&#39;re just somehow unable to locate it. <br/> <br/> Which would be a very bad news because that component is not directly our code but part of erlang&#39;s stdlib. <br/> <br/> Raised to blocker as global component problem implies possibility of much worse issues. rebalance is likely unrelated here. What is potentially related is other nodes connecting/disconnecting. <a href="http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/596/consoleFull">http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/596/consoleFull</a><br/> 2.0.2-773-rel See also <a href="http://www.couchbase.com/issues/browse/MB-7967">http://www.couchbase.com/issues/browse/MB-7967</a> Workaround:<br/> <br/> wget --user=Administrator --password=asdasd --post-data=&#39;rpc:call(mb_master:master_node(), erlang, apply ,[fun () -&gt; erlang:exit(erlang:whereis(mb_master), kill) end, []]).&#39; <a href="http://localhost:8091/diag/eval">http://localhost:8091/diag/eval</a> Date of First Response Wed, 28 Nov 2012 09:06:44 -0600 Flagged Operating System Rank 3028 [MB-6232] ep-engine needs 1.5 minutes to create 1k vbuckets. Seems too slow (but gets fast with barrier=0) http://www.couchbase.com/issues/browse/MB-6232 Couchbase Server SUBJ. <br/> <br/> Just try single node dev cluster and try to load beers sample (it&#39;ll fail for perhaps different reason). <br/> <br/> You can observe in logs: <br/> <br/> We&#39;re doing set_vbucket_state requests from janitor (ns_memcached messages) and this is the first messages from mccouch about just created vbucket (capi_set_view_manager messages and mc_connection messages). Note: I&#39;ve added extra log messages at the beginning of sync_notify and end to see if it&#39;s couchdb&#39;s or ns_servers&#39; fault. From timestamps it&#39;s clear that it&#39;s not. <br/> <br/> [ns_server:debug,2012-08-15T9:47:40.309,<a href='mailto:n_0@127.0.0.1'>n_0@127.0.0.1</a>:&lt;0.868.0&gt;:mc_connection:do_notify_vbucket_update:106]sending out sync_notify <br/> [ns_server:info,2012-08-15T9:47:40.309,<a href='mailto:n_0@127.0.0.1'>n_0@127.0.0.1</a>:&lt;0.870.0&gt;:ns_memcached:do_handle_call:485]Changed vbucket 740 state to active <br/> [views:debug,2012-08-15T9:47:40.309,<a href='mailto:n_0@127.0.0.1'>n_0@127.0.0.1</a>:mc_couch_events:capi_set_view_manager:handle_mc_couch_event:418]Got set_vbucket event for beer-sample/1023. Updated state: active <br/> [ns_server:info,2012-08-15T9:47:40.309,<a href='mailto:n_0@127.0.0.1'>n_0@127.0.0.1</a>:&lt;0.870.0&gt;:ns_memcached:do_handle_call:485]Changed vbucket 739 state to active <br/> [ns_server:debug,2012-08-15T9:47:40.309,<a href='mailto:n_0@127.0.0.1'>n_0@127.0.0.1</a>:&lt;0.868.0&gt;:mc_connection:do_notify_vbucket_update:113]done <br/> [ns_server:info,2012-08-15T9:47:40.310,<a href='mailto:n_0@127.0.0.1'>n_0@127.0.0.1</a>:&lt;0.870.0&gt;:ns_memcached:do_handle_call:485]Changed vbucket 738 state to active <br/> [ns_server:info,2012-08-15T9:47:40.310,<a href='mailto:n_0@127.0.0.1'>n_0@127.0.0.1</a>:&lt;0.870.0&gt;:ns_memcached:do_handle_call:485]Changed vbucket 737 state to active <br/> <br/> And last message is at: <br/> <br/> [ns_server:debug,2012-08-15T9:49:14.410,<a href='mailto:n_0@127.0.0.1'>n_0@127.0.0.1</a>:&lt;0.868.0&gt;:mc_connection:do_notify_vbucket_update:106]sending out sync_notify <br/> [views:debug,2012-08-15T9:49:14.410,<a href='mailto:n_0@127.0.0.1'>n_0@127.0.0.1</a>:mc_couch_events:capi_set_view_manager:handle_mc_couch_event:418]Got set_vbucket event for beer-sample/1. Updated state: active <br/> [ns_server:debug,2012-08-15T9:49:14.410,<a href='mailto:n_0@127.0.0.1'>n_0@127.0.0.1</a>:&lt;0.868.0&gt;:mc_connection:do_notify_vbucket_update:113]done <br/> [ns_server:debug,2012-08-15T9:49:14.501,<a href='mailto:n_0@127.0.0.1'>n_0@127.0.0.1</a>:&lt;0.868.0&gt;:mc_connection:do_notify_vbucket_update:106]sending out sync_notify <br/> [views:debug,2012-08-15T9:49:14.502,<a href='mailto:n_0@127.0.0.1'>n_0@127.0.0.1</a>:mc_couch_events:capi_set_view_manager:handle_mc_couch_event:418]Got set_vbucket event for beer-sample/0. Updated state: active <br/> [ns_server:debug,2012-08-15T9:49:14.502,<a href='mailto:n_0@127.0.0.1'>n_0@127.0.0.1</a>:&lt;0.868.0&gt;:mc_connection:do_notify_vbucket_update:113]done <br/> <br/> <br/> I&#39;ve just retried with remounting with barrier=0 and it was blazing fast. What this mount option changes is short fsync are nearly instant because they merely send data to disk&#39;s buffer without waiting until data actually hits platter. <br/> <br/> So that&#39;s evidence of some excessive use of fsyncs somewhere during vbucket db files creation and that&#39;s clearly on ep-engine/couchstore side. <br/> <br/> Set this to blocker as it&#39;ll affect people trying to create buckets on real hardware, hard disks and any modern linux distro (where barriers are on by default). <br/> MB-6232 ep-engine needs 1.5 minutes to create 1k vbuckets. Seems too slow (but gets fast with barrier=0) Bug Blocker Open Unresolved Public Mike Wiederhold Aleksey Kondratenko Wed, 15 Aug 2012 12:07:16 -0500 Tue, 7 May 2013 00:54:57 -0500 2.0 2.0.1 2.1 couchbase-bucket storage-engine 0 10 Note that on <a href="http://www.couchbase.com/issues/browse/MB-6305" title="default bucket ends up in limbo between delete and create"><strike>MB-6305</strike></a> this caused quite a bit of trouble. Currently, one cannot reliably go through a cycle of delete/create bucket in any reliable fashion using public interfaces. Sometime between build 1495 and build 2.0.0c 709, the delete/recreate cycle now takes multiple minutes. <br/> <br/> In automated testing some SDKs used to use flush_all(). At the recommendation of engineering, we moved to delete/create. Now using delete/create is intractable. This has broken our testing again, unfortunately. <br/> <br/> The larger concern is that our users will probably see this pretty quickly too. What we were doing here isn&#39;t all that unusual. Matt, I think you can try workaround of: <br/> <br/> mount / -o remount,nobarrier <br/> <br/> And if it&#39;s not helping in your case then there&#39;s something else that makes it slower then needed. <br/> Matt, I think you can try workaround of: <br/> <br/> mount / -o remount,nobarrier <br/> <br/> And if it&#39;s not helping in your case then there&#39;s something else that makes it slower then needed. <br/> BTW I recall now having similar problem pre DP4. I&#39;ve fixed it by creating all vbuckets in parallel. But that was ofcourse when mccouch was at it&#39;s full power. <br/> @alk, I presume that should still be in here, so the issue is further down it would seem. <br/> <br/> thanks for the additional info Just tried to reproduce this with 2.0 build 1689 and loading bear samples times out:<br/> <br/> [ns_server:error,2012-09-06T10:49:42.436,<a href='mailto:ns_1@127.0.0.1'>ns_1@127.0.0.1</a>:&lt;0.752.0&gt;:menelaus_web:handle_post_sample_buckets:506]Loading sample buckets timed out<br/> <br/> Same error is shown in the UI. If I click to retry it says data is already loaded, however if I inspect the db there are no documents in the bucket. Not sure if this is similar issue...attaching diag<br/> more info here : <a href="http://www.couchbase.com/issues/browse/MB-6305">http://www.couchbase.com/issues/browse/MB-6305</a> Matt, when I referenced mccouch that was old mccouch that was our persistence path. This is not the case anymore. So same issue is still hitting us. We&#39;re seemingly doing (potentially few) fsyncs per vbucket and we&#39;re doing all that sequentially. <br/> <br/> Tommie, docloader thing is most likely distinct issue. I think we already have bug for that. Alk, <br/> <br/> I investigated this issue further and found that the latency to send a vbucket state transition to mccouch was 100ms. Interestingly to do a state transition to dead the latency was an order of magnitude quicker so I think there might be something going on on the mccouch side here. Mike, by looking at timestamps in logs I&#39;m pretty sure there are no delays on mccouch side. Mike, can you continue driving to analyze this? The fix for this bug will require changes on both the ep-engine and mccouch side and requires modifying how we store vbucket state data and how we notify mccouch about vbucket state changes. The changes will not be small and will be risky for the 2.0 release. I recommend this change be move to 2.0.1. For more details. we currently store a vbucket state file in its vbucket database as a local doc, which means that it will require one fsync for every state change. I was trying to replace those individual local docs with a global vbucket state doc and store it in the &quot;master&quot; meta database, but realized that it requires changes in interactions between ep-engine and erlang, and also affects our 2.0 backup and restore tools at this time. <br/> <br/> This is the design defect and limitation and risky changes for 2.0 at this time. Can we consider not using fsync for initial vbucket creation ? Also see <a href="http://www.couchbase.com/issues/browse/MB-6234" title="We&#39;re leaving some disk performance on the table">MB-6234</a> which is a duplicate of this issue. <a href="http://www.couchbase.com/issues/browse/MB-6234" title="We&#39;re leaving some disk performance on the table">MB-6234</a> is not necessarily a dup. This bug is about metadata ops.<br/> <br/> Particularly, why ep-engine is even needs tons of fsync to create brand new vbuckets ?<br/> <br/> Why ep-engine cannot just drop those ops on the floor when during that 1.5 minute there&#39;s flush ?<br/> <br/> I think at least those actions are possible and perhaps more and sdk folks will finally have a working flush.<br/> We need to better understand this issue in ep_engine and couchstore and come up with options to fix it in the 2.0.2 / 2.1 timeframe Yay! There are some other issues still in 2.0.2 that will likely help improve this issue, but I am moving this task out to 2.1 Created CBQE for QE to add in functional automation suite: <a href="http://www.couchbase.com/issues/browse/CBQE-1192">http://www.couchbase.com/issues/browse/CBQE-1192</a><br/> &nbsp;&nbsp; Duplicate MB-7160 Date of First Response Mon, 20 Aug 2012 12:17:31 -0500 Rank 2830 [MB-8258] Complete Metadata ejection when item becomes non-resident http://www.couchbase.com/issues/browse/MB-8258 Couchbase Server Memory overhead from an item&#39;s metadata would be huge in case of a very large number of items. Moving the metadata to the value blob allows us to evict the metadata with the value together and fetch it from disk on demand. MB-8258 Complete Metadata ejection when item becomes non-resident Improvement Blocker Open Unresolved Public Dipti Borkar Mike Wiederhold Tue, 4 Sep 2012 17:58:00 -0500 Mon, 13 May 2013 19:19:57 -0500 2.0 2.0.1 2.0.2 2.1 couchbase-bucket 0 2 Date of First Response Mon, 13 May 2013 19:19:57 -0500 Rank 2092 [MB-8145] 1 read-only user for UI and REST API http://www.couchbase.com/issues/browse/MB-8145 Couchbase Server <a href="http://www.pivotaltracker.com/story/show/25735995">http://www.pivotaltracker.com/story/show/25735995</a> MB-8145 1 read-only user for UI and REST API Improvement Blocker Open Unresolved Public Aleksey Kondratenko Dipti Borkar Mon, 21 May 2012 19:57:41 -0500 Mon, 13 May 2013 19:26:41 -0500 2.0 2.0.1 2.0.2 2.1 ns_server 0 4 basically, with this feature you will have 2 couchbase users. <br/> <br/> 1 - admin (can do anything and everything) <br/> 2 - read-only user (cannot change any setting , can only see and access UI and REST API settings - no POST / PUT only GET) BTW CBD is already restricted to Membase inc.<br/> <br/> We&#39;ve had ticket for more general user management elsewhere. Maybe. But IMHO don&#39;t expect that more general stuff be done very soon. I.e. it doesn&#39;t make much sense until we can comfortably handle many tens of buckets in cluster.<br/> Thanks Alk. I think the argument still stands even for less than 10 buckets. This is really for their development environment where they may have 10 different &quot;teams&quot; working off the same cluster and for compliance reasons need to ensure that if one team gets compromised, they are not able to see the data for the rest of the company. Date of First Response Thu, 21 Jun 2012 16:08:44 -0500 Rank 514 [MB-7960] Replace XDCR CAPI protocol on the destination with memcached protocol http://www.couchbase.com/issues/browse/MB-7960 Couchbase Server MB-7960 Replace XDCR CAPI protocol on the destination with memcached protocol Improvement Blocker Open Unresolved Public Junyi Xie Dipti Borkar Tue, 5 Feb 2013 03:15:11 -0600 Thu, 16 May 2013 16:22:22 -0500 2.0 2.0.1 2.1 cross-datacenter-replication 0 4 Prototype in Erlang is under test. Great ! <br/> subtask <br/> <br/> 1) implement batch getMeta and setMeta in ep_engine: <br/> <a href="http://www.couchbase.com/issues/browse/MB-8213" title="Implement batch getMeta and setMeta/delMeta operations in ep_engine ">MB-8213</a> <br/> <br/> owner: Mike <br/> <br/> 2) expand remote_cluster_info: <br/> <a href="http://www.couchbase.com/issues/browse/MB-8299" title="remote_cluster_info module should return remote memcached access info">MB-8299</a> <br/> <br/> owner: Aliaskey A. <br/> <br/> 3) ns_server new memcahced API for batch getMeta and setMeta: <br/> <a href="http://www.couchbase.com/issues/browse/MB-8300" title="Implement ns_server side memcached API for new get_meta_batch and update_meta_batch">MB-8300</a> <br/> <br/> owner: Junyi <br/> <br/> 0) core XDCR code change <br/> <a href="http://www.couchbase.com/issues/browse/MB-8301" title="XDCR core infrastructure change to enable replicate to remote memcached">MB-8301</a> <br/> <br/> owner: Junyi <br/> MB-8213 MB-8299 MB-8300 MB-8301 Date of First Response Thu, 11 Apr 2013 17:27:30 -0500 Rank 79 Sprint Status [MB-8012] Need a purging mechanism for Couchstore to remove deleted / expired items http://www.couchbase.com/issues/browse/MB-8012 Couchbase Server let&#39;s discuss more in person. customers are hitting this already <br/> <br/> discussion: <a href="http://hub.internal.couchbase.com/confluence/display/cbeng/Deletion+Purging">http://hub.internal.couchbase.com/confluence/display/cbeng/Deletion+Purging</a> MB-8012 Need a purging mechanism for Couchstore to remove deleted / expired items Improvement Blocker Open Unresolved Public Aaron Miller Dipti Borkar Tue, 4 Dec 2012 14:06:28 -0600 Mon, 13 May 2013 20:58:35 -0500 2.0 2.0.1 2.1 storage-engine 0 4 Note: this has very serious implications for indexes, that can lead to incorrectness. <br/> Possibly for xcdr as well (at least for couchdb replication it has). +1. This is a harder problem than it sounds like, especially for XDCR, where both correctness and the eventual consistency property could break if it is not handled correctly. <br/> <br/> I&#39;ve done some minor brainstorming around this w/ damien, but we hadn&#39;t really come up with a solution, as whatever we do will add a good amount of complexity to XDCR and possibly views as well. To add more visibility, here&#39;s what I commented in <a href="http://hub.internal.couchbase.com/confluence/display/cbeng/Deletion+Purging?focusedCommentId=6784114&#comment-6784114">http://hub.internal.couchbase.com/confluence/display/cbeng/Deletion+Purging?focusedCommentId=6784114&amp;#comment-6784114</a> : <br/> <br/> &quot; <br/> This is equally concerning for indexes. <br/> Basically the purging problem was never fully &quot;solved&quot; in CouchDB. <br/> <br/> In CouchDB, when a database purge is done, we store an object in the file that lists the IDs (and revisions) that were purged. The header than has a field that points to this object. <br/> This is however kept only for the last purge operation. <br/> <br/> When index updates are triggered, they check the last purge object, and then know they have to remove all key/values previously produced for those documents listed in the purge object. <br/> <br/> However, if the index missed the last 2 purges, it has no way to remove some no longer valid key/value pairs. <br/> Database headers in CouchDB have a purge_seq field, incremented every time a purge is performed. Indexes (CouchDB) also store the last purge_seq they saw in a database. If they see that the difference between the current database purge_seq and the last purge_seq seen by the index is greater than 1, then the index is reset - recreated from scratch, because has said before, it has no way to know which documents were purged in previous purge operations. <br/> <br/> To me this seems like a change with a very big impact for a minor release such as 2.0.2. Not only there are these functional issues, it would also imply changing the index file format to keep track of purge_seqs for each active/passive vbucket. <br/> &quot; Damien has created the design doc here : <a href="https://github.com/couchbaselabs/cbpurge/blob/master/purge_2.0.2_design.md">https://github.com/couchbaselabs/cbpurge/blob/master/purge_2.0.2_design.md</a> Since this is promoted to blocker I should probably detail exactly what that is. <br/> <br/> That document does not implement a &quot;correct&quot; solution, but a probabilistic one. If the system is slower than expected we can incorrectly remove unprocessed deletions.<br/> <br/> For views that&#39;s not a big problem as it&#39;s not difficult to make views able to detect that this has happened (if it does happen the index must be destroyed and regenerated to ensure correctness, but we *can* ensure correctness).<br/> For XDCR we cannot ensure correctness (that is, we cannot guarantee eventual consistency), as we cannot detect that we have lost track of deletions. We can fix the problem if we notice it in the field (as detailed in the design doc), but since we can&#39;t automatically detect it, it follows that our users won&#39;t be able to either, and if the issue does occur it will likely manifest as subtle incorrect behavior, and largely go unnoticed. We should have a threshold based approach. If items have been deleted more than 30 days for example, purge them. <br/> Do we persist delete timestamps now? <br/> This is the threshold based approach. We will persist delete timestamps as part of implementing this. Dependency MB-8261 Date of First Response Tue, 4 Dec 2012 15:50:09 -0600 Rank 563 [MB-7250] Mac OS X App should be signed by a valid developer key http://www.couchbase.com/issues/browse/MB-7250 Couchbase Server Currently launching the Mac OS X version tells you it&#39;s from an unidentified developer. You have to right click to launch the app. We can fix this. MB-7250 Mac OS X App should be signed by a valid developer key Improvement Blocker In Progress Unresolved Public Phil Labee J Chris Anderson Thu, 22 Nov 2012 08:57:59 -0600 Thu, 16 May 2013 18:12:06 -0500 2.0-beta-2 2.0.2 2.0.2 build 0 6 Chris, <br/> <br/> do you know what needs to change on the build machine to embed our developer key ? I have no idea. I could start researching how to get a key from Apple but maybe after the weekend. :) we can discuss this next week : ) . Thanks for reporting the issue Chris. <br/> we&#39;ll want separate, related bugs (tasks) for other platforms, too (windows, linux) We need to get a developer ID from Apple; this will give us some kind of cert, and a local private key for signing. <br/> Then we need to figure out how to get that key and cert onto the build machine, in the Keychain of the account that runs the buildbot. the instructions to build is available here : <br/> <a href="https://github.com/couchbase/couchdbx-app">https://github.com/couchbase/couchdbx-app</a> <br/> we need to add codesign as a build step there Phil, <br/> <br/> do you have any update on this ticket. ? <br/> I have signing cert installed on 10.17.21.150 (MacBuild). <br/> <br/> Change to Makefile: <a href="http://review.couchbase.org/#/c/24149/">http://review.couchbase.org/#/c/24149/</a> <br/> need to change master.cfg and pass env.var. to package-mac disregard previous. Have added signing to Xcode projects. <br/> <br/> see <a href="http://review.couchbase.org/#/c/24273/">http://review.couchbase.org/#/c/24273/</a> <br/> To test this go to System Preferences / Security &amp; Privacy, and on the General tab set &quot;Allow applications downloaded from&quot; to &quot;Mac App Store and Identified Developers&quot;. Set this before running Couchbase Server.app the first time. Once an app has been allowed to run this setting is no longer checked for that app, and there doesn&#39;t seem to be a way to reset that. <br/> <br/> What is odd is that on my system, I allowed one unsigned build to run before restricting the app run setting, and then no other unsigned builds would be checked (and would all be allowed to run). Either there is a flaw in my testing methodology, or a serious weakness in this security setting: Just because one app called Couchbase Server was allowed to run should confer this privilege to other apps with the same name. A common malware tactic is to modify a trusted app and distribute it as update, and if the security setting keys off the app name it will do nothing to prevent that. <br/> <br/> I&#39;m approving this change without having satisfactorily tested it. Strictly speaking it&#39;s not the app name but its bundle ID, i.e. &quot;com.couchbase.CouchbaseServer&quot; or whatever we use. <br/> <br/> &gt; I allowed one unsigned build to run before restricting the app run setting, and then no other unsigned builds would be checked <br/> <br/> By OK&#39;ing an unsigned app you&#39;re basically agreeing to toss security out the window, at least for that app. This feature is really just a workaround for older apps. By OK&#39;ing the app you&#39;re not really saying &quot;yes, I trust this build of this app&quot; so much as &quot;yes, I agree to run this app even though I don&#39;t trust it&quot;. <br/> <br/> &gt; A common malware tactic is to modify a trusted app and distribute it as update <br/> <br/> If it&#39;s a trusted app it&#39;s hopefully been signed, so the user wouldn&#39;t have had to waive signature checking for it. Further thought: It might be a good idea to change the bundle ID in the new signed version of the app, because users of 2.0 with strict security settings have presumably already bypassed security on the unsigned version. Per bug scrubs, keep this a blocker since customers ran into this issues (and originally reported it). revert the change so that builds can complete. App is currently not being signed. i suggest for 2.0.1 release we do this build manually. As one-off fix, add the signature manually and automate the required steps later in 2.0.2 or beyond. Please move this bug to 2.0.2 after populating the required signature manually. I am lowing the severity to critical for it isn&#39;t no longer a blocking issue. Phil to upload the binary to latestbuilds , ( 2.0.1-101-rel.zip ) Please verify:<br/> <br/> <a href="http://packages.northscale.com/latestbuilds/couchbase-server-community_x86_64_2.0.1-160-rel-signed.zip">http://packages.northscale.com/latestbuilds/couchbase-server-community_x86_64_2.0.1-160-rel-signed.zip</a><br/> uploaded: <br/> <br/> <a href="http://packages.northscale.com/latestbuilds/couchbase-server-community_x86_64_2.0.1-160-rel-signed.zip">http://packages.northscale.com/latestbuilds/couchbase-server-community_x86_64_2.0.1-160-rel-signed.zip</a> <br/> <br/> I can rename it when uploading for release. i still do get the error that it is from an identified developer.<br/> <br/> operator error.<br/> <br/> I rebuilt the app, this time verifying that the codesign step occurred.<br/> <br/> Uploaded now file to same location:<br/> <br/> <a href="http://packages.northscale.com/latestbuilds/couchbase-server-community_x86_64_2.0.1-160-rel-signed.zip">http://packages.northscale.com/latestbuilds/couchbase-server-community_x86_64_2.0.1-160-rel-signed.zip</a><br/> still need to perform manual workaround release candidate has been uploaded to:<br/> <br/> <a href="http://packages.northscale.com/latestbuilds/couchbase-server-community_x86_64_2.0.1-172-signed.zip">http://packages.northscale.com/latestbuilds/couchbase-server-community_x86_64_2.0.1-172-signed.zip</a><br/> Phil, looks like version 172/185 is still getting the error. My Mac version is 10.8.2 Install couchbase server (build 2.0.1-172 community version) in my mac osx 10.7.4 , I only see the warning message Latest version (04.03.13) : <a href="http://builds.hq.northscale.net/latestbuilds/couchbase-server-community_x86_64_2.0.1-185-rel.zip">http://builds.hq.northscale.net/latestbuilds/couchbase-server-community_x86_64_2.0.1-185-rel.zip</a> works in 10.7 but not in 10.8.<br/> if we can get the fix for 10.8 by tomorrow, end of day, QE is willing to test for release on tuesday, april 9. The mac builds are not being automatically signed, so build 185 is not signed. The original 172 is also not signed. <br/> <br/> Did you try<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://packages.northscale.com/latestbuilds/couchbase-server-community_x86_64_2.0.1-172-signed.zip">http://packages.northscale.com/latestbuilds/couchbase-server-community_x86_64_2.0.1-172-signed.zip</a><br/> <br/> to see if that was signed correctly?<br/> <br/> Phil,<br/> Yes, we did try the 172-signed version. It works on 10.7 but not 10.8. Can you take a look? I rebuilt 2.0.1-185 and uploaded a signed app to:<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://packages.northscale.com/latestbuilds/couchbase-server-community_x86_64_2.0.1-185-rel.SIGNED.zip">http://packages.northscale.com/latestbuilds/couchbase-server-community_x86_64_2.0.1-185-rel.SIGNED.zip</a><br/> <br/> Test on a machine that has never had Couchbase Server installed, and has the security setting to only allow Appstore or signed apps.<br/> <br/> If you get the &quot;Couchbase Server.app was downloaded from the internet&quot; warning and you can click OK and install it, then this bug is fixed. The quarantining of files downloaded by a browser is part of the operating system and is not controlled by signing. Tried the 185-signed version (see attached screen shot). Same error message. This is not an error message related to this bug.<br/> <br/> per bug triage, we need to have mac 10.8 osx working since it is a supported platform (published in the website). Date of First Response Thu, 22 Nov 2012 09:10:32 -0600 Rank 98 Sprint 15 Sprint Priority 5.0 Sprint Status [MB-6726] Rebalance is slow when indexing/compaction and query load are going on in parallel http://www.couchbase.com/issues/browse/MB-6726 Couchbase Server Cluster information: <br/> - 8 centos 6.2 64bit server with 4 cores CPU <br/> - Each server has 32 GB RAM and 400 GB SSD disk. <br/> - SSD disk format ext4 on /data <br/> - Each server has its own drive, no disk sharing with other server. <br/> - Cluster has 2 buckets, default (11GB) and saslbucket (11GB) with consistent view enable. For 2 buckets, we use only 68% total RAM of system. <br/> - Load 12 million items to saslbucket and 45 million items to default bucket. Each key has size from 512 bytes to 1500 bytes <br/> - Each bucket has one doc and 2 views for each doc (default d1 and saslbucket d11) <br/> <br/> * Create cluster with 4 nodes installed couchbase server 2.0.0-1746 <br/> <br/> 10.6.2.37 <br/> 10.6.2.38 <br/> 10.6.2.39 <br/> 10.6.2.40 <br/> <br/> * Data path /data <br/> * View path /data <br/> <br/> * Let load running at 6K ops, query 400 to 500 per second <br/> * Add 4 nodes to cluster and rebalance <br/> 10.6.2.42 <br/> 10.6.2.43 <br/> 10.6.2.44 <br/> 10.6.2.45 <br/> <br/> * rebalance hang. Filed bug <a href="http://www.couchbase.com/issues/browse/MB-6706" title="[system test] rebalance hang when add nodes to cluster"><strike>MB-6706</strike></a> <br/> * Try rebalance again. Cluster rebalances saslbucket first and took 9 hours to complete rebalance saslbucket <br/> * default bucket took 32 hours to complete <br/> <br/> Link to collect_info of all nodes <a href="https://s3.amazonaws.com/packages.couchbase/collect_info/orange/2_0_0/201209/8nodes-col-info-1746-reb-slow-constn-enable-20120925-141314.tgz">https://s3.amazonaws.com/packages.couchbase/collect_info/orange/2_0_0/201209/8nodes-col-info-1746-reb-slow-constn-enable-20120925-141314.tgz</a> <br/> <br/> Each atop file size from 1.5 GB to 2.5 GB. If atop file needed, I will upload them as request <br/> centos 6.2 64bit build 2.0.0-1746 MB-6726 Rebalance is slow when indexing/compaction and query load are going on in parallel Bug Blocker In Progress Unresolved Public Aleksey Kondratenko Thuan Nguyen Tue, 25 Sep 2012 16:31:27 -0500 Thu, 25 Apr 2013 17:40:01 -0500 2.0-beta-2 2.1 performance 0 14 <a href="http://www.couchbase.com/issues/browse/MB-6714">http://www.couchbase.com/issues/browse/MB-6714</a><br/> <br/> The fix was just merged today. Please test it with the next build. Seeing slower rebalance as on <a href="http://www.couchbase.com/issues/browse/MB-6769" title="Very slow rebalance progress while rebalancing-In on Source cluster during Unidirectional XDCR."><strike>MB-6769</strike></a> with latest ep-engine changes. Ketaki, XDCR is a very different beast I&#39;ve seen it to cause very severe CPU load and weird memory usage spikes which could affect anything.<br/> <br/> I need:<br/> <br/> a) diags<br/> <br/> b) I think brand new bug is a good idea here<br/> Ok, I had filed bug 6769 for the same issue. But after some discussion w/ other folks, closed that as a duplicate of this one.<br/> <br/> We can have a xdcr-specific bug for understanding this behaviour.<br/> Closing this one and re-opening 6769 , with more logs. This bug was opened by Tony for slowness on rebalance and was fixed with <a href="http://www.couchbase.com/issues/browse/MB-6714">http://www.couchbase.com/issues/browse/MB-6714</a>. <br/> <br/> Closed this bug since we have another rebalance-xdcr slow bug <a href="http://www.couchbase.com/issues/browse/MB-6769" title="Very slow rebalance progress while rebalancing-In on Source cluster during Unidirectional XDCR."><strike>MB-6769</strike></a>. <br/> <br/> Please re-open this, if this case is valid on the above system-test scenarios. blocker for 2.0 beta-2 as consistent views needs to be functional and performs reasonably fast for 2.0 beta 2 _Original_ bug as indeed fixed by ep-engine folks.<br/> <br/> I&#39;ve heard we&#39;re still having some slowness with indexes. Please open new bug with new and relevant diagnostics. We are still investigating slowness with indexes+rebalance+consistent views isn&#39;t this a duplicate of <a href="http://www.couchbase.com/issues/browse/MB-6796" title="perf - 10x slower rebalance performance with consistent view turned on"><strike>MB-6796</strike></a> once thats fixed we can retest pavel&#39;s test yesterday shows views + rebalance performance of... <br/> <br/> &gt; Rebalance from 3 to 4 nodes with views (consistent view enabled). <br/> &gt; just 4.7h (2.0.0-1858) vs. 5.3 h (2.0.0-1792). <br/> <br/> <a href="https://www.yammer.com/couchbase.com/#/Threads/show?threadId=224286191">https://www.yammer.com/couchbase.com/#/Threads/show?threadId=224286191</a> Re-test orange cluster with build 2.0.0-1862, swap rebalance took 43 hours to complete. Saslbucket took 20 hours and default bucket took 23. <br/> <br/> Cluster has 2 buckets (2 replica), default and saslbucket. Each bucket has 15 million items with key size from 512 to 1024 bytes. <br/> Do swap rebalace, add 2 nodes and remove 2 nodes. During rebalance, maintain the load about 6K ops on each bucket and queries about 200 to 300 per second on each bucket. Link to manifest file of build 2.0.0-1862 <a href="http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1862-rel.rpm.manifest.xml">http://builds.hq.northscale.net/latestbuilds/couchbase-server-enterprise_x86_64_2.0.0-1862-rel.rpm.manifest.xml</a> <br/> Farshid,<br/> <br/> I don&#39;t think there is anything that I can do in ep-engine to resolve this issue.<br/> <br/> We know that this slowness is caused by enabling indexing / querying during rebalance with consistence view.<br/> <br/> Please assign it to Filipe or Alk for further investigations.<br/> assigning this to Siri/Mike for furthere investigation and traige assigning to mike as sriram is on plane at the moment. need to get alk + damien on this. latest update: pavel, alk &amp; steve have agreed on a new larger test with more isolation. awaiting results. notes from bug-scrub discussion... <br/> <br/> - farshid: QA now trying system testing with fewer vbuckets (64), which is unsupported configuration but need to turnaround test results can be turned around faster. <br/> <br/> - alk&#39;s considering batching -- on small-scale his tests show it might help (but unknown what would happen on larger scale). <br/> <br/> - alk: consider recommending users disable consistent views before they rebalance. <br/> <br/> - yaseen: need to found out boundary conditions around this? Results for Alk&#39;s spec: <br/> <br/> <a href="https://docs.google.com/spreadsheet/ccc?key=0AgLUessE73UXdFVINGEyZ0hzOGUtbWVHeUU3T3Rlc1E#gid=0">https://docs.google.com/spreadsheet/ccc?key=0AgLUessE73UXdFVINGEyZ0hzOGUtbWVHeUU3T3Rlc1E#gid=0</a> <br/> <br/> We should use more ddoc/views. Otherwise indexing is too fast. Conclusions: <br/> -- indexing is not a bottleneck <br/> -- compaction does&#39;t affect results significantly (again) <br/> -- we have to try three additional dimensions: more items (20M), more design ddocs (2), less vbuckets (512). Alk says no more experiment ideas at the moment for Pavel. options reviewed in bug-scrub... <br/> <br/> - sequential vs parallel fix for rebalancing (start next phase right after backfill&#39;s done) [alk] <br/> <br/> - vbucket-id as prefix to back-index [filipe] - early laptop results look promising bug-scrub moved to 2.0.1 Editing the component to &quot;view engine&quot; and &quot;ns_server&quot;. If there is anything that needs to be done for this on the ep-engine side please let us know. system test team needs to rerun the view tests ( 8 node SSD ) As for build 121 there is no significant improvement in perf. tests. can you paste the link to the results ? ( comparing this build against 2.0 GA ) 2.0.0-1971: <br/> <br/> <a href="http://qa.hq.northscale.net/job/eperf-graph-loop/1217/artifact/reb-vperf-10M.loop_2.0.0-1971-rel-enterprise_2.0.0-1971-rel-enterprise_run_1_Dec-04-2012_22:54:50.pdf">http://qa.hq.northscale.net/job/eperf-graph-loop/1217/artifact/reb-vperf-10M.loop_2.0.0-1971-rel-enterprise_2.0.0-1971-rel-enterprise_run_1_Dec-04-2012_22:54:50.pdf</a> <br/> <br/> 2.0.1-121: <br/> <br/> <a href="http://qa.hq.northscale.net/view/eperf-graphs/job/eperf-graph-loop/1334/artifact/reb-vperf-10M.loop_2.0.1-121-rel-enterprise_2.0.1-121-rel-enterprise_run_1_Jan-09-2013_06%3A40%3A40.pdf">http://qa.hq.northscale.net/view/eperf-graphs/job/eperf-graph-loop/1334/artifact/reb-vperf-10M.loop_2.0.1-121-rel-enterprise_2.0.1-121-rel-enterprise_run_1_Jan-09-2013_06%3A40%3A40.pdf</a> <br/> <br/> It&#39;s not quite fair comparison but re-balance time is still ~ 4 hours. rebalance is 10% faster but could be environment noise <br/> looks better : <br/> cpu util 5% better <br/> the memory usage numbers dont look quite right ( ns-server avg reports 21235692 MB == 21 GB ? ) <br/> <br/> looks worse: <br/> &nbsp;runtime is 8 hours longer , <br/> query ops/sec dropped from 537 to 339 but query latency is far better ( 48msec in 2..0.1 compared to 58 in 2.0 ) <br/> drain rate is slower by 15% percent <br/> <br/> we need to find out why query ops/sec is very slow. <br/> <br/> reb−vperf−10M.conf <br/> # Perf−rebalance test with views <br/> # 1 design ddoc, 8 views <br/> # 8K ops/sec <br/> # 80% reads, 20% writes (12% updates/deletes, 8% inserts) <br/> # 10M dataset <br/> # Rebalance from 3 to 4 nodes Alk/Filipe,<br/> <br/> there are initial results available for the litmus test. can you also review the results and comment on what we need to change in the test case.<br/> <br/> from my discussion with Alk/Filipe i heard adding more ddocs would help. is there anything else we want to change in the tes t? I do have measured results, and documented steps to reproduce, for rebalance out cases. <br/> What I see, is that specially for cases of 1 ddoc with 1 view case, the view engine (indexing and compaction) is poorly utilized: <br/> <br/> <a href="https://docs.google.com/document/d/1eUV53B5pXj-5X5FPcbOA3jz1U6jOjh1hnZmZa9bUNs0/edit">https://docs.google.com/document/d/1eUV53B5pXj-5X5FPcbOA3jz1U6jOjh1hnZmZa9bUNs0/edit</a> <br/> <br/> So you a bit more than theory there. As I mentioned before it is *not* fair comparison. <br/> <br/> After 2.0 release I revised config and rebalance tests now use 3 ddocs with 3 views per doc. <br/> Moreover after 2.0 release we discovered issue with one of ssd drives, so for many reasons we need a re-run. <br/> <br/> But in any case, 4 hours don&#39;t look like something ultra fast. <br/> <br/> We do more experiments, for instance rebalance out 4 -&gt; 3 with no ops/queries took 7h (40M items) and 14h (100M items). Is it something that sounds good? New results, for build 2.0.1-126: <br/> <br/> <a href="https://docs.google.com/document/d/1aHHhY-ami84-aEQyx42h95rlp7Vi6WTwlozaIstFgfA/edit">https://docs.google.com/document/d/1aHHhY-ami84-aEQyx42h95rlp7Vi6WTwlozaIstFgfA/edit</a> <br/> <br/> In the meanwhile Pavel already re-triggered the tests he mentioned for this new build. Sorry, it&#39;s not faster at all.<br/> The same regression as Ronnie reports:<br/> <br/> <a href="https://docs.google.com/spreadsheet/ccc?key=0AgLUessE73UXdDV1SXhUZjJ0b0RhU3gtdlUzZGloUFE#gid=0">https://docs.google.com/spreadsheet/ccc?key=0AgLUessE73UXdDV1SXhUZjJ0b0RhU3gtdlUzZGloUFE#gid=0</a> summarizing what i see in the google doc above : <br/> <br/> rebalance out 10M items 2.0.1 ( +A ) is 15 percent slower than 2.0 GA ( +S16) <br/> rebalance in 10M items 2.0.1 ( +A ) is 40% slower than 2.0 GA ( +S16) <br/> <br/> Pavel is planning on rerunning some tests with +A option and post more results. we should thank Filipe, w/o latest improvements it&#39;d be much worse. evperf results for build 2.0.1-139: <br/> <br/> <a href="https://docs.google.com/document/d/16BQu-sD32U4qZxEK8KcvVZMdcqH_klV90DEbnoCu-_E/edit">https://docs.google.com/document/d/16BQu-sD32U4qZxEK8KcvVZMdcqH_klV90DEbnoCu-_E/edit</a> <br/> <br/> specially for 40M, 4 -&gt; 3 rebalance out without load in parallel, indexing + index compaction is still far from being a bottleneck. <br/> <br/> Previously, until a month or so ago, indexing was triggered sequentially in every node for rebalance out, so improving that would theoretically give close to 3x faster rebalance, but that didn&#39;t happen, as the rebalancer now triggers compactions and waits for them to finish before issuing more indexing work, amongst other details possibly. However index compaction doesn&#39;t justify not seeing a significant faster rebalance, lets say around 2x for example. <br/> <br/> Talking with Pavel, no significant speedups were observed for evperf rebalance tests after those improvements were merged. <br/> <br/> There&#39;s still something missing somewhere. based on recent perf runs from Pavel the 2.0.1-139-rel runs are much faster than 2.0.0 but are marked as not consistent. <br/> <br/> we are still waiting for Pavel to rerun these tests. looking at the chart <br/> 2.0.1-139 IN 10M 3 x 3 ASYNC 2063 <br/> 2.0.1-139 OUT 10M 3 x 3 ASYNC 2401 <br/> <br/> but both are marked as not consistent <br/> compared to 2.0.0 1976 GA <br/> <br/> 2.0.0-1976 OUT 10M 3 x 3 ASYNC 18491, 15391 <br/> 2.0.0-1976 IN 10M 3 x 3 ASYNC 12035 &quot;not consistent&quot; means rebalance aware index disabled. <br/> <br/> Alk requested these results. Slightly related, <a href="http://www.couchbase.com/issues/browse/MB-7413" title="Speedup incremental index updates, and make them generate less fragmentation"><strike>MB-7413</strike></a> brought a 3x speedup to incremental index updates, which was though to be the bottleneck. But after it was added, it didn&#39;t seem to have any significant impact on the rebalance time. <br/> <br/> There&#39;s now an evperf test to verify this, outside the rebalance context (added by CBD-767, thanks Pavel!). <br/> <br/> It confirms the 3x speedup: <br/> <br/> <br/> ** build 2.0.1-120 (before <a href="http://www.couchbase.com/issues/browse/MB-7413" title="Speedup incremental index updates, and make them generate less fragmentation"><strike>MB-7413</strike></a>) <br/> <br/> jenkins: <a href="http://qa.hq.northscale.net/view/eperf-parents/job/vesta-views/62/console">http://qa.hq.northscale.net/view/eperf-parents/job/vesta-views/62/console</a> <br/> <br/> incremental update time: 5676.96956992 seconds <br/> <br/> ** build 2.0.1-121 (when <a href="http://www.couchbase.com/issues/browse/MB-7413" title="Speedup incremental index updates, and make them generate less fragmentation"><strike>MB-7413</strike></a> was first introduced) <br/> <br/> jenkins: <a href="http://qa.hq.northscale.net/view/eperf-parents/job/vesta-views/63/console">http://qa.hq.northscale.net/view/eperf-parents/job/vesta-views/63/console</a> <br/> <br/> incremental update time: 1812.51270294 seconds <br/> <br/> ** build 2.0.1-140 (latest 2.0.1) <br/> <br/> jenkins: <a href="http://qa.hq.northscale.net/view/eperf-parents/job/vesta-views/64/console">http://qa.hq.northscale.net/view/eperf-parents/job/vesta-views/64/console</a> <br/> <br/> incremental update time: 1939.70245409 <br/> <br/> build 143<br/> master events logs<br/> in 3-&gt;4, out 4-&gt;3, swap 3-&gt;3. Pavel, please provide latest performance number based on the latest build. I believe we have made a good progress since build 145? Since build 148 actually. <br/> <br/> <a href="http://dashboard.hq.couchbase.com/litmus/dashboard/">http://dashboard.hq.couchbase.com/litmus/dashboard/</a> <br/> (search for reb-vperf) <br/> <br/> + another summary <br/> <br/> <a href="https://docs.google.com/spreadsheet/ccc?key=0AgLUessE73UXdDV1SXhUZjJ0b0RhU3gtdlUzZGloUFE#gid=0">https://docs.google.com/spreadsheet/ccc?key=0AgLUessE73UXdDV1SXhUZjJ0b0RhU3gtdlUzZGloUFE#gid=0</a> <br/> <br/> In some cases we observe up to 20% improvement, but basically we just fixed <a href="http://www.couchbase.com/issues/browse/MB-7676" title="Rebalance hangs when [presumably] data compaction takes place"><strike>MB-7676</strike></a> regression. Please, point me to logs and master events from this runs. For instance, rebalance-in 60M items.<br/> <br/> 4DDOCS 2.0.0-1976:<br/> <a href="http://172.23.96.10:8080/job/apollo-views/26/artifact/">http://172.23.96.10:8080/job/apollo-views/26/artifact/</a><br/> <br/> 4DDOCS 2.0.1-148:<br/> <a href="http://172.23.96.10:8080/job/apollo-views/25/artifact/">http://172.23.96.10:8080/job/apollo-views/25/artifact/</a><br/> <br/> 1DDOC 2.0.0-1976:<br/> <a href="http://172.23.96.10:8080/job/apollo-views/23/artifact/">http://172.23.96.10:8080/job/apollo-views/23/artifact/</a><br/> <br/> 1DDOC 2.0.1-148:<br/> <a href="http://172.23.96.10:8080/job/apollo-views/22/artifact/">http://172.23.96.10:8080/job/apollo-views/22/artifact/</a><br/> May I have all of them and not just &quot;for instance&quot; :) ? Like 30 logs? Sure. If there&#39;s some way for me to find all of them, then feel free to explain me how. Well, I get these logs from 2 jobs:<br/> <a href="http://172.23.96.10:8080/job/apollo-views/">http://172.23.96.10:8080/job/apollo-views/</a><br/> <a href="http://qa.hq.northscale.net/view/eperf-parents/job/vesta-views/">http://qa.hq.northscale.net/view/eperf-parents/job/vesta-views/</a><br/> <br/> You can check parameters and artifacts of every build. Or you can wait and so that later I can grab all of them and upload single archive.<br/> This is something that I&#39;d not recommend to do manually. Unfortunately I cannot make sense of stuff you linked to above.<br/> <br/> I will not necessarily need all logs. I&#39;ll start looking at |2.0.1-153 IN 60M 1 x 3 YES 19806 5.50| and it&#39;s corresponding 2.0.0 events. But I may find other stuff useful. And thus I may need logs of any test run.<br/> I added *logs* column to that spreadsheet. In some cases there are no master events in build artifacts. Thanks a lot. I&#39;ll take a look See master_events.log.svg generated from master events of rebalance (cut-n-pasting from spreadsheet) <br/> <br/> 2.0.1-153 IN 60M 1 x 3 YES 19806 5.5 no replica index <a href="http://172.23.96.10:8080/job/apollo-views/28/artifact/">http://172.23.96.10:8080/job/apollo-views/28/artifact/</a> <br/> <br/> empty intervals (and I&#39;ve confirmed from logs) represent view compactions. Surprisingly, they eat about 1/2 of all time. At least by looking at picture. <br/> <br/> As a workaround I&#39;d like you to try same setup with increased value of vbucket moves before forced view compaction: <a href="http://i.imgur.com/T38iPCS.png">http://i.imgur.com/T38iPCS.png</a> Internal setting for this variable is named rebalanceMovesBeforeCompaction <br/> <br/> Let&#39;s try bumping it up a lot, say to 256 and see if that helps. <br/> <br/> cc-ing Filipe, in case he wants to add anything. Moving it to 2.1 since we see noticeable improvements in systems tests and even performance tests demonstrate some positive changes. Also we don&#39;t expect any particular fix from Filipe and Alk for 2.0.1 and 2.0.2.<br/> <br/> So it&#39;s not fast enough but obviously this bug is not 2.0.1/2.0.2 blocker anymore. Disagree with 2.0.2 a bit. There may still be reasonably low hanging fruit left. I.e. currently it appears we&#39;re spending a bit too much on views compaction. We can at least trade higher disk space usage during rebalance for faster rebalance time at this moment. I&#39;ve tried 2.0.1-153 with rebalanceMovesBeforeCompaction=256. Brief comparison:<br/> -- rebalance time 3h vs 5.5h with default settings and 5.4h with 2.0.0-1976<br/> -- peak disk size is 415GB vs 366GB with default settings and 613GB with 2.0.0-1976<br/> <br/> Build artifacts:<br/> <a href="http://172.23.96.10:8080/job/apollo-views/28/artifact/">http://172.23.96.10:8080/job/apollo-views/28/artifact/</a><br/> <br/> PDF Report:<br/> <a href="http://172.23.96.10:8080/job/apollo-graph-loop/25/artifact/reb-vperf-60M-in-1d-256rmbc.loop_2.0.1-153-rel-enterprise_2.0.1-153-rel-enterprise_APPOLO_Feb-14-2013_19%3A52%3A00.pdf">http://172.23.96.10:8080/job/apollo-graph-loop/25/artifact/reb-vperf-60M-in-1d-256rmbc.loop_2.0.1-153-rel-enterprise_2.0.1-153-rel-enterprise_APPOLO_Feb-14-2013_19%3A52%3A00.pdf</a><br/> 415 vs 366 is including data. Views different is apparently quite a lot bigger.<br/> <br/> May I ask you to do few runs with 32, 64 and 128 ?<br/> <br/> It appears that we should consider bumping that value for great gain in rebalance time, but it remains to be seen which value works best. That&#39;s going to be trade off between disk space usage and rebalance time. No problem, will do. I observe very strange results in recent builds. For instance, &quot;60M+1 ddoc+load&quot; case is 2x slower when compared with 2.0. See spreadsheet for details (rows 16-18):<br/> <a href="https://docs.google.com/spreadsheet/ccc?key=0AgLUessE73UXdDV1SXhUZjJ0b0RhU3gtdlUzZGloUFE#gid=0">https://docs.google.com/spreadsheet/ccc?key=0AgLUessE73UXdDV1SXhUZjJ0b0RhU3gtdlUzZGloUFE#gid=0</a><br/> <br/> Alk, do you have any idea why it happens? + chart for rebalance progress 153 vs 160. Behaviour similar to <a href="http://www.couchbase.com/issues/browse/MB-7785">http://www.couchbase.com/issues/browse/MB-7785</a> Just spotted Pavel&#39;s message.<br/> <br/> May I have manifest of build 160 where we supposedly jumped backwards about 2x perf-wise ? Ok, re-run of 160 was just 1.2x slower. <br/> <br/> But it&#39;s still hard to understand whether it&#39;s environment variation or something software specific. From NS SRV team (Alk):<br/> <br/> wget --post-data=&#39;rebalanceMovesBeforeCompaction=256&#39; --user=Administrator --password=asdasd <a href="http://lh:9000/internalSettings">http://lh:9000/internalSettings</a><br/> <br/> So that&#39;s usual POST to usual /internalSettings API with field name rebalanceMovesBeforeCompaction.<br/> Default value is 16. And that&#39;s how many in- or out- going vbuckets will be moved per node before pausing all moves, triggering views compaction and awaiting its completion.<br/> View compaction is forbidden during vbucket moves. So larger values means less frequent view compactions. Which means less time &quot;spent&quot; on it, but it also means greater disk usage by views.<br/> For this PCI sprint, we need to find the best default and better understand the impact of changing this parameter. Pavel to drive. Integrated in ui-testing #11 (See [<a href="http://qa.hq.northscale.net/job/ui-testing/11/">http://qa.hq.northscale.net/job/ui-testing/11/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.couchbase.com/issues/browse/MB-6726" title="Rebalance is slow when indexing/compaction and query load are going on in parallel">MB-6726</a>: add config with rebalance_moves_before_compaction=256 (Revision 7fc2baf69f359946f978d6d59f0b7f0458805ac1)<br/> <a href="http://www.couchbase.com/issues/browse/MB-6726" title="Rebalance is slow when indexing/compaction and query load are going on in parallel">MB-6726</a>: add config with tuned reb. moves before comp. (Revision 043ce1685781d5b7fa207ffc265698cd62db906d)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> pavelpaulau : <br/> Files : <br/> * conf/perf/reb-vperf-60M-in-1d-256rmbc.conf<br/> <br/> pavelpaulau : <br/> Files : <br/> * conf/perf/reb-vperf-60M-in-1d-128rmbc.conf<br/> * conf/perf/reb-vperf-60M-in-1d-32rmbc.conf<br/> * conf/perf/reb-vperf-60M-in-1d-64rmbc.conf<br/> I&#39;ve created separate task for rebalanceMovesBeforeCompaction. deferred out of 2.0.2 release.<br/> aliaksey and team need further investigation. Dependency MB-8045 Duplicate MB-6769 Date of First Response Tue, 25 Sep 2012 16:43:03 -0500 Rank 318 Sprint 1 9 Sprint Priority 1.0 Sprint Status [MB-7074] xdcr: data compaction doesn't catch up http://www.couchbase.com/issues/browse/MB-7074 Couchbase Server While it works fine in bidirectional case, there is obvious issue with 2 buckets and double unidirectional replication - even after 5 hours of access phase server didn&#39;t manage to compact data. <br/> <br/> I believe it may be related to general issues with capacity planning (as <a href="http://www.couchbase.com/issues/browse/MB-6172" title="XDCR: High CPU utilization on destination nodes">MB-6172</a>) but apparently we should not ignore that. Build 1925. <br/> <br/> Windows, 2&lt;-&gt;2 nodes, 2 buckets, 2 unidir stream MB-7074 xdcr: data compaction doesn't catch up Bug Blocker In Progress Unresolved Public Abhinav Dangeti Pavel Paulau Thu, 1 Nov 2012 12:51:38 -0500 Mon, 13 May 2013 00:53:43 -0500 2.0 2.1 storage-engine 0 13 Looks like an issue in storage layer. Aaron, could you plesae look at why compactor did not catch up? Nothing in the logs seems to indicate compaction having issues. It was still running at the end of when they were captured. Compaction is pretty I/O heavy, so if there&#39;s much other I/O going on it can take a while. Chiyoung has done some experiments around compaction scheduling inside ep-engine that are the sort of thing that would probably help in this type of situation, but I don&#39;t think actually making that change is on our radar in the near-term. Aaron, thanks a lot for explanation. <br/> <br/> Pavel, it looks a generic performance issue with no quick fix. Probably we could put it on hold to post-2.0 after all optimization Aaron mentioned checked in. <br/> Talked more with Pavel and Junyi about this. Please add guidance on where the bottleneck is if the system is not sized appropriately so that users can take the appropriate action. Aaron, could you please address Dipti&#39;s comments here? Say, when users hit this issue that compaction cannot catch up with ongoing operations, what guidelines we need to provide to them? Thanks. <br/> The only resource compaction really competes for is I/O, and XDCR can be I/O heavy. <br/> <br/> Unfortunately I think the only good way around these problems is for the things contending for I/O to be managed in such a way as that they don&#39;t step on each other. That&#39;s why Chiyoung&#39;s change that I mentioned earlier helped in plain K/V situations, as it coordinated the compaction against regular K/V persistence. It ought to help in the XDCR case too, as most of the XDCR I/O is through that path. <br/> <br/> The only guideline I could think of as-is if compaction can&#39;t catch up with ongoing operations is do fewer of those operations. Since this is still an issue, moving this back to 2.0.1 Pavel, <br/> <br/> are there any xperf results from 2.0.1? I have results for build 107 and it&#39;s still the issues: <br/> <br/> <a href="http://qa.hq.northscale.net/job/eperf-graph-loop/1305/artifact/xperf-mixed-uni-2-nodes.loop_2.0.1-107-rel-enterprise_2.0.1-107-rel-enterprise_SOURCE_Dec-19-2012_17%3A54%3A34.pdf">http://qa.hq.northscale.net/job/eperf-graph-loop/1305/artifact/xperf-mixed-uni-2-nodes.loop_2.0.1-107-rel-enterprise_2.0.1-107-rel-enterprise_SOURCE_Dec-19-2012_17%3A54%3A34.pdf</a> <br/> <br/> <a href="http://qa.hq.northscale.net/job/eperf-graph-loop/1307/artifact/xperf-mixed-uni-2-nodes.loop_2.0.1-107-rel-enterprise_2.0.1-107-rel-enterprise_DEST_Dec-19-2012_18%3A11%3A38.pdf">http://qa.hq.northscale.net/job/eperf-graph-loop/1307/artifact/xperf-mixed-uni-2-nodes.loop_2.0.1-107-rel-enterprise_2.0.1-107-rel-enterprise_DEST_Dec-19-2012_18%3A11%3A38.pdf</a> <br/> <br/> But frankly speaking I don&#39;t remember that we tried to fix that. As an aside, the retry phase of database compaction, still done in Erlang, can be made more efficient (and faster of course). Pretty much in the same way that view compaction retry phase was made much faster some time ago (it used to suffer the same issue several months ago). Jin to coordinate a meeting to discuss the scope of the issue and detailed plan of attack to it. per bug scrub . <br/> <br/> Damien says there is a possibility that processes inside erlang vm are crashing per bug scrub: <br/> <br/> Yaseen: rerun the test and ask if this is a capacity issue. if so we can defer this to the next release. As issue description says the same workload works pretty well in case of single bucket. It might be helpful insight for investigation. <br/> <br/> I tried two extra workloads: <br/> -- reduced ratio of write operations (50% -&gt; 20%) <br/> -- reduced total ops/sec (4K -&gt; 2K) <br/> <br/> It didn&#39;t help, disk size is growing while compaction doesn&#39;t catch up. <br/> <br/> I&#39;m trying 1K ops/sec workload now. Aaron, it appears to be that &quot;heavy I/O + limited resource capacity&quot; might not be culprit for data compaction slowness. Please take a look at Pavel finding so far and provide your insight. This is becoming a high priority issue. Please assign it back to Pavel after you input. Thanks! <br/> Thanks Pavel. That is quite helpful. From users perspective, double replication but tune down the workload by &gt;50% should not give very different performance results. <br/> <br/> I am not sure the resource limitation comes from. Here are two possible sources in my mind <br/> <br/> 1. Double replicators on each node due to 2nd bucket bring a lot more overhead in Erlang. <br/> 2. The overhead of scheduling compaction of two buckets are much more than twice of compacting single bucket <br/> <br/> Pavel, one way to isolate the problem is to reduce the # of replicators from 32 to16, thus in the case of two replications, # of replicators is still 32 per node which is the same as single bucket. That will in some extent remove the impact of 1), and if the compaction cannot still catch up, probably the culprit is 2). <br/> <br/> Another way is to reduce the frequency of compactions, by doubling the interval of compaction to see if there is any difference. <br/> <br/> The most recent results with 1K ops/sec: disk data size grew from 11GB to 22GB in 6 hours with rather pessimistic perspectives. <br/> <br/> <a href="http://qa.hq.northscale.net/job/eperf-graph-loop/1396/artifact/xperf-mixed-uni-2-nodes-low-3.loop_2.0.1-145-rel-enterprise_2.0.1-145-rel-enterprise_SOURCE_Jan-30-2013_16%3A29%3A56.pdf">http://qa.hq.northscale.net/job/eperf-graph-loop/1396/artifact/xperf-mixed-uni-2-nodes-low-3.loop_2.0.1-145-rel-enterprise_2.0.1-145-rel-enterprise_SOURCE_Jan-30-2013_16%3A29%3A56.pdf</a> <br/> <br/> <a href="http://qa.hq.northscale.net/job/eperf-graph-loop/1397/artifact/xperf-mixed-uni-2-nodes-low-3.loop_2.0.1-145-rel-enterprise_2.0.1-145-rel-enterprise_DEST_Jan-30-2013_16%3A49%3A21.pdf">http://qa.hq.northscale.net/job/eperf-graph-loop/1397/artifact/xperf-mixed-uni-2-nodes-low-3.loop_2.0.1-145-rel-enterprise_2.0.1-145-rel-enterprise_DEST_Jan-30-2013_16%3A49%3A21.pdf</a> <br/> <br/> There are diags as well: <br/> <br/> <a href="http://qa.hq.northscale.net/job/xperf-win/43/artifact/">http://qa.hq.northscale.net/job/xperf-win/43/artifact/</a> and 30 GB after 12 hours. @Filipe I&#39;m not familiar with the retry compaction thing. Does it look like this issue is more with retry compaction than initial compaction? Per bug scrubs this issue might have fixed by the recent optimization fix, <a href="http://www.couchbase.com/issues/browse/MB-7671" title="We&#39;re sometimes compacting perfectly compact vbuckets"><strike>MB-7671</strike></a>. Assign it to Alex for back-porting the fix to 2.0.1 branch and rerun the test. Hi Aliaksey, Alk mentioned that <a href="http://www.couchbase.com/issues/browse/MB-7671" title="We&#39;re sometimes compacting perfectly compact vbuckets"><strike>MB-7671</strike></a> might have fixed the issue described above. Please backport the fix to 2.0.1 branch and let Pavel know so he can verify it. This is a high priority issue for 2.0.1 release. Thanks much! Just merged backport to 2.0.1 branch: <a href="http://review.couchbase.org/#/c/24391/">http://review.couchbase.org/#/c/24391/</a> Pavel - Aliaksey merged his code change that optimizes data compaction. I am marking this as &quot;resolved&quot; for now but please make sure to validate the fix with running the test. Thanks. Please see above comments. reproduced in 2.0.1-153. Pavel, Aaron is working on some optimizations for data compaction that target for 2.0.2 and beyond. In the mean time, do you happen to concur with doing Junyi&#39;s suggestion below? <br/> At least we may want to give a try with less number of replicators (from 32 - 16)? Please advise and assign it back to Jin or Aaron for tracking this with coming optimizations. <br/> <br/> ======================================================================================= <br/> Pavel, one way to isolate the problem is to reduce the # of replicators from 32 to16, thus in the case of two replications, <br/> # of replicators is still 32 per node which is the same as single bucket. That will in some extent remove the impact of 1), <br/> &nbsp;and if the compaction cannot still catch up, probably the culprit is 2). <br/> ======================================================================================= Sorry, I missed that comment from Junyi. Ok, I will try 16 replicators.<br/> <br/> I&#39;d also recommend @ronnie to run KV test with 2 buckets on Windows w/o XDCR. Thanks, will forward your suggestion to Ronnie &amp; performance team. compaction did kick in in kv testcase @ronnie<br/> Can you describe your workload and environment configuration. 4 windows vms, 2 buckets. active resident ratio : 70% <br/> <br/> details<br/> <br/> <a href="https://github.com/couchbase/testrunner/blob/master/conf/perf/mixed-2suv-2buckets.conf">https://github.com/couchbase/testrunner/blob/master/conf/perf/mixed-2suv-2buckets.conf</a> Ok, Ronnie&#39;s workload is way more aggressive so it&#39;s not just &quot;windows 2 buckets&quot; issue.<br/> <br/> 16 replicators didn&#39;t help. Now it makes sense to try physical environment, it will be my next step. Thanks Ronnie and Pavel for your time and help on investigating this. <br/> <br/> Summarize what we have found so far:<br/> * Overhead from having multiple buckets doesn&#39;t seem to be root cause<br/> * Reducing number of replicators doesn&#39;t seem to be root cause either<br/> * However, this seems to be Windows + XDCR only issue - KV only doesn&#39;t manifest the same symptom<br/> * easy to reproduce, so users may run into this issue fairly easily on Windwos env<br/> <br/> Based on the fact that it is Windows + XDCR only issue (and per bug scrubs) we move it to Windows to do list for 2.0.2. Pavel, please assign it to Sriram after your test run with physical environment. Sriram/Jin will coordinate who/how we address this issue in 2.0.2. Thanks! We need to understand the behavior on physical hardware. Pavel to try to address this for this sprint. Integrated in ui-testing #11 (See [<a href="http://qa.hq.northscale.net/job/ui-testing/11/">http://qa.hq.northscale.net/job/ui-testing/11/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.couchbase.com/issues/browse/MB-7074" title="xdcr: data compaction doesn&#39;t catch up">MB-7074</a>: add config with 16 replicators (Revision 530e1e63f1471bfe9f0994e20888e25aa5207802)<br/> <a href="http://www.couchbase.com/issues/browse/MB-7074" title="xdcr: data compaction doesn&#39;t catch up">MB-7074</a>: add terra-win-xdcr config (Revision 689d09c1ba25a648e1cda909a61a8f57fecb13ed)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> pavelpaulau : <br/> Files : <br/> * conf/perf/xperf-mixed-uni-2-nodes-low-4-16.conf<br/> <br/> pavelpaulau : <br/> Files : <br/> * resources/perf/terra-win-xdcr.ini<br/> From 2.0.2 development kickoff meeting today:<br/> * NS SRV team has made some changes that should alleviate this issue in the latest 2.0.1. <br/> * Before further investigation we should rerun the test with the latest 2.0.1 (or 2.0.2) and verify if the symptom still persist.<br/> * If so, first please assign this to NS SRV team for their initial triage 1. The only physical machine with windows that we have occupied by system tests team and Siri.<br/> 2. Problem still exist in 2.0.1 release.<br/> <br/> What kind of input does ns_server team need for investigation? Thanks for update Pavel, but in fragmentation graph from Feb I&#39;m seeing fragmentation staying below 30 versus November staying above 50. <br/> <br/> Please state more specifically how you concluded that problem still exists in 2.0.1? That data is all ns_server team needs as of now. Sorry, attaching valid reports with reduced front-end workload. <br/> <br/> Looks like fragment was attached by mistake. Indeed. Looked at graphs and it&#39;s clear that issue was not fixed. My thinking was that given previously autocompactor compacted _all_ vbuckets even if just few of them actually needed attention and given that xdcr was producing exactly that (few vbuckets fragmented others largely untouched) I was expecting our fix in autocompactor to address that. Apparently this is not the case. per bug scrub: Alk --- are there any more known optimization that can be made for 2.0.2? pls advise. Damien and Aaron were working on some plausibly looking speedup of DB compaction. I&#39;m not familiar with their results. From ns_server side I have nothing in mind for helping this case. This was originally assigned on Damien and I&#39;m returning it to Damien. per bug scrub: not yet supported in windows.<br/> QE action item: verify this is not happening in linux. if it is, update this bug. Looks like somebody updated wrong ticket. This bug has nothing to do with windows or linux. Abhinav, are you seeing this compaction issue in xdcr in 2.0.2? you can close this bug if you are not seeing this issue. Thanks. Dependency Date of First Response Thu, 1 Nov 2012 15:43:08 -0500 Rank 430 Sprint 1 [MB-7149] cbbackup loops infinitely http://www.couchbase.com/issues/browse/MB-7149 Couchbase Server I&#39;m trying to backup my cluster via cbbackup. I start the backup via <br/> /opt/couchbase/bin/cbbackup <a href="couchbase://Administrator:">couchbase://Administrator:</a><a href='mailto:password@mymachine'>password@mymachine</a>:8091 /backups/couchbase_backup_test <br/> The backup appears to work fine and a progress bar appears, but then exceeds 100% progress and never stops! Example: <br/> ^Cinterrupted.###############################] 210.8% (26141950/12403784 msgs) <br/> (I ctrl-C&#39;d to kill it after 200% because it seems that this can&#39;t possibly work. Note that both the percent and the number of messages are off). I only have ~12 million items in the bucket, but it went right past that limit when backing up. <br/> Help? Linux MB-7149 cbbackup loops infinitely Bug Blocker Reopened Unresolved Public Bin Cui Mike Wiederhold Sat, 10 Nov 2012 13:53:40 -0600 Tue, 21 May 2013 13:05:52 -0500 2.0-beta-2 2.0.2 tools 0 6 There&#39;s a couple things going on here that probably need documentation... <br/> <br/> - cbbackup first contacts the server to get the # of items. But, if the cluster is changing (there are item mutations), that # of items will just be an estimate. <br/> <br/> - then, cbbackup uses the TAP protocol to perform the backup. But, under some conditions (not all item values are resident in memory), the TAP protocol might actually send duplicate messages. That&#39;s why cbbackup reports &quot;msgs&quot; for progress instead of &quot;items&quot; in its numerator, but uses &quot;items&quot; in its denominator. That can lead to &gt;100% in some cases. <br/> <br/> Whether it leads to &gt;200% is somewhat unexpected, but it depends on the situation and what couchbase server is doing in generating the TAP stream. <br/> The 2.0.2 filter isn&#39;t correct at the moment. Putting this into 2.0.1 for the moment as it&#39;ll be revisited again. (I am the original poster) <br/> <br/> Changing the line above to use an IP address rather than hostname seems to have fixed the problem. My backups now run to 100% and then complete as expected. <br/> <br/> As for the root cause: I don&#39;t believe it has anything to do with the cluster changing, since I first encountered this when trying to backup an essentially idle cluster. Verified on a multi-node cluster that cbtransfer get the total item number correctly. And fail to reproduce the bug on the idle cluster. I asked the user on the forums for more information to reproduce this issue. I will post the information here if and when he responds. deferring to 2.1 per bug scrub meeting ( Dipti &amp; Farshid -December 7th ) I have the same problem. <br/> Observed progress over 32000%. <br/> The expected msgs to save is far less than the actual msgs saved. <br/> Restore will load the same number of msgs as were actually saved. <br/> This will impact backup and restore time. <br/> This will impact diskspace. Version info: 2.0.0 community edition (build-1723) <br/> Using ip-address (local,external) or hostname (localhost) does not make any difference, issue remains. bug scrub: Bin -- have you had a chance to take a look? pls update.<br/> thanks. Cannot reproduce it in house. per bug scrub: abhinav -- can you please repro in latest 2.0.2 build? thanks. Cannot reproduce on 2.0.2-749-rel.<br/> - 3 nodes, 2 buckets<br/> [<a href='mailto:root@orange-11601'>root@orange-11601</a> ~]# /opt/couchbase/bin/cbbackup <a href="couchbase://Administrator:">couchbase://Administrator:</a><a href='mailto:password@localhost'>password@localhost</a>:8091 ~/backup<br/> &nbsp;&nbsp;[####################] 100.0% (8557766/8558766 msgs)<br/> bucket: default, msgs transferred...<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;: total | last | per sec<br/> &nbsp;batch : 10657 | 10657 | 14.7<br/> &nbsp;byte : 1131794345 | 1131794345 | 1562011.5<br/> &nbsp;msg : 8558766 | 8558766 | 11812.1<br/> &nbsp;&nbsp;[####################] 100.0% (2024739/2024739 msgs)<br/> bucket: saslbucket, msgs transferred...<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;: total | last | per sec<br/> &nbsp;batch : 14775 | 14775 | 86.0<br/> &nbsp;byte : 1367390279 | 1367390279 | 7955075.3<br/> &nbsp;msg : 10583505 | 10583505 | 61571.7<br/> done<br/> <br/> not reproducible. First, the error itself is harmless. The tool tried to transfer design docs and the source cluster doesn&#39;t contain any. Since 2.0.2, customer can specify --data-only option for cbtransfer/cbback/cbrestore tool.<br/> <br/> But we still dont know the root cause why there is such a big difference between the initial msgs to be sent and the final msgs that are transferred. One possible explanation about the deviant number: <br/> <br/> 1. the estimate number is the total active item number <br/> 2. the actual msg tranferred = total(tap_mutations + tap_delete) <br/> <br/> For the above customer case where they have 2 million item deleted, we will transfer not only the current active items, but also any deleted items. <br/> At again, there will be more msgs transferred if any key will have repeated set/deletions. Reopening for visibility. Whether the tool is doing the &quot;right&quot; thing or not, there is still a major impact to the user both in terms of disk space being taken up, time being taken and perception of confidence, etc in the backup. Anil to work with Bin on customer use case. Talked to Bin, here&#39;s the update. <br/> <br/> There are 2 issues here… <br/> 1). In case of heavy DGM – tool only captures the &#39;total active items&#39; in memory and not include the items on disk. Fix for this is to consider also resident-ratio to get the current_item. As per Bin, we have already the stats and this should be low-risk fix. He will be making this fix for 2.0.2. <br/> 2). In case of Deletes on items – tool currently only captures the snapshot of &#39;active items&#39; and doesn&#39;t consider any items getting deleted. Hence when it transfers it not only transfers current active items but also any deleted items which is unnecessary. To fix this we require some changes in EP-Engine side to provide stats on deleted items so that tool can smartly ignore those. Considering the timeframe for release this won&#39;t make it for 2.0.2 but we will have documentation explaining this to users. [Doc ticket on Karen ] <br/> <br/> When cbbackup was run on a node with 6104188 items (~45% active resident ratio),<br/> <br/> <a href='mailto:root@plum-009'>root@plum-009</a>:~# /opt/couchbase/bin/cbbackup <a href="http://localhost:8091">http://localhost:8091</a> /data/backup<br/> &nbsp;&nbsp;[##############################] 147.6% (9007021/6104188 msgs)<br/> bucket: default, msgs transferred...<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;: total | last | per sec<br/> &nbsp;batch : 29230 | 29230 | 33.1<br/> &nbsp;byte : 10030064183 | 10030064183 | 11365547.3<br/> &nbsp;msg : 9007021 | 9007021 | 10206.3<br/> done<br/> <a href="http://review.couchbase.org/#/c/26431/">http://review.couchbase.org/#/c/26431/</a> per bug triage, this fix from bin only addresses item 1 (see below):<br/> 1). In case of heavy DGM – tool only captures the &#39;total active items&#39; in memory and not include the items on disk. Fix for this is to consider also resident-ratio to get the current_item. As per Bin, we have already the stats and this should be low-risk fix. He will be making this fix for 2.0.2. Date of First Response Mon, 12 Nov 2012 15:00:53 -0600 Rank 330 Sprint 7 [MB-7168] [Doc'd] failover of node that's completely down is still not quick (was: Rebalance exited with reason {not_all_nodes_are_ready_yet after failover node) http://www.couchbase.com/issues/browse/MB-7168 Couchbase Server version=2.0.0-1947-rel <br/> <a href="http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/448/consoleFull">http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/448/consoleFull</a> <br/> /testrunner -i resources/jenkins/centos-64-5node-failover.ini get-logs=True,GROUP=BAT -t failovertests.FailoverTests.test_failover_firewall,replica=1,keys_count=100000,dgm_run=True,GROUP=BAT <br/> <br/> steps: <br/> 1.3 nodes in cluster:10.1.3.114, 10.1.3.118,10.1.3.116 <br/> 2.enabled firewall on ip:10.1.3.118 : /sbin/iptables -A INPUT -p tcp -i eth0 --dport 1000:60000 -j REJECT <br/> 3 <br/> [2012-11-12 11:28:37,102] - [failovertests:260] INFO - node 10.1.3.118:8091 is &#39;unhealthy&#39; as expected <br/> [2012-11-12 11:29:07,544] - [rest_client:849] INFO - fail_over successful <br/> [2012-11-12 11:29:07,545] - [failovertests:278] INFO - failed over node : <a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a> <br/> [2012-11-12 11:29:07,545] - [failovertests:292] INFO - 10 seconds sleep after failover before invoking rebalance... <br/> 4.[2012-11-12 11:29:17,545] - [rest_client:883] INFO - rebalance params : password=password&amp;ejectedNodes=ns_1%4010.1.3.118&amp;user=Administrator&amp;knownNodes=ns_1%4010.1.3.114%2Cns_1%4010.1.3.118%2Cns_1%4010.1.3.116 <br/> <br/> result: <br/> [2012-11-12 11:29:17,562] - [rest_client:890] INFO - rebalance operation started <br/> [2012-11-12 11:29:17,569] - [rest_client:986] INFO - rebalance percentage : 0 % <br/> [2012-11-12 11:29:19,573] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:21,577] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:23,584] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:25,589] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:27,593] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:29,599] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:31,603] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:33,607] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:35,612] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:37,616] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:39,621] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:41,626] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:43,630] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:45,635] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:47,640] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:49,644] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:51,648] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:53,652] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:55,657] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:57,662] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:29:59,667] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:30:01,671] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:30:03,676] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:30:05,682] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:30:07,698] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:30:09,703] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:30:11,708] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:30:13,712] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:30:15,716] - [rest_client:986] INFO - rebalance percentage : 0.0 % <br/> [2012-11-12 11:30:17,721] - [rest_client:971] ERROR - {u&#39;status&#39;: u&#39;none&#39;, u&#39;errorMessage&#39;: u&#39;Rebalance failed. See logs for detailed reason. You can try rebalance again.&#39;} - rebalance failed <br/> <br/> [rebalance:info,2012-11-12T11:49:03.581,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:&lt;0.20240.1&gt;:ns_rebalancer:rebalance:258]Waiting for bucket &quot;default&quot; to be ready on [&#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39;] <br/> [ns_server:info,2012-11-12T11:49:09.386,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:&lt;0.769.0&gt;:ns_orchestrator:handle_info:282]Skipping janitor in state rebalancing: {rebalancing_state,&lt;0.20215.1&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{dict,3,16,16,8,80,48, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{[],[],[],[],[],[],[],[],[],[],[],[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[],[],[],[]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{{[],[],[],[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[],[],[],[],[],[],[]}}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39;,&#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39;], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39;]} <br/> [ns_server:info,2012-11-12T11:49:19.386,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:&lt;0.769.0&gt;:ns_orchestrator:handle_info:282]Skipping janitor in state rebalancing: {rebalancing_state,&lt;0.20215.1&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{dict,3,16,16,8,80,48, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{[],[],[],[],[],[],[],[],[],[],[],[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[],[],[],[]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{{[],[],[],[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[],[],[],[],[],[],[]}}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39;,&#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39;], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39;]} <br/> [ns_server:info,2012-11-12T11:49:23.120,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:&lt;0.20319.1&gt;:compaction_daemon:try_to_cleanup_indexes:439]Cleaning up indexes for bucket `default` <br/> [ns_server:info,2012-11-12T11:49:23.121,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:&lt;0.20319.1&gt;:compaction_daemon:spawn_bucket_compactor:404]Compacting bucket default with config: <br/> [{database_fragmentation_threshold,{30,undefined}}, <br/> &nbsp;{view_fragmentation_threshold,{30,undefined}}] <br/> [ns_server:info,2012-11-12T11:49:24.348,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:ns_config_rep&lt;0.358.0&gt;:ns_config_rep:do_pull:341]Pulling config from: &#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39; <br/> <br/> [ns_server:info,2012-11-12T11:49:29.386,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:&lt;0.769.0&gt;:ns_orchestrator:handle_info:282]Skipping janitor in state rebalancing: {rebalancing_state,&lt;0.20215.1&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{dict,3,16,16,8,80,48, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{[],[],[],[],[],[],[],[],[],[],[],[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[],[],[],[]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{{[],[],[],[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[],[],[],[],[],[],[]}}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39;,&#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39;], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39;]} <br/> [ns_server:warn,2012-11-12T11:49:29.546,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:capi_set_view_manager-default&lt;0.15508.1&gt;:capi_set_view_manager:handle_info:345]Remote server node {&#39;capi_ddoc_replication_srv-default&#39;,&#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39;} process down: noconnection <br/> [ns_server:warn,2012-11-12T11:49:29.546,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:xdc_rdoc_replication_srv&lt;0.470.0&gt;:xdc_rdoc_replication_srv:handle_info:128]Remote server node {xdc_rdoc_replication_srv,&#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39;} process down: noconnection <br/> [user:warn,2012-11-12T11:49:29.546,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:ns_node_disco&lt;0.351.0&gt;:ns_node_disco:handle_info:168]Node &#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39; saw that node &#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39; went down. <br/> [error_logger:error,2012-11-12T11:49:29.546,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:error_logger&lt;0.5.0&gt;:ale_error_logger_handler:log_msg:76]** Node &#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39; not responding ** <br/> ** Removing (timedout) connection ** <br/> <br/> [ns_server:info,2012-11-12T11:49:30.536,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:ns_config_rep&lt;0.358.0&gt;:ns_config_rep:do_pull:341]Pulling config from: &#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39; <br/> <br/> [ns_server:info,2012-11-12T11:49:39.386,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:&lt;0.769.0&gt;:ns_orchestrator:handle_info:282]Skipping janitor in state rebalancing: {rebalancing_state,&lt;0.20215.1&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{dict,3,16,16,8,80,48, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{[],[],[],[],[],[],[],[],[],[],[],[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[],[],[],[]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{{[],[],[],[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[],[],[],[],[],[],[]}}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39;,&#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39;], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39;]} <br/> [ns_server:info,2012-11-12T11:49:49.386,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:&lt;0.769.0&gt;:ns_orchestrator:handle_info:282]Skipping janitor in state rebalancing: {rebalancing_state,&lt;0.20215.1&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{dict,3,16,16,8,80,48, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{[],[],[],[],[],[],[],[],[],[],[],[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[],[],[],[]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{{[],[],[],[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[],[],[],[],[],[],[]}}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39;,&#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39;], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39;]} <br/> [ns_server:info,2012-11-12T11:49:53.135,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:&lt;0.20445.1&gt;:compaction_daemon:try_to_cleanup_indexes:439]Cleaning up indexes for bucket `default` <br/> [ns_server:info,2012-11-12T11:49:53.136,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:&lt;0.20445.1&gt;:compaction_daemon:spawn_bucket_compactor:404]Compacting bucket default with config: <br/> [{database_fragmentation_threshold,{30,undefined}}, <br/> &nbsp;{view_fragmentation_threshold,{30,undefined}}] <br/> [ns_server:info,2012-11-12T11:49:59.386,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:&lt;0.769.0&gt;:ns_orchestrator:handle_info:282]Skipping janitor in state rebalancing: {rebalancing_state,&lt;0.20215.1&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{dict,3,16,16,8,80,48, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{[],[],[],[],[],[],[],[],[],[],[],[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[],[],[],[]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{{[],[],[],[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[&#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39;|0.0]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[],[],[],[],[],[],[]}}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39;,&#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39;], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.1.3.118'>ns_1@10.1.3.118</a>&#39;]} <br/> [user:info,2012-11-12T11:50:03.582,<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>:&lt;0.769.0&gt;:ns_orchestrator:handle_info:319]Rebalance exited with reason {not_all_nodes_are_ready_yet, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39;,&#39;<a href='mailto:ns_1@10.1.3.116'>ns_1@10.1.3.116</a>&#39;]} <br/> <br/> <br/> <br/> <br/> MB-7168 [Doc'd] failover of node that's completely down is still not quick (was: Rebalance exited with reason {not_all_nodes_are_ready_yet after failover node) Bug Blocker Reopened Unresolved Public Aleksey Kondratenko Andrei Baranouski Tue, 13 Nov 2012 05:16:46 -0600 Mon, 20 May 2013 12:46:14 -0500 2.0 2.0.1 2.0.2 2.1 0 9 almost all test fail due to rebalance failure after failover nodes on version=2.0.0-1949-rel <br/> <a href="http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/449/consoleFull">http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/449/consoleFull</a> if this is an automated test please rerun and confrm the behavior twice per bug-scrub2 3 tests are constantly falling with the same reason <a href="http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/457/consoleFull">http://qa.hq.northscale.net/job/centos-64-2.0-failover-tests/457/consoleFull</a> ( 1954 build) can be easily reproduced for windows - more info is in duplicate bug <a href="http://www.couchbase.com/issues/browse/MB-7205" title="Rebalance exited with reason {not_all_nodes_are_ready_yet,[&#39;ns_1@10.1.3.147&#39;]}"><strike>MB-7205</strike></a> (reproduced manually) so this failure occurs when the other node is unavailable ? <br/> <br/> Iryna , when you reproduce this on windows manually does the rebalance succeed second time ? <br/> do you see any data loss ? does rebalance fail immediately ? it not succeed after retrying during some time window, after about 15 minutes it succeed <br/> there is no data loss, <br/> rebalance fails in first 1-5 mins of rebalancing I cannot see any clear evidence of what caused this. But it looks bad enough. 60 seconds timeout to query vbucket states on both nodes was hit here. <br/> <br/> I also see that janitor pass immediately preceding rebalance also failed but in different phase, where it was waiting for vbucket change requests to be complete also failed on both nodes. And this timeout is 30 seconds. <br/> <br/> <br/> I&#39;d like diag to be grabbed _immediately_ after rebalance fails so that I could see what is the state of janitor_ageng and ns_memcached on each node. I.e. without doing any cleanup please. May I have that ? See above logs immediately after rebalance failure rebalance is successful in second time It&#39;s not quite immediately, sadly. Few seconds after problem. And backtraces on .118 (node where we failed waiting) don&#39;t have anything bad sadly. given that rebalance succeeded second time moving this to 2.0.1 Andrei,<br/> <br/> is this test fialing with the latest 2.0.1 build ? yes, 3 tests constantly fall in failover job: <a href="http://qa.hq.northscale.net/view/2.0.1/job/centos-64-2.0-failover-tests/501/consoleFull">http://qa.hq.northscale.net/view/2.0.1/job/centos-64-2.0-failover-tests/501/consoleFull</a> 2.0.1-120-rel per bug scrub <br/> <br/> retrying rebalance after a few minutes should work. Aliaksey, <br/> <br/> i know this was discussed before but we want to confirm what the test does with your conclusion <br/> <br/> we put the node behind the firewall , then wait until node is marked as unhealthy by NS_SERVER <br/> then failover this node and click on rebalance to eject the node ( ejectedNodes = 10.1.3.118 ) <br/> <br/> we want to know why when the node was already failed over from the cluster we wait for it be ready as part of rebalance. ( or the timeouts happened on existing nodes ? ) <br/> <br/> This is quite subtle to explain fully, but main problem is we have to wait until failover actually completes internally and that is subject to timeouts in certain areas.<br/> <br/> So if you keep trying for 2-3 minutes it should eventually work. &gt;&gt;but main problem is we have to wait until failover actually completes internally and that is subject to timeouts in certain areas. <br/> so when failover REST api returns it does not mean that failover process internally is completed. <br/> is failover a synchronous call or asynchronous ? in case its asyncronous is there a way for a test to check before initiating a rebalance so that we have a test that is more deterministic It is best-effort sync. If it fails to be sync with reasonably short timeout, it&#39;ll silently become async.<br/> <br/> There&#39;s no way to detect that right now. per bug scrub - assigning to Jin A few things came out of the engineering talk regarding this issue: <br/> 1) This is a good catch in that it is confirming we need a better way of handling the api since it cannot %100 warranty the completion of failover. <br/> 2) However, it is not a critical or blocker since the symptom is more obvious (highly probable) while running an automated testing case. <br/> 3) Only feasible approach (per ns server team) for now is to wait and retry. <br/> <br/> Based on this and the fact the fix would require changes across components (ep engine, etc) we may want to consider to put this into a future enhancement. <br/> Assign this to Yaseen for his input here. Pelase assign it back to Jin or Dipti afterwards. Thanks. Please see the above comment. Thanks Jin for confirming this. I think this expected behavior can be included in the release notes as well so that users and support team can be aware of the issue and the suggested workaround. <br/> <br/> Andrei, <br/> can you then modify the test accordingly. <br/> <br/> per bug scrub : <br/> <br/> please revise the test accordingly and after running the test for few times can you propose how long customer should wait before kicking the rebalance again Please assign it back to Jin after having enough information for recommendation. Will follow up with the doc team. <br/> with timeout 90 sec after failover befor rebalance - tests passed <br/> will launch with 60 sec can you try with lower timeout ( 10 seconds for instance ) ? 10 secs was in initial state for ticket. <br/> <br/> [2012-11-12 11:29:07,545] - [failovertests:292] INFO - 10 seconds sleep after failover before invoking rebalance... <br/> <br/> 60 secs tests passed <br/> <br/> will try with 30 sec any update on the test w/ 30 sec? 30 sec, build 151, tests passed <a href="http://qa.hq.northscale.net/view/2.0.1/job/centos-64-2.0-failover-tests/536/">http://qa.hq.northscale.net/view/2.0.1/job/centos-64-2.0-failover-tests/536/</a> Karen,<br/> <br/> could you please add this to the release notes that the user needs to wait for 30 seconds before they attempt to run rebalance operation after failing over a node.<br/> comment son this bug should explain under what conditions this 30 seconds delay is neeeded Jin, <br/> <br/> <br/> Can you summarize the situation when someone should wait 30 seconds to reattempt rebalance? <br/> <br/> <br/> Thanks! Failover REST api is sync operation with timeout. When it fails to complete the failover process within the timeout period, it internally switches to async operation (continues the failover to completion) and immediately returns. Subsequent rebalance in this case would fail because the failover process is still running. User can wait between 30 seconds upto a minute and reattempt rebalance. The failover REST api timeout is 30 second. &lt;para&gt; <br/> A cluster rebalance may exit and produce the error <br/> {not_all_nodes_are_ready_yet} if you perform the rebalance right <br/> after failing over a node in the cluster. You may need to <br/> wait 30 seconds after the node failover before you <br/> attempt the cluster rebalance. <br/> &lt;/para&gt; <br/> &lt;para&gt;This is because the failover REST API is a synchronous operation with a timeout. If it fails to <br/> complete the failover process by the timeout, the operation internally switches into a <br/> asynchronous operation. It will immediately return and re-attempt failover in the background which will cause <br/> rebalance to fail since the failover operation is still running.&lt;/para&gt; Added to RN as : <br/> <br/> &lt;para&gt; <br/> A cluster rebalance may exit and produce the error <br/> {not_all_nodes_are_ready_yet} if you perform the rebalance right <br/> after failing over a node in the cluster. You may need to <br/> wait 30 seconds after the node failover before you <br/> attempt the cluster rebalance. <br/> &lt;/para&gt; <br/> &lt;para&gt;This is because the failover REST API is a synchronous operation with a timeout. If it fails to <br/> complete the failover process by the timeout, the operation internally switches into a <br/> asynchronous operation. It will immediately return and re-attempt failover in the background which will cause <br/> rebalance to fail since the failover operation is still running.&lt;/para&gt; Jin, I got the same error with 30 seconds waiting after failover on build 2.0.2-761-rel<br/> Should we update RN for 2.0.1/next 2.0.2 release?<br/> <a href="http://qa.hq.northscale.net/view/2.0.1/job/centos-64-2.0-failover-tests/583/consoleFull">http://qa.hq.northscale.net/view/2.0.1/job/centos-64-2.0-failover-tests/583/consoleFull</a><br/> <br/> [2013-04-08 19:41:13,981] - [failovertests:148] INFO - failed over node : <a href='mailto:ns_1@10.1.3.115'>ns_1@10.1.3.115</a><br/> [2013-04-08 19:41:13,981] - [failovertests:163] INFO - 30 seconds sleep after failover before invoking rebalance...<br/> [2013-04-08 19:41:43,981] - [rest_client:834] INFO - rebalance params : password=password&amp;ejectedNodes=ns_1%4010.1.3.115&amp;user=Administrator&amp;knownNodes=ns_1%4010.1.3.114%2Cns_1%4010.1.3.115%2Cns_1%4010.1.3.118%2Cns_1%4010.1.3.116<br/> [2013-04-08 19:41:43,991] - [rest_client:838] INFO - rebalance operation started<br/> [2013-04-08 19:41:44,004] - [rest_client:940] INFO - rebalance percentage : 0 %<br/> [2013-04-08 19:41:46,009] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:41:48,014] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:41:50,018] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:41:52,023] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:41:54,029] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:41:56,033] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:41:58,039] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:00,043] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:02,048] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:04,056] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:06,061] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:08,066] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:10,071] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:12,077] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:14,084] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:16,089] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:18,096] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:20,102] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:22,108] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:24,113] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:26,119] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:28,125] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:30,130] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:32,137] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:34,143] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:36,152] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:38,157] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:40,163] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:42,168] - [rest_client:940] INFO - rebalance percentage : 0.0 %<br/> [2013-04-08 19:42:44,174] - [rest_client:923] ERROR - {u&#39;status&#39;: u&#39;none&#39;, u&#39;errorMessage&#39;: u&#39;Rebalance failed. See logs for detailed reason. You can try rebalance again.&#39;} - rebalance failed<br/> [2013-04-08 19:42:44,175] - [rest_client:924] INFO - Latest logs from UI:<br/> [2013-04-08 19:42:44,273] - [rest_client:925] ERROR - {u&#39;node&#39;: u&#39;<a href='mailto:ns_1@10.1.3.114'>ns_1@10.1.3.114</a>&#39;, u&#39;code&#39;: 2, u&#39;text&#39;: u&quot;Rebalance exited with reason {not_all_nodes_are_ready_yet,\n it&#39;s happened twice in this run Thanks for the update, Andrei. Before we update the RN first let&#39;s figure out how long a user should wait before retrying the rebalance. Can we upgrade the wait period to 1 minute and see if that is long enough? Thanks. set timeout in 1 min <a href="http://review.couchbase.org/#/c/25587/">http://review.couchbase.org/#/c/25587/</a><br/> before that we had a timeout for 60 sec, the tests do not fall<br/> <a href="http://www.couchbase.com/issues/browse/MB-7168?focusedCommentId=49128&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-49128">http://www.couchbase.com/issues/browse/MB-7168?focusedCommentId=49128&amp;page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-49128</a> Integrated in ui-testing #42 (See [<a href="http://qa.hq.northscale.net/job/ui-testing/42/">http://qa.hq.northscale.net/job/ui-testing/42/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS tested with timeout for 60 sec, all tests passed Jin, I think to add in RN waiting for 60 seconds after failover should be okay Karen, please update the release notes to suggest 60 sec (not 30). Also lowering the priority from Blocker to Critical as it&#39;s not a 2.0.2 release blocking issue. Ok, changed to : A cluster rebalance may exit and produce the error <br/> {not_all_nodes_are_ready_yet} if you perform the rebalance right <br/> after failing over a node in the cluster. You may need to <br/> wait 60 seconds after the node failover before you <br/> attempt the cluster rebalance.<br/> <br/> in RN 2.0.2 added to RN 2.0.2 Integrated in win-ui-testing-P0 #56 (See [<a href="http://qa.hq.northscale.net/job/win-ui-testing-P0/56/">http://qa.hq.northscale.net/job/win-ui-testing-P0/56/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.couchbase.com/issues/browse/MB-7168" title="[Doc&#39;d] failover of node that&#39;s completely down is still not quick (was: Rebalance exited with reason {not_all_nodes_are_ready_yet after failover node)">MB-7168</a>: sleep 30 sec after reb falied when killed memcached (Revision 44f755b962b7987d5b972caf9a283baa95edaed1)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> andrei : <br/> Files : <br/> * pytests/swaprebalance.py<br/> Verified that the doc change has also gone to 2.0.1 RN. <br/> <br/> <a href="http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-server-rn_2-0-0l.html">http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-server-rn_2-0-0l.html</a> <br/> <br/> Assigning to PM for the next step. discussed with ALK, this bug will be fixed in 2.1. talked about having UI alert but since this happens only when node dies completely for now Release Note as Known Issue should fine. thanks Updated ticket to reflect badness of this and must-have-ness for 2.1 added flag to include 2.0.1 release note to 2.0.2. Integrated in windows32_sanity_P0 #29 (See [<a href="http://qa.hq.northscale.net/job/windows32_sanity_P0/29/">http://qa.hq.northscale.net/job/windows32_sanity_P0/29/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = UNSTABLE Integrated in windows32_view_P0 #6 (See [<a href="http://qa.hq.northscale.net/job/windows32_view_P0/6/">http://qa.hq.northscale.net/job/windows32_view_P0/6/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = UNSTABLE Date of First Response Mon, 19 Nov 2012 13:26:06 -0600 Flagged Rank 506 [MB-4451] s/Membase/Couchbase/g - replace all file, folder, code mentions from Membase to Couchbase http://www.couchbase.com/issues/browse/MB-4451 Couchbase Server need to fix this before 1.8 MB-4451 s/Membase/Couchbase/g - replace all file, folder, code mentions from Membase to Couchbase Epic Blocker Resolved Won't Fix Public Steve Yen Benjamin Young Thu, 17 Nov 2011 13:26:44 -0600 Thu, 26 Jan 2012 19:12:21 -0600 Wed, 14 Dec 2011 11:15:47 -0600 2.0-beta 1.8.0 ns_server RESTful-APIs UI 0 0 Why do we need to replace all source code mentions of Membase? Only end-user visible stuff is necessary for renaming. Yeah. Just passing along what I was given. I understand that &quot;mb_&quot; prefixed tools should change (and have, I believe). <br/> <br/> I&#39;m guessing that the point was to make the shift during a major point release rather than having these &quot;historic&quot; mentions stay around indefinitely. <br/> <br/> But again, I reported on behalf of someone else. Date of First Response Wed, 14 Dec 2011 11:15:47 -0600 Epic Status Rank 3281 Time In Status [MB-5324] --enable-64bit no longer works due to C++ code http://www.couchbase.com/issues/browse/MB-5324 Couchbase Server We need to set CXXFLAGS in addition to CFLAGS MB-5324 --enable-64bit no longer works due to C++ code Bug Blocker Resolved Fixed Public Trond Norbye Trond Norbye Fri, 18 May 2012 03:00:46 -0500 Mon, 23 Jul 2012 14:52:55 -0500 Fri, 18 May 2012 03:30:00 -0500 1.8.1-release-candidate 1.8.2 2.0-beta 1.8.1-release-candidate 1.8.2 2.0-beta bucket-engine 0 0 <a href="http://review.couchbase.org/#change,16176">http://review.couchbase.org/#change,16176</a><br/> <a href="http://review.couchbase.org/#change,16177">http://review.couchbase.org/#change,16177</a> Rank 5333 Sprint Priority -1.0 Time In Status [MB-5025] flush should return an error until such a time as it works http://www.couchbase.com/issues/browse/MB-5025 Couchbase Server Until such a time as flush works, it should return an error. Optionally, flush should be enabled through cbflushctl in order to support testing with small datasets. <br/> <br/> See also: <br/> <a href="http://www.couchbase.com/issues/browse/MB-4717" title="flush_all doesn&#39;t work on the latest 2.0 builds"><strike>MB-4717</strike></a> and <a href="http://www.couchbase.com/issues/browse/MB-4601" title="flush_all works, but does not block in violation of protocol"><strike>MB-4601</strike></a> MB-5025 flush should return an error until such a time as it works Bug Blocker Resolved Fixed Public Junyi Xie Matt Ingenthron Fri, 6 Apr 2012 20:39:19 -0500 Wed, 13 Jun 2012 10:35:11 -0500 Wed, 23 May 2012 16:50:42 -0500 1.7.2 1.8.0 2.0-developer-preview-3 2.0-developer-preview-4 1.8.1 couchbase-bucket 0 4 Assigning to Frank for triage. assigning to Dipti for a faster decision on this decision is to disable flush_all for 1.8.1 and provide a method for enabling it Fix submitted to Gerrit at link: <br/> <br/> <a href="http://review.couchbase.org/#change,15002">http://review.couchbase.org/#change,15002</a> <br/> <br/> under code review. <br/> <br/> <br/> moxi needs a test against this, as I&#39;m not sure moxi expects an error from flush_all. Hi Steve, <br/> We have already merged the change set. Can you please elaborate how to test moxi against this change set? <br/> <br/> Thanks, <br/> <br/> Junyi <a href="http://review.couchbase.org/#change,15002">http://review.couchbase.org/#change,15002</a> Hi Dipti, <br/> <br/> To enable flushall, you can add this in your the engine parameter file <br/> <br/> &quot;flushall_enabled=true&quot; <br/> <br/> To turn it off, you just need to remove this line. The default is off. <br/> <br/> Thanks. <br/> <br/> Junyi We need to document that we need to enable it on each node separately . <br/> Engine parameters are not a &quot;cluster&quot; parameters. Integrated in github-ep-engine-2-0 #262 (See [<a href="http://qa.hq.northscale.net/job/github-ep-engine-2-0/262/">http://qa.hq.northscale.net/job/github-ep-engine-2-0/262/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.couchbase.com/issues/browse/MB-5025" title="flush should return an error until such a time as it works"><strike>MB-5025</strike></a> disable flushall by default (Revision d9d123f4a54def7cf2cbaed6e0189d9b846002da)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> Junyi Xie : <br/> Files : <br/> * ep_engine.cc<br/> * ep_testsuite.cc<br/> * docs/engine-params.org<br/> * ep_engine.h<br/> * Makefile.am<br/> * configuration.json<br/> Has this not yet been merged into a build? I am unable to toggle the flush_all command with build 789: <br/> <br/> [<a href='mailto:root@localhost'>root@localhost</a> ~]# /opt/couchbase/bin/cbflushctl localhost:11210 set flushall_enabled true <br/> setting flush param: flushall_enabled true <br/> Traceback (most recent call last): <br/> &nbsp;&nbsp;File &quot;/opt/couchbase/lib/python/cbflushctl&quot;, line 87, in ? <br/> &nbsp;&nbsp;&nbsp;&nbsp;c.execute() <br/> &nbsp;&nbsp;File &quot;/opt/couchbase/lib/python/clitool.py&quot;, line 43, in execute <br/> &nbsp;&nbsp;&nbsp;&nbsp;f[0](mc, *args[2:], **opts.__dict__) <br/> &nbsp;&nbsp;File &quot;/opt/couchbase/lib/python/cbflushctl&quot;, line 42, in set_param <br/> &nbsp;&nbsp;&nbsp;&nbsp;mc.set_flush_param(key, val) <br/> &nbsp;&nbsp;File &quot;/opt/couchbase/lib/python/mc_bin_client.py&quot;, line 214, in set_flush_param <br/> &nbsp;&nbsp;&nbsp;&nbsp;return self._doCmd(memcacheConstants.CMD_SET_FLUSH_PARAM, key, val) <br/> &nbsp;&nbsp;File &quot;/opt/couchbase/lib/python/mc_bin_client.py&quot;, line 104, in _doCmd <br/> &nbsp;&nbsp;&nbsp;&nbsp;return self._handleSingleResponse(opaque) <br/> &nbsp;&nbsp;File &quot;/opt/couchbase/lib/python/mc_bin_client.py&quot;, line 97, in _handleSingleResponse <br/> &nbsp;&nbsp;&nbsp;&nbsp;cmd, opaque, cas, keylen, extralen, data = self._handleKeyedResponse(myopaque) <br/> &nbsp;&nbsp;File &quot;/opt/couchbase/lib/python/mc_bin_client.py&quot;, line 93, in _handleKeyedResponse <br/> &nbsp;&nbsp;&nbsp;&nbsp;raise MemcachedError(errcode, rv) <br/> mc_bin_client.MemcachedError: Memcached error #1: Unknown config param <br/> [<a href='mailto:root@localhost'>root@localhost</a> ~]# <br/> Hi Dipiti, <br/> <br/> The fix has been merged to 1.8.1 as CBD-152. <br/> <br/> <a href="http://review.couchbase.org/#change,16283">http://review.couchbase.org/#change,16283</a> <br/> <br/> Let me close this bug if it is ok with you. <br/> <br/> Thanks <br/> <br/> Junyi <a href="http://review.couchbase.org/#change,16283">http://review.couchbase.org/#change,16283</a> Date of First Response Wed, 11 Apr 2012 11:24:30 -0500 Rank 5219 Sprint Priority 2.0 Story Points 1.0 Time In Status [MB-5201] server sends not-my-vbucket replies during warmup rather than tmpfail or something else useful http://www.couchbase.com/issues/browse/MB-5201 Couchbase Server In a recent investigation of something unrelated, I found that the server will send not-my-vbucket replies during warmup. <br/> <br/> Note that in a discussion with Trond, both he and I expected warmup (as currently designed) would send a tmpfail. <br/> <br/> I&#39;ve actually observed that the server sends not-my-vbucket responses. This is bad, as many clients will retry (possibly as fast as they can) and moxi will hit every server twice and give up. <br/> <br/> What is the intended behavior of the server on the memcached port number during warmup? Is that behavior defined anywhere? We need to verify that client reactions are appropriate during warmup. <br/> <br/> Note that Aliaksey said a related issue is document editing from the UI in this state. <br/> MB-5201 server sends not-my-vbucket replies during warmup rather than tmpfail or something else useful Bug Blocker Resolved Fixed Public Chiyoung Seo Matt Ingenthron Tue, 1 May 2012 11:57:44 -0500 Tue, 3 Jul 2012 15:45:46 -0500 Thu, 24 May 2012 01:05:29 -0500 1.8.0 1.8.1 2.0-beta couchbase-bucket 0 2 Given this has always been the case, we will not be fixing in 1.8.1. <br/> <br/> We will fix it in 1.8.2 when we move to fast warmup or in 2.0 which also has fast warmup. <a href="http://review.couchbase.org/#change,16357">http://review.couchbase.org/#change,16357</a> Integrated in github-ep-engine-2-0 #293 (See [<a href="http://qa.hq.northscale.net/job/github-ep-engine-2-0/293/">http://qa.hq.northscale.net/job/github-ep-engine-2-0/293/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.couchbase.com/issues/browse/MB-5201" title="server sends not-my-vbucket replies during warmup rather than tmpfail or something else useful"><strike>MB-5201</strike></a> Convert NOT_MY_VBUCKET to TMP_FAIL response during warmup (Revision 802e2e8fcdfe7bb1ed55c60ac2150ebc03827149)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> Chiyoung Seo : <br/> Files : <br/> * ep_engine.cc<br/> * ep_testsuite.cc<br/> * ep.cc<br/> * ep_engine.h<br/> Date of First Response Fri, 4 May 2012 15:12:06 -0500 Rank 5403 Sprint Priority 2.0 Time In Status [MB-4859] Full backfill of replicas after any restart http://www.couchbase.com/issues/browse/MB-4859 Couchbase Server Just two nodes in cluster_run. I&#39;ve tried with 1k vbuckets and with just 32 (to have more items). 1600 k of bucket quota. <br/> <br/> * Loaded 1.2M of 1k objects (all zero). It&#39;ll be slightly DGM <br/> <br/> * Waited until replica count and item count is ok and until everything is persisted. <br/> <br/> * Waited 10-20 second more. <br/> <br/> * killall -SIGTERM memcached.org <br/> <br/> Observe that replicas are rebuilt: <br/> <br/> rebalance:info] [2012-03-01 15:31:52] [<a href='mailto:n_1@10.17.2.163'>n_1@10.17.2.163</a>:&lt;0.1597.0&gt;:ebucketmigrator_srv:process_upstream:354] Initial stream for vbucket 27 <br/> <br/> and see that replica items count grows from 0 to 1.2 M and write queue is non-zero for some time. <br/> MB-4859 Full backfill of replicas after any restart Bug Blocker Resolved Fixed Public Trond Norbye Aleksey Kondratenko Thu, 1 Mar 2012 17:37:18 -0600 Tue, 10 Apr 2012 21:03:23 -0500 Tue, 6 Mar 2012 09:35:04 -0600 1.8.2 1.8.2 2.0-developer-preview-4 couchbase-bucket 0 0 This bug was introduced in commit 26250d064c714898f12ed2536633c11cd70b9741 Fix in gerrit at <a href="http://review.couchbase.org/13787">http://review.couchbase.org/13787</a> Integrated in github-ep-engine-2-0 #218 (See [<a href="http://qa.hq.northscale.net/job/github-ep-engine-2-0/218/">http://qa.hq.northscale.net/job/github-ep-engine-2-0/218/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.couchbase.com/issues/browse/MB-4859" title="Full backfill of replicas after any restart"><strike>MB-4859</strike></a> Add back the callback for CMD_LAST_CLOSED_CHECKPOINT (Revision fa324e38eee559a86b78abdf1a71458f3f14b5ea)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> Trond Norbye : <br/> Files : <br/> * ep_engine.cc<br/> Date of First Response Tue, 6 Mar 2012 05:51:30 -0600 Rank 5615 Time In Status [MB-7148] cbbackup/cbrestore from 181 to latest 2.0 build, or 181 to 181, with wrong CAS and item_flag value restored. http://www.couchbase.com/issues/browse/MB-7148 Couchbase Server I insert 1000 items to a 181 cluster, do cbbackup and cbrestore them to another 181 cluster using latest membase-cli code. The items in the other 181 cluster are with wrong flag and cas value. The return value for get is (Flag, CAS, item_value) <br/> <br/> import sys <br/> sys.path.append(&quot;.&quot;) <br/> sys.path.append(&quot;lib&quot;) <br/> import mc_bin_client <br/> client = mc_bin_client.MemcachedClient(&#39;10.5.2.22&#39;, 11211) #source <br/> for i in range (1000): <br/> ... client.set(&quot;chisheng%d&quot;%(i), 0, 100000+i, &quot;hong&quot;) <br/> client1 = mc_bin_client.MemcachedClient(&#39;10.5.2.23&#39;, 11211) #destination <br/> client1.get(&quot;chisheng133&quot;) <br/> (629604608, 31, &#39;hong&#39;) <br/> client.get(&quot;chisheng133&quot;) <br/> (100133, 134, &#39;hong&#39;) <br/> <br/> <br/> Things doesn&#39;t work for backup from 181 and restore to 200 either: <br/> client = mc_bin_client.MemcachedClient(&#39;10.5.2.22&#39;, 11211)#source <br/> for i in range (1000): <br/> ... client.set(&quot;chisheng%d&quot;%(i), 0, 100000+i, &quot;hong&quot;) <br/> client1 = mc_bin_client.MemcachedClient(&#39;10.6.2.66&#39;, 11211)#destination <br/> client1.get(&quot;chisheng133&quot;) <br/> (629604608, 355638486346888, &#39;hong&#39;) <br/> client.get(&quot;chisheng133&quot;) <br/> (100133, 134, &#39;hong&#39;) <br/> <br/> <br/> But things are working when I do backup/restore from 200 to 200. (CAS, Flag, Value) <br/> client = mc_bin_client.MemcachedClient(&#39;10.5.2.22&#39;, 11211)#source <br/> for i in range (1000): <br/> ... client.set(&quot;chisheng%d&quot;%(i), 0, 100000+i, &quot;hong&quot;) <br/> client1 = mc_bin_client.MemcachedClient(&#39;10.5.2.23&#39;, 11211)#destination <br/> client.get(&quot;chisheng133&quot;) <br/> (100133, 13004603940813618, &#39;hong&#39;) <br/> client1.get(&quot;chisheng133&quot;) <br/> (100133, 13004603940813618, &#39;hong&#39;) centos5.6 MB-7148 cbbackup/cbrestore from 181 to latest 2.0 build, or 181 to 181, with wrong CAS and item_flag value restored. Bug Blocker Resolved Fixed Public Steve Yen Chisheng Hong Sat, 10 Nov 2012 01:30:16 -0600 Thu, 3 Jan 2013 12:21:29 -0600 Mon, 12 Nov 2012 14:53:02 -0600 2.0 1.8.2 2.0 tools 0 3 It is understandable backup/restore cas/flag fail from 181 to 200 since cas behavior in 181 and 200 is quite different, for some aspect, like default cas value for a new item. But it&#39;s abnormal that backup/restore from 181 to 181 doesn&#39;t work. <br/> <br/> &nbsp; Hi Chisheng, <br/> Can you add the cbbackup and cbrestore command-lines that you used? Did you use the same command-line for all these cases? <br/> <br/> 1.8.1-&gt;2.0.0 <br/> 1.8.1-&gt;1.8.1 <br/> 2.0.0-&gt;2.0.0 <br/> <br/> Related, if you used http:// or couchbase:// protocol for the restore, like... <br/> &nbsp;&nbsp;&nbsp;&nbsp;./cbrestore /backup/dir <a href="http://HOST:8091">http://HOST:8091</a> <br/> <br/> ... can you also try using memcached protocol (going through moxi) and also see if there&#39;s item-flag corruption? That is, please try... <br/> &nbsp;&nbsp;&nbsp;&nbsp;./cbrestore /backup/dir <a href="memcached://HOST:11211">memcached://HOST:11211</a> <br/> <br/> That can help narrow down the offending code paths. Please reassign back to me after that. <br/> <br/> Thanks, <br/> Steve btw, on the numbers from from Chisheng&#39;s description, you can see the flags value has been permuted... <br/> <br/> 629604608 == 0x25870100 <br/> <br/> 100133 == 0x00018725 <br/> This appears to be on the TAP side of things, where looking through the code there&#39;s a special TAP connection flag... <br/> <br/> &nbsp;&nbsp;management/memcacheConstants.py:TAP_FLAG_TAP_FIX_FLAG_BYTEORDER = 0x100 <br/> &nbsp;&nbsp;c/c++ version: TAP_CONNECT_TAP_FIX_FLAG_BYTEORDER <br/> <br/> This flag appears in couchbase 2.0 but not couchbase 1.8. Also, trying with a 2.0, I get the right flags when cbtransfer&#39;ing from TAP to stdout... <br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;$ ./cbtransfer <a href="http://10.3.121.192:8091">http://10.3.121.192:8091</a> stdout: | grep 100133 <br/> &nbsp;&nbsp;&nbsp;&nbsp;set abcd 100133 0 5 <br/> <br/> But with 1.8, I get the wrong flags when cbtransfer&#39;ing from TAP to stdout... <br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;$ ./cbtransfer <a href="http://10.3.121.194:8091">http://10.3.121.194:8091</a> stdout: | grep abcd <br/> &nbsp;&nbsp;&nbsp;&nbsp;set abcd 629604608 0 5 <br/> assigning back to me as I think this is diagnosed enough. <a href="http://review.couchbase.org/22428">http://review.couchbase.org/22428</a> <br/> Hi Steve, although it looks like you already spot where the problem is, I still want to provide the info about the test case you asked in your first comment. <br/> The command I use is through smart client: <br/> <br/> ./cbbackup <a href="http://10.5.2.22:8091">http://10.5.2.22:8091</a> ~/backup <br/> ./cbrestore ~/backup/ <a href="http://10.6.2.66:8091">http://10.6.2.66:8091</a> <br/> <br/> <br/> The node 10.5.2.22 (181 node) store all those 1K items as the source. 10.5.2.23 is also a 181 node I try to restore to (181 to 181). You can also backup all the items from the source node and restore to any 2.0 cluster (181 to 200). I have a 20 node 10.3.3.13 <br/> <br/> &nbsp;&nbsp; Same thing happened if I use moxi for restore to 2.0 from 1.8.1: <br/> <br/> ./cbrestore ~/backup/ <a href="memcached://10.6.2.66:11211">memcached://10.6.2.66:11211</a> <br/> <br/> client1 = mc_bin_client.MemcachedClient(&#39;10.6.2.66&#39;, 11211) # destination <br/> client1.get(&quot;chisheng133&quot;) <br/> (629604608, 505727160359962, &#39;hong&#39;) <br/> <br/> client.get(&quot;chisheng133&quot;) #source <br/> (100133, 134, &#39;hong&#39;) I git clone the membase-cli code and do backup restore at my own laptop. There will likely not be a 1.8.2, so we&#39;ll need to give this out as a hotfix to existing customers. <br/> Date of First Response Sat, 10 Nov 2012 08:42:46 -0600 Rank 3562 Time In Status [MB-4831] REST API for 'max_parallel_indexers' http://www.couchbase.com/issues/browse/MB-4831 Couchbase Server Based on the parameter &#39;max_parallel_indexers&#39; , indexer are forked off. <br/> <br/> The couchdb configuration system is at the moment not accessible to users by default, each setting needs an http rest handler in ns_server (and corresponding logic to propagate it to all other nodes). We want users to have the ability to increase the number of indexer via a REST API if needed depending on the machine they are using. <br/> <br/> We need to update ns_config with this parameter to ensure that it is replicated to all cluster nodes. <br/> <br/> MB-4831 REST API for 'max_parallel_indexers' Improvement Blocker Resolved Fixed Public Tommie McAfee Dipti Borkar Thu, 23 Feb 2012 19:19:28 -0600 Thu, 1 Mar 2012 14:19:05 -0600 Mon, 27 Feb 2012 12:41:07 -0600 2.0-developer-preview-3 2.0-developer-preview-4 ns_server 0 2 <a href="http://review.couchbase.org/13613">http://review.couchbase.org/13613</a> Integrated in github-ns-server-2-0 #314 (See [<a href="http://qa.hq.northscale.net/job/github-ns-server-2-0/314/">http://qa.hq.northscale.net/job/github-ns-server-2-0/314/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;implemented REST API for max_parallel_indexers setting.<a href="http://www.couchbase.com/issues/browse/MB-4831" title="REST API for &#39;max_parallel_indexers&#39;"><strike>MB-4831</strike></a> (Revision b0779cf284dafdd4c7b5a0b849ac858405d3be93)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> Aliaksey Artamonau : <br/> Files : <br/> * src/menelaus_web.erl<br/> merged Date of First Response Fri, 24 Feb 2012 21:59:33 -0600 Rank 5641 Time In Status [MB-4750] rebalancing two buckets under light write load is stuck at 50% ( possibly second bucket is not being scheduled for rebalance ) http://www.couchbase.com/issues/browse/MB-4750 Couchbase Server Ok, looking at the vbucekts per node, it seems that the &quot;couchbase&quot; bucket, which is no empty, not used, no load, is the one that got the rebalance stuck. <br/> &nbsp; <br/> This is a removal node that got stuck. <br/> There are no active vbuckets on the node I removed for the default-active bucket, which is running fine. <br/> <br/> Seems an ns_server issue. build 631 MB-4750 rebalancing two buckets under light write load is stuck at 50% ( possibly second bucket is not being scheduled for rebalance ) Bug Blocker Resolved Fixed Public Filipe Manana Farshid Ghods Thu, 2 Feb 2012 13:42:23 -0600 Wed, 3 Apr 2013 07:47:37 -0500 Wed, 3 Apr 2013 07:47:37 -0500 2.0-developer-preview-3 2.0-developer-preview-4 couchbase-bucket ns_server 0 2 possible duplicate of <a href="http://www.couchbase.com/issues/browse/MB-4752" title="remove one node and rebalance out the node gets stuck at 100% if clients are still waiting for the view results ( 10 index process still in progress )"><strike>MB-4752</strike></a> Part here is due to: <br/> <br/> [error_logger:error] [2012-02-02 19:00:42] [<a href='mailto:ns_1@10.176.150.73'>ns_1@10.176.150.73</a>:error_logger:ale_error_logger_handler:log_msg:76] ** Generic server &lt;0.8037.1&gt; terminating <br/> ** Last message in was {&#39;EXIT&#39;,&lt;0.8041.1&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{{compact_filter_error, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{error,badarith}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{part_id,dups}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{cbitmask,0}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{couch_set_view_compactor, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;-compact_group/5-fun-1-&#39;,2}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_btree_copy,fold_copy,3}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_btree,stream_kv_node2,8}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_btree,stream_kp_node,7}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_btree,fold,4}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_btree_copy,copy,3}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_set_view_compactor,compact_view,5}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{lists,mapfoldl,3}]}} <br/> ** When Server state == {file,&lt;0.8038.1&gt;,&lt;0.8039.1&gt;,1242101} <br/> ** Reason for termination == <br/> ** {{compact_filter_error,{error,badarith},{part_id,dups},{cbitmask,0}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;[{couch_set_view_compactor,&#39;-compact_group/5-fun-1-&#39;,2}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_btree_copy,fold_copy,3}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_btree,stream_kv_node2,8}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_btree,stream_kp_node,7}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_btree,fold,4}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_btree_copy,copy,3}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_set_view_compactor,compact_view,5}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{lists,mapfoldl,3}]} <br/> <br/> addressed by WIP commit here: <a href="http://review.couchbase.org/13067">http://review.couchbase.org/13067</a> <br/> <br/> Part is bad error handling. One case is fixed by this: <a href="http://review.couchbase.org/13072">http://review.couchbase.org/13072</a> <br/> <br/> Assigning to Filipe and confirming couch set nature of this. Duplicate of <a href="http://www.couchbase.com/issues/browse/MB-4774" title="compactor_died while loading data and querying view"><strike>MB-4774</strike></a> and <a href="http://www.couchbase.com/issues/browse/MB-4732" title=" Compaction seems to be stuck (or not running)"><strike>MB-4732</strike></a>.<br/> Already fixed. Date of First Response Mon, 6 Feb 2012 19:18:56 -0600 Rank 5716 Time In Status [MB-4710] view immediately returns ""reason": "conn_failed" if one node is down and not failed over yet. it should return partial results instead http://www.couchbase.com/issues/browse/MB-4710 Couchbase Server 10.114.167.27 was stopped using //etc/init.d/couchbase-server command <br/> <br/> view defintion : function (doc) { <br/> &nbsp;&nbsp;emit(doc._id, doc); <br/> } <br/> <br/> <br/> Bucket Name Nodes Item Count Ops/sec Disk Fetches/sec RAM Usage/Quota Disk Usage <br/> default 6 108002 1504 0 61.9MB / 12.9GB 123MB <br/> Views <br/> Documents <br/> <br/> <br/> <br/> Subset of nodes failed with the following error: <br/> [ <br/> { <br/> &quot;from&quot;: &quot;<a href="http://10.114.167.27:8092/_view_merge/?stale=false">http://10.114.167.27:8092/_view_merge/?stale=false</a>&quot;, <br/> &quot;reason&quot;: &quot;conn_failed&quot; <br/> } <br/> ] <br/> <br/> <a href="http://10.212.167.16:8092/default/_design/dev_hi/_view/hi?full_set=true&stale=false&connection_timeout=60000&limit=10&skip=0">http://10.212.167.16:8092/default/_design/dev_hi/_view/hi?full_set=true&amp;stale=false&amp;connection_timeout=60000&amp;limit=10&amp;skip=0</a> build 599 MB-4710 view immediately returns ""reason": "conn_failed" if one node is down and not failed over yet. it should return partial results instead Bug Blocker Resolved Fixed Public Filipe Manana Farshid Ghods Sat, 28 Jan 2012 20:46:18 -0600 Wed, 3 Apr 2013 07:47:42 -0500 Wed, 3 Apr 2013 07:47:42 -0500 2.0-developer-preview-3 2.0-developer-preview-4 ns_server view-engine 0 1 Filipe,<br/> <br/> im not sure if replica index is enabled or disabled in this build <br/> the error goes away right after i &quot;failover&quot; the node Farshid, can&#39;t reproduce. <br/> Created a 2 nodes cluster with 160 000 docs, reduce function is &quot;_count&quot;. <br/> <br/> 1) Queried the view, returned a row with value 160 000 (correct); <br/> <br/> 2) kill -9 node 2, node 1&#39;s UI tells me node 2 is not alive; <br/> <br/> 3) Without doing a failover, queried the view and got: <br/> <br/> fdmanana 14:55:30 ~ &gt; curl &#39;<a href="http://localhost:9500/default/_design/test/_view/view1?stale=false&#39;">http://localhost:9500/default/_design/test/_view/view1?stale=false&amp;#39;</a> <br/> {&quot;rows&quot;:[ <br/> {&quot;key&quot;:null,&quot;value&quot;:80000} <br/> ], <br/> &quot;errors&quot;:[ <br/> {&quot;from&quot;:&quot;<a href="http://192.168.1.80:9501/_view_merge/?stale=false">http://192.168.1.80:9501/_view_merge/?stale=false</a>&quot;,&quot;reason&quot;:&quot;conn_failed&quot;} <br/> ]} <br/> <br/> This is correct. <br/> I tested this with the latest build of everything (clean repo checkout of branch-2.0.xml). expected behavior Date of First Response Sun, 29 Jan 2012 08:59:12 -0600 Rank 5756 Time In Status [MB-4714] When mccouch stops responding ep-engine hangs forever. http://www.couchbase.com/issues/browse/MB-4714 Couchbase Server MB-4714 When mccouch stops responding ep-engine hangs forever. Bug Blocker Resolved Fixed Public Mike Wiederhold Mike Wiederhold Mon, 30 Jan 2012 00:34:26 -0600 Tue, 10 Apr 2012 21:03:21 -0500 Wed, 1 Feb 2012 19:52:56 -0600 2.0-developer-preview-3 2.0-developer-preview-4 couchbase-bucket 0 0 Integrated in github-ep-engine-2-0 #171 (See [<a href="http://qa.hq.northscale.net/job/github-ep-engine-2-0/171/">http://qa.hq.northscale.net/job/github-ep-engine-2-0/171/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.couchbase.com/issues/browse/MB-4714" title="When mccouch stops responding ep-engine hangs forever."><strike>MB-4714</strike></a>: Reset connection if mccouch stops responding (Revision 6d5e89f5763988d573a810e34b138389fe19066f)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> Mike Wiederhold : <br/> Files : <br/> * docs/engine-params.org<br/> * mc-kvstore/mc-engine.cc<br/> * configuration.json<br/> * ep_engine.cc<br/> Date of First Response Wed, 1 Feb 2012 19:00:53 -0600 Rank 5751 Time In Status [MB-4661] active/replica item count not 0 when all items are deleted. http://www.couchbase.com/issues/browse/MB-4661 Couchbase Server MB-4661 active/replica item count not 0 when all items are deleted. Bug Blocker Resolved Fixed Public Chiyoung Seo Mike Wiederhold Wed, 18 Jan 2012 13:24:20 -0600 Sun, 22 Jan 2012 19:03:28 -0600 Sun, 22 Jan 2012 19:03:28 -0600 2.0-developer-preview-3 2.0-developer-preview-4 0 0 <a href="http://review.couchbase.org/#change,12591">http://review.couchbase.org/#change,12591</a> Date of First Response Sun, 22 Jan 2012 19:03:28 -0600 Rank 5802 Time In Status [MB-4594] "Undefined set view `default` for `_design/dev_t` design document." on view after rebalance http://www.couchbase.com/issues/browse/MB-4594 Couchbase Server Minimized down to the following: <br/> <br/> 1) started 2 nodes development cluster <br/> 2) setup first node <br/> 3) upload 5k of 10k documents (via mc-loader from github.com/alk/mc-loader fork): cat tiny-keyset | ./mc-loader lh:12001 - valuesize 10000 (not with valuesize option mc-loader ignores values, it honors only keys from data set) <br/> 4) wait till write queue drops to 0 <br/> 5) upload 5k of small json docs (from tiny-keyset) : cat tinykey | ./mc-loader lh:12001 - (not valuesize _is not_ passed) <br/> 6) wait till write queue drops to 0 <br/> 7) create &#39;identity&#39; view via &#39;Create Development View&#39; button (in my case _design/dev_t with view name t) <br/> 8) run this view in full mode and observe that everything is ok (5k rows) <br/> <br/> 9) add second node and rebalance <br/> 10) wait till write queue drops to 0 <br/> <br/> 11) run same view and observe that only half items is there. Most of the time I was getting same half items silently, but this time I&#39;m also getting this: <br/> <br/> pi:~/src/altoros/moxi/repo20# wget -O- &#39;<a href="http://192.168.1.103:9500/default/_design/dev_t/_view/t?full_set=true&connection_timeout=2000000&limit=10&skip=0&#39;">http://192.168.1.103:9500/default/_design/dev_t/_view/t?full_set=true&amp;connection_timeout=2000000&amp;limit=10&amp;skip=0&amp;#39;</a> | json_reformat <br/> --2011-12-28 17:10:20-- <a href="http://192.168.1.103:9500/default/_design/dev_t/_view/t?full_set=true&connection_timeout=2000000&limit=10&skip=0">http://192.168.1.103:9500/default/_design/dev_t/_view/t?full_set=true&amp;connection_timeout=2000000&amp;limit=10&amp;skip=0</a> <br/> Connecting to 192.168.1.103:9500... connected. <br/> HTTP request sent, awaiting response... 200 OK <br/> Length: unspecified [text/plain] <br/> Saving to: `STDOUT&#39; <br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;[ &lt;=&gt; ] 642 --.-K/s in 0s <br/> <br/> 2011-12-28 17:10:20 (2.15 MB/s) - written to stdout [642] <br/> <br/> { <br/> &nbsp;&nbsp;&quot;total_rows&quot;: 2500, <br/> &nbsp;&nbsp;&quot;rows&quot;: [ <br/> &nbsp;&nbsp;&nbsp;&nbsp;{ <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;id&quot;: &quot;00000008&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;key&quot;: &quot;00000008&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;value&quot;: null <br/> &nbsp;&nbsp;&nbsp;&nbsp;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;{ <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;id&quot;: &quot;00000009&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;key&quot;: &quot;00000009&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;value&quot;: null <br/> &nbsp;&nbsp;&nbsp;&nbsp;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;{ <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;id&quot;: &quot;00000018&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;key&quot;: &quot;00000018&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;value&quot;: null <br/> &nbsp;&nbsp;&nbsp;&nbsp;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;{ <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;id&quot;: &quot;00000019&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;key&quot;: &quot;00000019&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;value&quot;: null <br/> &nbsp;&nbsp;&nbsp;&nbsp;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;{ <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;id&quot;: &quot;00000028&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;key&quot;: &quot;00000028&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;value&quot;: null <br/> &nbsp;&nbsp;&nbsp;&nbsp;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;{ <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;id&quot;: &quot;00000029&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;key&quot;: &quot;00000029&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;value&quot;: null <br/> &nbsp;&nbsp;&nbsp;&nbsp;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;{ <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;id&quot;: &quot;00000038&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;key&quot;: &quot;00000038&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;value&quot;: null <br/> &nbsp;&nbsp;&nbsp;&nbsp;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;{ <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;id&quot;: &quot;00000039&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;key&quot;: &quot;00000039&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;value&quot;: null <br/> &nbsp;&nbsp;&nbsp;&nbsp;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;{ <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;id&quot;: &quot;00000048&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;key&quot;: &quot;00000048&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;value&quot;: null <br/> &nbsp;&nbsp;&nbsp;&nbsp;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;{ <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;id&quot;: &quot;00000049&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;key&quot;: &quot;00000049&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;value&quot;: null <br/> &nbsp;&nbsp;&nbsp;&nbsp;} <br/> &nbsp;&nbsp;], <br/> &nbsp;&nbsp;&quot;errors&quot;: [ <br/> &nbsp;&nbsp;&nbsp;&nbsp;{ <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;from&quot;: &quot;local&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;reason&quot;: &quot;Undefined set view `default` for `_design/dev_t` design document.&quot; <br/> &nbsp;&nbsp;&nbsp;&nbsp;} <br/> &nbsp;&nbsp;] <br/> } <br/> pi:~/src/altoros/moxi/repo20# wget -O- &#39;<a href="http://192.168.1.103:9501/default/_design/dev_t/_view/t?full_set=true&connection_timeout=2000000&limit=10&skip=0&#39;">http://192.168.1.103:9501/default/_design/dev_t/_view/t?full_set=true&amp;connection_timeout=2000000&amp;limit=10&amp;skip=0&amp;#39;</a> | json_reformat <br/> --2011-12-28 17:10:34-- <a href="http://192.168.1.103:9501/default/_design/dev_t/_view/t?full_set=true&connection_timeout=2000000&limit=10&skip=0">http://192.168.1.103:9501/default/_design/dev_t/_view/t?full_set=true&amp;connection_timeout=2000000&amp;limit=10&amp;skip=0</a> <br/> Connecting to 192.168.1.103:9501... connected. <br/> HTTP request sent, awaiting response... 200 OK <br/> Length: unspecified [text/plain] <br/> Saving to: `STDOUT&#39; <br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;[ &lt;=&gt; ] 675 --.-K/s in 0.001s <br/> <br/> 2011-12-28 17:10:34 (623 KB/s) - written to stdout [675] <br/> <br/> { <br/> &nbsp;&nbsp;&quot;total_rows&quot;: 2500, <br/> &nbsp;&nbsp;&quot;rows&quot;: [ <br/> &nbsp;&nbsp;&nbsp;&nbsp;{ <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;id&quot;: &quot;00000008&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;key&quot;: &quot;00000008&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;value&quot;: null <br/> &nbsp;&nbsp;&nbsp;&nbsp;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;{ <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;id&quot;: &quot;00000009&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;key&quot;: &quot;00000009&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;value&quot;: null <br/> &nbsp;&nbsp;&nbsp;&nbsp;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;{ <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;id&quot;: &quot;00000018&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;key&quot;: &quot;00000018&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;value&quot;: null <br/> &nbsp;&nbsp;&nbsp;&nbsp;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;{ <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;id&quot;: &quot;00000019&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;key&quot;: &quot;00000019&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;value&quot;: null <br/> &nbsp;&nbsp;&nbsp;&nbsp;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;{ <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;id&quot;: &quot;00000028&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;key&quot;: &quot;00000028&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;value&quot;: null <br/> &nbsp;&nbsp;&nbsp;&nbsp;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;{ <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;id&quot;: &quot;00000029&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;key&quot;: &quot;00000029&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;value&quot;: null <br/> &nbsp;&nbsp;&nbsp;&nbsp;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;{ <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;id&quot;: &quot;00000038&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;key&quot;: &quot;00000038&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;value&quot;: null <br/> &nbsp;&nbsp;&nbsp;&nbsp;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;{ <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;id&quot;: &quot;00000039&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;key&quot;: &quot;00000039&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;value&quot;: null <br/> &nbsp;&nbsp;&nbsp;&nbsp;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;{ <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;id&quot;: &quot;00000048&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;key&quot;: &quot;00000048&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;value&quot;: null <br/> &nbsp;&nbsp;&nbsp;&nbsp;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;{ <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;id&quot;: &quot;00000049&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;key&quot;: &quot;00000049&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;value&quot;: null <br/> &nbsp;&nbsp;&nbsp;&nbsp;} <br/> &nbsp;&nbsp;], <br/> &nbsp;&nbsp;&quot;errors&quot;: [ <br/> &nbsp;&nbsp;&nbsp;&nbsp;{ <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;from&quot;: &quot;<a href="http://192.168.1.103:9500/_view_merge/">http://192.168.1.103:9500/_view_merge/</a>&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;reason&quot;: &quot;Undefined set view `default` for `_design/dev_t` design document.&quot; <br/> &nbsp;&nbsp;&nbsp;&nbsp;} <br/> &nbsp;&nbsp;] <br/> } <br/> <br/> <br/> Note that somehow first uploading 10k 0-s of crap and then rewriting it with json is seemingly required for reproduction. <br/> <br/> Will attach diags from both nodes and tiny keyset <br/> Development environment MB-4594 "Undefined set view `default` for `_design/dev_t` design document." on view after rebalance Bug Blocker Resolved Fixed Public Aliaksey Artamonau Aleksey Kondratenko Wed, 28 Dec 2011 08:23:42 -0600 Mon, 23 Jul 2012 16:03:48 -0500 Tue, 24 Jan 2012 13:41:55 -0600 2.0-developer-preview-3 2.0-developer-preview-4 ns_server 0 0 Hi Aliaksey A, <br/> Any news/status on this one? <br/> Thanks, <br/> Steve Diags seem to be from different clusters. The only error I&#39;m seeing in the first diag is fixed by <a href="http://review.couchbase.org/12091">http://review.couchbase.org/12091</a>. I&#39;m closing it as fixed since I&#39;m not able to reproduce it. Date of First Response Tue, 24 Jan 2012 11:00:21 -0600 Rank 5861 Time In Status [MB-4555] Document editing UI must be able to handle (random) documents from a node that is down http://www.couchbase.com/issues/browse/MB-4555 Couchbase Server Currently, the doc editing UI cannot handle the case when a randomly selected document for display lives on a node that is down. <br/> The UI must be able to withstand node failure. MB-4555 Document editing UI must be able to handle (random) documents from a node that is down Bug Blocker Resolved Fixed Public Dale Harvey Dipti Borkar Mon, 19 Dec 2011 13:38:22 -0600 Tue, 24 Jan 2012 15:36:17 -0600 Tue, 24 Jan 2012 15:36:17 -0600 2.0-developer-preview-3 2.0-developer-preview-4 0 1 note there&#39;s also similar issue during warmup when all vbuckets are dead. <br/> fix is in review <a href="http://review.couchbase.org/#change,12403">http://review.couchbase.org/#change,12403</a> Hi Dale, ping on the status on this one? Hey, Fixed all the outstanding issues on the commit, but I am guessing alks been too busy to review, happy for you to review and I will double check everything / deal with any regressions Hi Dale, <br/> I don&#39;t know that code very well, but nothing in your change offended me, so approved. <br/> Please push the ball into the goal. <br/> Thanks! Date of First Response Fri, 6 Jan 2012 07:15:25 -0600 Rank 5905 Time In Status [MB-4807] Set view group craches with reason {noproc, {gen_server,call, [<0.4370.0>,{drop,<0.4362.0>},infinity]}} http://www.couchbase.com/issues/browse/MB-4807 Couchbase Server I was running a test against 4 node cluster (set up by cluster_run): <br/> <br/> python2 ./testrunner -i b/resources/dev-4-nodes.ini -t viewtests.ViewBasicTests.test_load_data_get_view_x_mins_y_design_docs,num-design-docs=25,load-time=10,run-view-time=2,num-tries=100 <br/> <br/> After some time set view group crashed with this error: <br/> <br/> =========================CRASH REPORT========================= <br/> &nbsp;&nbsp;crasher: <br/> &nbsp;&nbsp;&nbsp;&nbsp;initial call: couch_set_view_group:init/1 <br/> &nbsp;&nbsp;&nbsp;&nbsp;pid: &lt;0.4362.0&gt; <br/> &nbsp;&nbsp;&nbsp;&nbsp;registered_name: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;exception exit: {noproc, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server,call, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&lt;0.4370.0&gt;,{drop,&lt;0.4362.0&gt;},infinity]}} <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in function couch_set_view_group:init/1 <br/> &nbsp;&nbsp;&nbsp;&nbsp;ancestors: [&lt;0.3972.0&gt;,&lt;0.3961.0&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;messages: [{&#39;EXIT&#39;,&lt;0.4371.0&gt;,normal}] <br/> &nbsp;&nbsp;&nbsp;&nbsp;links: [&lt;0.3972.0&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;dictionary: [{random_seed,{26506,11927,3989}}] <br/> &nbsp;&nbsp;&nbsp;&nbsp;trap_exit: true <br/> &nbsp;&nbsp;&nbsp;&nbsp;status: running <br/> &nbsp;&nbsp;&nbsp;&nbsp;heap_size: 10946 <br/> &nbsp;&nbsp;&nbsp;&nbsp;stack_size: 24 <br/> &nbsp;&nbsp;&nbsp;&nbsp;reductions: 116161 <br/> &nbsp;&nbsp;neighbours: <br/> <br/> Seems to be similar to <a href="http://www.couchbase.com/issues/browse/MB-4686" title="View returns partial results"><strike>MB-4686</strike></a>. Log is attached. MB-4807 Set view group craches with reason {noproc, {gen_server,call, [<0.4370.0>,{drop,<0.4362.0>},infinity]}} Bug Blocker Resolved Fixed Public Filipe Manana Aliaksey Artamonau Wed, 15 Feb 2012 17:19:27 -0600 Wed, 22 Feb 2012 13:20:18 -0600 Wed, 22 Feb 2012 13:20:18 -0600 2.0-developer-preview-4 2.0-developer-preview-4 0 1 I believe this bug is as easy at catching and ignoring any error in couchdb close, as what is happening is the storage file is being deleted as it is rebalanced to another node. What I&#39;m unsure about is if there are places where we might still be using the db (not just closing it) when this happens. If so, we might need to rethink how file deletions occurs. I believe that the cluster was in steady state when this happened (i.e. no recent rebalance). I didn&#39;t save all the log files. But I will double check by the log files attached to <a href="http://www.couchbase.com/issues/browse/MB-4809" title="couchbase_compaction_daemon failed:"><strike>MB-4809</strike></a>. It seems that <a href="http://www.couchbase.com/issues/browse/MB-4809" title="couchbase_compaction_daemon failed:"><strike>MB-4809</strike></a> is a bit different. There it really happened right after rebalance. Here last rebalance happened 30 minutes before these errors. Stupid of me not to save all the diags. Fixed with several changes merged this week. Date of First Response Wed, 15 Feb 2012 21:43:10 -0600 Rank 5664 Time In Status [MB-4811] View query returns {"error":"unknown_error","reason":"killed"} http://www.couchbase.com/issues/browse/MB-4811 Couchbase Server test:- <br/> test_load_data_get_view_x_mins_y_design_docs,num-design-docs=25,load-time=10,run-view-time=2,num-tries=100 <br/> <br/> 25 design docs, continuous mutation, parallel query on 25 design docs. <br/> <br/> All Design Docs have the following map function:- <br/> <br/> {&quot;_id&quot;:&quot;_design/dev_test_view_on_10000_docs-1f75728&quot;,&quot;_rev&quot;:&quot;1-b9cf93a0&quot;,&quot;language&quot;:&quot;javascript&quot;,&quot;views&quot;:{&quot;dev_test_view_on_10000_docs-1f75728&quot;:{&quot;map&quot;:&quot;function (doc) {if(doc.name.indexOf(\&quot;dev_test_view_on_10000_docs-1f75728\&quot;) != -1) { emit(doc.name, doc);}}&quot;}}} <br/> <br/> <br/> Query:- <br/> <a href="http://10.1.2.94:8091/couchBase/default/_design/dev_test_view_on_10000_docs-07fc776/_view/dev_test_view_on_10000_docs-07fc776?limit=0&connection_timeout=60000&full_set=true">http://10.1.2.94:8091/couchBase/default/_design/dev_test_view_on_10000_docs-07fc776/_view/dev_test_view_on_10000_docs-07fc776?limit=0&amp;connection_timeout=60000&amp;full_set=true</a> <br/> <br/> Log: <br/> <a href="http://www.couchbase.com/issues/secure/attachment/12308/10.1.2.94-8091-diag.txt.zip">http://www.couchbase.com/issues/secure/attachment/12308/10.1.2.94-8091-diag.txt.zip</a> <br/> MB-4811 View query returns {"error":"unknown_error","reason":"killed"} Bug Blocker Resolved Fixed Public Aliaksey Artamonau Aliaksey Artamonau Wed, 15 Feb 2012 20:20:42 -0600 Mon, 23 Jul 2012 14:53:47 -0500 Thu, 16 Feb 2012 15:19:56 -0600 2.0-developer-preview-4 2.0-developer-preview-4 ns_server 0 0 Integrated in github-couchdb-preview #346 (See [<a href="http://qa.hq.northscale.net/job/github-couchdb-preview/346/">http://qa.hq.northscale.net/job/github-couchdb-preview/346/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.couchbase.com/issues/browse/MB-4811" title="View query returns {&quot;error&quot;:&quot;unknown_error&quot;,&quot;reason&quot;:&quot;killed&quot;}"><strike>MB-4811</strike></a> Unlink all the folders before killing any of them. (Revision c90bdcc6996fa9905ae7bf2673714b15fdcb237a)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> Filipe David Borba Manana : <br/> Files : <br/> * src/couchdb/couch_index_merger.erl<br/> Date of First Response Thu, 16 Feb 2012 07:35:51 -0600 Rank 5660 Time In Status [MB-4805] ep-engine says warmup is done too early http://www.couchbase.com/issues/browse/MB-4805 Couchbase Server * Load a bunch of stuff to single node cluster <br/> * kill memcached <br/> * observe how ns_server will quite quickly mark this node as green and double check with stats that warmup is done <br/> * but try to see if loaded data is there and observe that it&#39;s not. Also it can be seen that item count is growing <br/> <br/> Note this is really bad because janitor will activate vbuckets that are being loaded and start replication from it. Thats very scary. It&#39;s quite likely path that caused data loss I&#39;ve observed in 2.0. <br/> MB-4805 ep-engine says warmup is done too early Bug Blocker Resolved Fixed Public Trond Norbye Aleksey Kondratenko Tue, 14 Feb 2012 19:16:12 -0600 Tue, 10 Apr 2012 21:03:22 -0500 Fri, 17 Feb 2012 08:36:42 -0600 2.0-developer-preview-4 2.0-developer-preview-4 couchbase-bucket 0 1 ep-engine reports that warmup is complete (to allow external traffic) when all metadata for items are loaded into memory. At this time it is safe to run all operations (get / set / add / remove etc), because we know if the data is there or not. The &quot;item count&quot; represented by curr_items is incremented by the second phase when we&#39;re actually loading the body for the data (we can always discuss if that&#39;s the wrong thing to do or if it should be reported as we load the meta-data). <br/> <br/> There is an ongoing task to rewrite the warmup phase by using a smarter way of determine the actual items we want to have loaded. <br/> <br/> Do you have a concrete bug I may try to find here? or is this just a suspicion that this may lead to a data loss? the observed behavior is that items that should be there are not there according to GET results. So there is some obvious bug. So you have called a get request and gotten a NOT_FOUND response (and not a tmpfail/busy) back for an item that you are 100% was there (and persisted before you killed the system)? Hm. I don&#39;t know if it was tmpfail or not_found. Will double check. Item was definitely persisted. <br/> I&#39;m going to write a test case to see if I can reproduce it :) <a href="http://review.couchbase.org/#change,13331">http://review.couchbase.org/#change,13331</a> Integrated in github-ep-engine-2-0 #196 (See [<a href="http://qa.hq.northscale.net/job/github-ep-engine-2-0/196/">http://qa.hq.northscale.net/job/github-ep-engine-2-0/196/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.couchbase.com/issues/browse/MB-4805" title="ep-engine says warmup is done too early"><strike>MB-4805</strike></a> Fix items lost during startup (Revision 3dc3329a3a97c71a592a45b983c00e80520c3309)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> Trond Norbye : <br/> Files : <br/> * ep.cc<br/> Integrated in github-ep-engine-2-0 #201 (See [<a href="http://qa.hq.northscale.net/job/github-ep-engine-2-0/201/">http://qa.hq.northscale.net/job/github-ep-engine-2-0/201/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;Backport: <a href="http://www.couchbase.com/issues/browse/MB-4805" title="ep-engine says warmup is done too early"><strike>MB-4805</strike></a> Fix items lost during startup (Revision 81f4e978a023b8b1c3c0e14d6e100db8679ebfe2)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> Trond Norbye : <br/> Files : <br/> * ep.cc<br/> Date of First Response Wed, 15 Feb 2012 08:09:33 -0600 Rank 5667 Time In Status [MB-4760] Node is in unusual state after failed rebalance http://www.couchbase.com/issues/browse/MB-4760 Couchbase Server While loading data, and querying the view, an attempt to rebalance out 2 nodes fails. It is now impossible to add the node back to the cluster as the rest-api throws: <br/> <br/> curl -u Administrator:password <a href="http://10.1.2.108:8091/nodes/self">http://10.1.2.108:8091/nodes/self</a> <br/> &quot;Node is unknown to this cluster.&quot; <br/> <br/> If I go to the UI, the node shows 1 cluster that isn&#39;t itself (10.1.2.105), even though I expected it to be reinitialized. <br/> <br/> I will leave this the state that it&#39;s in for the next hour or so if needed: <br/> <br/> 2012-02-03 06:29:54.360 menelaus_web:19:warning:server error during request processing(<a href='mailto:ns_1@10.1.2.104'>ns_1@10.1.2.104</a>) - Server error during processing: [&quot;web request failed&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{path,&quot;//pools/default/buckets/default&quot;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{type,error}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{what,{case_clause,rebalance_running}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{trace, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{menelaus_web_buckets, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;handle_bucket_delete,3}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{menelaus_web,loop,3}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{mochiweb_http,headers,5}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{proc_lib,init_p_do_apply,3}]}] <br/> 2012-02-03 06:29:54.431 ns_orchestrator:2:info:message(<a href='mailto:ns_1@10.1.2.104'>ns_1@10.1.2.104</a>) - Rebalance exited with reason stopped <br/> build 639, .deb <br/> 6 node cluster MB-4760 Node is in unusual state after failed rebalance Bug Blocker Resolved Fixed Public Filipe Manana Tommie McAfee Fri, 3 Feb 2012 16:33:08 -0600 Wed, 3 Apr 2013 07:47:40 -0500 Wed, 3 Apr 2013 07:47:40 -0500 2.0-developer-preview-4 2.0-developer-preview-4 bucket-engine 0 0 regression and this would break our automated tests which depend on cleanup following process links in diag looks like deadlock between somewhere in set_view code and couch_server {delete,...} stuff. <br/> <br/> Will work around by deleting /master database (with design docs) first. and that case_clause error is a separate issue thats going to be addressed soon workaround is here: <a href="http://review.couchbase.org/13017">http://review.couchbase.org/13017</a> <br/> <br/> Assigning to Filipe because deadlock is deadlock. <br/> <br/> Filipe, easiest way to see it is looking for processes executing do_call function in list of processes. And observing their backtraces and links. also happens when running the test suite against cluster_run<br/> <br/> to repro run this target and observe the cluster during the run. at some point few nodes go into this state.<br/> <br/> scripts/start_cluster_and_run_tests.sh b/resources/dev-4-nodes.ini conf/py-view.conf <a href="http://review.couchbase.org/13032">http://review.couchbase.org/13032</a> is _not_ fix. <br/> <br/> Here&#39;s what I&#39;m seeing in logs: <br/> <br/> <br/> [couchdb:info] [2012-02-03 7:06:56] [<a href='mailto:ns_1@10.1.2.104'>ns_1@10.1.2.104</a>:&lt;0.18429.2&gt;:couch_log:info:39] Set view `default`, main group `_design/dev_test_view-a4ed105`, terminating with reason: {cleaner_died, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{badarith, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{couch_set_view_util, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;btree_purge_fun, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_btree, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;kv_guided_purge, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;5}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_btree, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;guided_purge, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_btree, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;kp_guided_purge, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;5}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_btree, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;guided_purge, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_btree, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;kp_guided_purge, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;5}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_btree, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;guided_purge, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_btree, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;guided_purge, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;3}]}} <br/> [couchdb:info] [2012-02-03 7:06:56] [<a href='mailto:ns_1@10.1.2.104'>ns_1@10.1.2.104</a>:&lt;0.18429.2&gt;:couch_log:info:39] Stopping cleanup process for set view `default`, group `_design/dev_test_view-a4ed105` <br/> [error_logger:error] [2012-02-03 7:06:56] [<a href='mailto:ns_1@10.1.2.104'>ns_1@10.1.2.104</a>:error_logger:ale_error_logger_handler:log_msg:76] Error in process &lt;0.20053.2&gt; on node &#39;<a href='mailto:ns_1@10.1.2.104'>ns_1@10.1.2.104</a>&#39; with exit value: {badarith,[{couch_set_view_util,btree_purge_fun,4},{couch_btree,kv_guided_purge,5},{couch_btree,guided_purge,4},{couch_btree,kp_guided_purge,5},{couch_btree,guided_purge,4},{couch_btree,kp_guided_purge,5},{couch_btree... <br/> <br/> <br/> And then we have main group &#39;waiting&#39; cleaner dead in terminate: <br/> <br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&lt;0.18429.2&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{registered_name,[]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{status,waiting}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{initial_call,{proc_lib,init_p,5}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{backtrace, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&lt;&lt;&quot;Program counter: 0x00007f32801145c0 (couch_set_view_group:stop_cleaner/1 + 624)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;CP: 0x0000000000000000 (invalid)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;arity = 0&quot;&gt;&gt;,&lt;&lt;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;0x00007f32809de7d0 Return addr 0x00007f328010c760 (couch_set_view_group:terminate/2 + 736)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(0) []&quot;&gt;&gt;,&lt;&lt;&quot;y(1) []&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(2) []&quot;&gt;&gt;,&lt;&lt;&quot;y(3) []&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;(4) {state,{\&quot;/opt/couchbase/var/lib/couchdb\&quot;,&lt;&lt;7 bytes&gt;&gt;,{set_view_group,&lt;&lt;16 bytes&gt;&gt;,&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;(5) {set_view_group,&lt;&lt;16 bytes&gt;&gt;,&lt;0.18433.2&gt;,&lt;&lt;7 bytes&gt;&gt;,&lt;&lt;29 bytes&gt;&gt;,&lt;&lt;10 bytes&gt;&gt;,[],&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;(6) {\&quot;/opt/couchbase/var/lib/couchdb\&quot;,&lt;&lt;7 bytes&gt;&gt;,{set_view_group,&lt;&lt;16 bytes&gt;&gt;,nil,&lt;&lt;7&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(7) {set_view_group_stats,1,0,10,0,2,0,[{[{&lt;&lt;8 bytes&gt;&gt;,3.250037e+01}]}],[],[]}&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(8) &lt;0.20053.2&gt;&quot;&gt;&gt;,&lt;&lt;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;0x00007f32809de820 Return addr 0x00007f328a107788 (gen_server:terminate/6 + 184)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(0) []&quot;&gt;&gt;,&lt;&lt;&quot;y(1) []&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;(2) {cleaner_died,{badarith,[{couch_set_view_util,btree_purge_fun,4},{couch_btree,kv_g&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(3) nil&quot;&gt;&gt;,&lt;&lt;&quot;y(4) nil&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;(5) {set_view_group,&lt;&lt;16 bytes&gt;&gt;,&lt;0.18433.2&gt;,&lt;&lt;7 bytes&gt;&gt;,&lt;&lt;29 bytes&gt;&gt;,&lt;&lt;10 bytes&gt;&gt;,[],&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;0x00007f32809de858 Return addr 0x00007f3280102440 (couch_set_view_group:init/1 + 488)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(0) []&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;(1) {state,{\&quot;/opt/couchbase/var/lib/couchdb\&quot;,&lt;&lt;7 bytes&gt;&gt;,{set_view_group,&lt;&lt;16 bytes&gt;&gt;,&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(2) couch_set_view_group&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;(3) {&#39;EXIT&#39;,&lt;0.20053.2&gt;,{badarith,[{couch_set_view_util,btree_purge_fun,4},{couch_btre&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(4) &lt;0.18429.2&gt;&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;(5) {cleaner_died,{badarith,[{couch_set_view_util,btree_purge_fun,4},{couch_btree,kv_g&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(6) Catch 0x00007f328a107788 (gen_server:terminate/6 + 184)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;0x00007f32809de898 Return addr 0x00007f328a088fe8 (proc_lib:init_p_do_apply/3 + 56)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(0) Catch 0x00007f3280102460 (couch_set_view_group:init/1 + 520)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(1) []&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;(2) {set_view_group,&lt;&lt;16 bytes&gt;&gt;,nil,&lt;&lt;7 bytes&gt;&gt;,&lt;&lt;29 bytes&gt;&gt;,&lt;&lt;10 bytes&gt;&gt;,[],[{set_vi&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;0x00007f32809de8b8 Return addr 0x00000000008a00b8 (&lt;terminate process normally&gt;)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(0) Catch 0x00007f328a089008 (proc_lib:init_p_do_apply/3 + 88)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&gt;&gt;]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{error_handler,error_handler}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{garbage_collection, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{min_bin_vheap_size,46368}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{min_heap_size,233}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{fullsweep_after,0}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{minor_gcs,0}]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{heap_size,1597}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{total_heap_size,1597}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{links,[&lt;0.18433.2&gt;,&lt;0.195.0&gt;]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{memory,13960}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{message_queue_len,2}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{reductions,20570}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{trap_exit,true}]}, <br/> Yes, but <a href="http://review.couchbase.org/13032">http://review.couchbase.org/13032</a> is based on the observation from the diag logs in this ticket. <br/> The stack trace you just pasted is another, different issue. For this last stack trace, it&#39;s the same issue as in <a href="http://www.couchbase.com/issues/browse/MB-4774" title="compactor_died while loading data and querying view"><strike>MB-4774</strike></a>. <br/> <a href="http://review.couchbase.org/#change,13067">http://review.couchbase.org/#change,13067</a> Integrated in github-couchdb-preview #333 (See [<a href="http://qa.hq.northscale.net/job/github-couchdb-preview/333/">http://qa.hq.northscale.net/job/github-couchdb-preview/333/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;don&#39;t wait for EXIT from cleaner when it&#39;s dead.<a href="http://www.couchbase.com/issues/browse/MB-4760" title="Node is in unusual state after failed rebalance"><strike>MB-4760</strike></a> (Revision 08e7f872750c5e3ff2104022e97fc8be4859a5b5)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> Filipe David Borba Manana : <br/> Files : <br/> * src/couch_set_view/src/couch_set_view_group.erl<br/> Both Alk&#39;s change and mine were merged. <br/> Closing this. <br/> <br/> <a href="https://github.com/couchbase/couchdb/commit/6319846fa68c73580e5ead96dbe27868447f730f">https://github.com/couchbase/couchdb/commit/6319846fa68c73580e5ead96dbe27868447f730f</a> <br/> <a href="https://github.com/couchbase/couchdb/commit/08e7f872750c5e3ff2104022e97fc8be4859a5b5">https://github.com/couchbase/couchdb/commit/08e7f872750c5e3ff2104022e97fc8be4859a5b5</a> Date of First Response Fri, 3 Feb 2012 18:25:38 -0600 Rank 5708 Time In Status [MB-4743] Totally messed up data files after night of constant data updating with compaction running http://www.couchbase.com/issues/browse/MB-4743 Couchbase Server To verify that couchdb compaction cannot keep up with mutations I&#39;ve run simple singe node test case for night. Same 200k documents of 100 bytes each were constantly SET through memcached. Autocompaction threshold was lowered to 2%. So mccouch was constantly writing stuff and compacting. <br/> <br/> On morning I&#39;ve found data files size to be huge and constant stream of error messages somewhere from couch. I stopped server. <br/> <br/> Now each time I start it starts that constant couch crashes and I cannot get my data back. <br/> <br/> Going to attach diag (search from the bottom for: <br/> <br/> [ns_server:info] [2012-02-01 13:15:14] [<a href='mailto:nonode@nohost'>nonode@nohost</a>:ns_server_cluster_sup:log_os_info:start_link:27] Manifest: <br/> <br/> to see events since last start attempt). I have data files archived as well and I&#39;m ready to provide them on first request. <br/> MB-4743 Totally messed up data files after night of constant data updating with compaction running Bug Blocker Resolved Fixed Public Damien Katz Aleksey Kondratenko Wed, 1 Feb 2012 15:56:49 -0600 Mon, 6 Feb 2012 19:20:21 -0600 Mon, 6 Feb 2012 19:20:21 -0600 2.0-developer-preview-4 2.0-developer-preview-4 view-engine 0 0 Based on what I&#39;m seeing in the logs, this is caused by a failure to rename the compaction file (foo.compact) after we&#39;ve deleted the old storage file. Possibly because of an out of disk error. Then on opening the database on restart, we get a bad match error when we expect there to be a properly named storage file, but only the compact file exists, causing a bad match. <br/> <br/> Patch here that doesn&#39;t delete old storage file until new compacted storage file is properly renamed, and will delete the .compact on open if found: <br/> <a href="http://review.couchbase.org/#change,12969">http://review.couchbase.org/#change,12969</a> <br/> <br/> Alk, your directory files should now open, but it will delete the .compact file, losing a vbuckets db. If you wish to preserve it, rename it from foo.couch.1.compact to foo.2.couch, then start the couchdb server. With the new fix, in the future it should be impossible to have a compact file without a storage file, assuming the FS hasn&#39;t corrupted, so no data loss should be possible. Date of First Response Fri, 3 Feb 2012 02:25:37 -0600 Rank 5718 Time In Status [MB-4735] Data loss during rebalance after service restart http://www.couchbase.com/issues/browse/MB-4735 Couchbase Server * start development cluster of two nodes <br/> * set up first node with default bucket <br/> * join second node but don&#39;t rebalance yet <br/> * load some data <br/> * wait till it hits disk <br/> * Ctrl-C and start this cluster back <br/> * wait till warmup is done and observe that data is there <br/> * run rebalance <br/> * observe that after rebalance new node has _zero_ items. Those are lost. <br/> <br/> It&#39;s not happening if we don&#39;t restart. And same happens if I just kill -9 memcached instead of restarting everything. <br/> MB-4735 Data loss during rebalance after service restart Bug Blocker Resolved Duplicate Public Chiyoung Seo Aleksey Kondratenko Tue, 31 Jan 2012 20:10:32 -0600 Tue, 10 Apr 2012 21:03:22 -0500 Fri, 3 Feb 2012 19:29:29 -0600 2.0-developer-preview-4 2.0-developer-preview-4 couchbase-bucket 0 1 Chiyoung, this showed up at the recent sprint demo... Can you analyze? this is blocking warmup performance testing as well just to make it a bit more clear. The data is there after restart. It&#39;s just somehow not there after rebalance. <br/> ok, i&#39;ll verify if i see this in my testing if i do a full service restart instead of just restarting memcached. if the data warms up properly with a service restart then i won&#39;t be blocked. Chiyoung, any update on this? Thx This issue is a duplicate of <a href="http://www.couchbase.com/issues/browse/MB-4595">http://www.couchbase.com/issues/browse/MB-4595</a>.<br/> <br/> It&#39;s a edge case that can only happen when all the nodes have the open checkpoint with id 1 and are restarted. I don&#39;t think this is a blocker, but will fix it for DP4 and 1.8.1. Date of First Response Thu, 2 Feb 2012 18:56:41 -0600 Rank 5729 Time In Status [MB-4704] couch_set_view_group dies with {badmatch,{error,enoent}} reason http://www.couchbase.com/issues/browse/MB-4704 Couchbase Server From time to time couch_set_view_group dies with error: <br/> <br/> =========================CRASH REPORT========================= <br/> &nbsp;&nbsp;crasher: <br/> &nbsp;&nbsp;&nbsp;&nbsp;initial call: couch_set_view_group:init/1 <br/> &nbsp;&nbsp;&nbsp;&nbsp;pid: &lt;0.2771.0&gt; <br/> &nbsp;&nbsp;&nbsp;&nbsp;registered_name: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;exception exit: {{badmatch,{error,enoent}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{couch_db,fast_reads,2}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_set_view_updater,&#39;-load_changes/6-fun-2-&#39;,6}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{lists,foldl,3}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_set_view_updater,load_changes,6}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_set_view_updater,update,6}]} <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in function couch_set_view_group:init/1 <br/> &nbsp;&nbsp;&nbsp;&nbsp;ancestors: [&lt;0.2760.0&gt;,&lt;0.2759.0&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;messages: [{updater_info,&lt;0.7668.0&gt;,{state,updating_passive}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&#39;EXIT&#39;,&lt;0.2780.0&gt;,normal}] <br/> &nbsp;&nbsp;&nbsp;&nbsp;links: [&lt;0.2760.0&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;dictionary: [{random_seed,{23671,3853,29824}}] <br/> &nbsp;&nbsp;&nbsp;&nbsp;trap_exit: true <br/> &nbsp;&nbsp;&nbsp;&nbsp;status: running <br/> &nbsp;&nbsp;&nbsp;&nbsp;heap_size: 6765 <br/> &nbsp;&nbsp;&nbsp;&nbsp;stack_size: 24 <br/> &nbsp;&nbsp;&nbsp;&nbsp;reductions: 128385 <br/> &nbsp;&nbsp;neighbours: <br/> <br/> When this happens views return partial results with connection_closed error for the failed node. Attached the log for such a failure. MB-4704 couch_set_view_group dies with {badmatch,{error,enoent}} reason Bug Blocker Resolved Fixed Public Damien Katz Aliaksey Artamonau Fri, 27 Jan 2012 17:46:54 -0600 Sat, 28 Jan 2012 01:53:45 -0600 Fri, 27 Jan 2012 18:51:40 -0600 2.0-developer-preview-4 2.0-developer-preview-4 0 0 Fix checked into gerrit <a href="http://review.couchbase.org/#change,12753">http://review.couchbase.org/#change,12753</a> Integrated in github-couchdb-preview #309 (See [<a href="http://qa.hq.northscale.net/job/github-couchdb-preview/309/">http://qa.hq.northscale.net/job/github-couchdb-preview/309/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;Revert to slow reads when old storage file deleted. <a href="http://www.couchbase.com/issues/browse/MB-4704" title="couch_set_view_group dies with {badmatch,{error,enoent}} reason"><strike>MB-4704</strike></a><br/> <br/> damien : <br/> Files : <br/> * src/couchdb/couch_db.erl<br/> ran into a simiar issue while stopping and restart rebalance few times while running views with reduce <br/> <br/> diags attached. <br/> <br/> please close the bug if what you see in the diag is a different bug <br/> <br/> Aliaksey just told me i had to clean the install folder before runnign cluster_run. <br/> <br/> i might not have picked up Filipe fix. <br/> Date of First Response Fri, 27 Jan 2012 18:47:56 -0600 Rank 5755 Time In Status [MB-4697] Sometimes view queries return {"error":"unknown_error","reason":"killed"} http://www.couchbase.com/issues/browse/MB-4697 Couchbase Server It seems to happen only when during rebalance. But not sure yet. MB-4697 Sometimes view queries return {"error":"unknown_error","reason":"killed"} Bug Blocker Resolved Fixed Public Aliaksey Artamonau Aliaksey Artamonau Wed, 25 Jan 2012 21:13:08 -0600 Mon, 23 Jul 2012 14:53:46 -0500 Sun, 29 Jan 2012 23:15:10 -0600 2.0-developer-preview-4 2.0-developer-preview-4 ns_server 0 0 Hi Aliaksey A, <br/> What&#39;s the news/status on this one? <br/> Thanks This happens very rarely with no useful information in the log file. Eyeballing the code with a hope to find what can cause this. <a href="http://review.couchbase.org/#change,12780">http://review.couchbase.org/#change,12780</a> Integrated in github-couchdb-preview #311 (See [<a href="http://qa.hq.northscale.net/job/github-couchdb-preview/311/">http://qa.hq.northscale.net/job/github-couchdb-preview/311/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.couchbase.com/issues/browse/MB-4697" title="Sometimes view queries return {&quot;error&quot;:&quot;unknown_error&quot;,&quot;reason&quot;:&quot;killed&quot;}"><strike>MB-4697</strike></a> Unlink couch_view_merger_queue early.<br/> <br/> aliaksiej.artamonau : <br/> Files : <br/> * src/couchdb/couch_index_merger.erl<br/> Date of First Response Thu, 26 Jan 2012 18:53:35 -0600 Rank 5771 Time In Status [MB-4686] View returns partial results http://www.couchbase.com/issues/browse/MB-4686 Couchbase Server Created 10 nodes cluster. Created a view: {&quot;reduce&quot;:{&quot;map&quot;:&quot;function (doc) {\n emit(doc._id, null);\n}&quot;,&quot;reduce&quot;:&quot;_count&quot;}. Uploaded 100k json documents using mcsoda. Queried a view with stale=false. Removed one node from the cluster. Queried the view with stale=false again while rebalance was in progress. View returned partial result: <br/> <br/> {&quot;rows&quot;:[ <br/> {&quot;key&quot;:null,&quot;value&quot;:88693} <br/> ], <br/> &quot;errors&quot;:[ <br/> {&quot;from&quot;:&quot;<a href="http://10.17.1.200:9503/_view_merge/?stale=false">http://10.17.1.200:9503/_view_merge/?stale=false</a>&quot;,&quot;reason&quot;:&quot;{&lt;&lt;\&quot;noproc\&quot;&gt;&gt;,&lt;&lt;\&quot;{gen_server,call,[&lt;0.2544.0&gt;,{add,&lt;0.6822.0&gt;},infinity]}\&quot;&gt;&gt;}&quot;} <br/> ]} <br/> <br/> This state is permanent: the view keeps returning partial result with this error. There are plenty of compaction_loop_died errors in the log on the failed node: <br/> <br/> =========================CRASH REPORT========================= <br/> &nbsp;&nbsp;crasher: <br/> &nbsp;&nbsp;&nbsp;&nbsp;initial call: couchbase_compaction_daemon:init/1 <br/> &nbsp;&nbsp;&nbsp;&nbsp;pid: &lt;0.12968.0&gt; <br/> &nbsp;&nbsp;&nbsp;&nbsp;registered_name: couchbase_compaction_daemon <br/> &nbsp;&nbsp;&nbsp;&nbsp;exception exit: {compaction_loop_died, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{noproc, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server,call, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&lt;0.2544.0&gt;,{add,&lt;0.12969.0&gt;},infinity]}}} <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in function gen_server:terminate/6 <br/> &nbsp;&nbsp;&nbsp;&nbsp;ancestors: [&lt;0.12967.0&gt;,ns_server_sup,ns_server_cluster_sup,&lt;0.42.0&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;messages: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;links: [&lt;0.12967.0&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;dictionary: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;trap_exit: true <br/> &nbsp;&nbsp;&nbsp;&nbsp;status: running <br/> &nbsp;&nbsp;&nbsp;&nbsp;heap_size: 233 <br/> &nbsp;&nbsp;&nbsp;&nbsp;stack_size: 24 <br/> &nbsp;&nbsp;&nbsp;&nbsp;reductions: 1074 <br/> &nbsp;&nbsp;neighbours: <br/> <br/> <br/> Attaching logs and diag from the node where the error happens. MB-4686 View returns partial results Bug Blocker Resolved Fixed Public Aleksey Kondratenko Aliaksey Artamonau Tue, 24 Jan 2012 14:14:52 -0600 Thu, 9 Feb 2012 20:38:01 -0600 Thu, 9 Feb 2012 20:38:01 -0600 2.0-developer-preview-4 2.0-developer-preview-4 0 0 This is either an issue with database or view reference counters. <br/> There&#39;s a TODO in the view group regarding this that could be the cause. <br/> I just uploaded a change to gerrit to do the todo: <br/> <br/> <a href="http://review.couchbase.org/#change,12638">http://review.couchbase.org/#change,12638</a> <br/> <br/> This can be a race condition, hard to reproduce. Happened to me again with the latest couchdb sources. It seems that new diag won&#39;t help a lot. But I&#39;m attaching it for completeness. The possible fix was merged today. <br/> <br/> Fix regarding the {noproc, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server,call, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&lt;0.2544.0&gt;,{add,&lt;0.12969.0&gt;},infinity]}}} Resolving this on your behalf (feel free to hit Resolve yourself next time to move these forward). Thanks! <br/> <br/> <a href="http://review.couchbase.org/#change,12638">http://review.couchbase.org/#change,12638</a> Encountered very similar error. <br/> <br/> <br/> [couchdb:info] [2012-01-26 17:45:19] [<a href='mailto:n_1@10.17.2.9'>n_1@10.17.2.9</a>:&lt;0.2400.0&gt;:couch_log:info:39] Set view `default`, main group `_design/dev_test`, terminating with reason: {noproc, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;call, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&lt;0.4008.0&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{add, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;0.6714.0&gt;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;infinity]}} <br/> [couchdb:info] [2012-01-26 17:45:19] [<a href='mailto:n_1@10.17.2.9'>n_1@10.17.2.9</a>:&lt;0.6165.0&gt;:couch_log:error:42] Uncaught error in HTTP request: {exit, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{normal, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;call, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&lt;0.2400.0&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{request_group, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ok}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;infinity]}}} <br/> [couchdb:info] [2012-01-26 17:45:19] [<a href='mailto:n_1@10.17.2.9'>n_1@10.17.2.9</a>:&lt;0.6165.0&gt;:couch_log:info:39] Stacktrace: [{couch_index_merger, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;do_query_index, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_index_merger, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;query_index_loop, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;5}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_httpd, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;handle_request_int, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{mochiweb_http, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;headers, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;5}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{proc_lib, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;init_p_do_apply, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;3}] <br/> <br/> <br/> And correspondingly it was reported to the client as: <br/> <br/> {&quot;from&quot;:&quot;<a href="http://10.17.2.9:9501/_view_merge/">http://10.17.2.9:9501/_view_merge/</a>&quot;,&quot;reason&quot;:&quot;{&lt;&lt;\&quot;normal\&quot;&gt;&gt;,\n &lt;&lt;\&quot;{gen_server,call,[&lt;0.2400.0&gt;,{request_group,ok},infinity]}\&quot;&gt;&gt;}&quot;} <br/> <br/> That&#39;s not obvious to me though why couch_set_view_group terminates with reason different from normal but the caller&#39;s error mentions &#39;normal&#39;. More of this: <br/> <br/> <br/> [couchdb:info] [2012-01-26 18:36:25] [<a href='mailto:n_4@10.17.2.9'>n_4@10.17.2.9</a>:&lt;0.2364.0&gt;:couch_log:info:39] Set view `default`, replica group `_design/dev_test`, terminating with reason: {noproc, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;call, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&lt;0.6551.0&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{add, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;0.28961.0&gt;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;infinity]}} <br/> [couchdb:info] [2012-01-26 18:36:25] [<a href='mailto:n_4@10.17.2.9'>n_4@10.17.2.9</a>:&lt;0.2353.0&gt;:couch_log:info:39] Set view `default`, main group `_design/dev_test`, terminating with reason: {function_clause, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{couch_set_view_group, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;handle_info, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{ack, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;0.2364.0&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{noproc, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;call, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&lt;0.6551.0&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{add, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;0.28961.0&gt;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;infinity]}}}, .... <br/> [couchdb:info] [2012-01-26 18:36:25] [<a href='mailto:n_4@10.17.2.9'>n_4@10.17.2.9</a>:&lt;0.28318.0&gt;:couch_log:error:42] Uncaught error in HTTP request: {exit, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{normal, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;call, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&lt;0.2364.0&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{request_group, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;ok}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;infinity]}}} <br/> [couchdb:info] [2012-01-26 18:36:25] [<a href='mailto:n_4@10.17.2.9'>n_4@10.17.2.9</a>:&lt;0.28318.0&gt;:couch_log:info:39] Stacktrace: [{couch_index_merger, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;do_query_index, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_index_merger, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;query_index_loop, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;5}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_httpd, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;handle_request_int, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{mochiweb_http, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;headers, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;5}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{proc_lib, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;init_p_do_apply, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;3}] <br/> <br/> Attached full logs for these two failures. And by the way. In the second case views are non-functional on this node anymore. Index folder always hangs on some couch_set_view_group call: <br/> <br/> Program counter: 0x00007fc3212c6e08 (gen:do_call/4 + 576) <br/> CP: 0x0000000000000000 (invalid) <br/> arity = 0 <br/> <br/> 0x00007fc2cd3e3288 Return addr 0x00007fc32136d9a8 (gen_server:call/3 + 128) <br/> y(0) #Ref&lt;0.0.5.155655&gt; <br/> y(1) &#39;<a href='mailto:n_4@10.17.2.9'>n_4@10.17.2.9</a>&#39; <br/> y(2) [] <br/> y(3) infinity <br/> y(4) {get_group_server,&lt;&lt;7 bytes&gt;&gt;,{set_view_group,&lt;&lt;16 bytes&gt;&gt;,nil,&lt;&lt;7 bytes&gt;&gt;,&lt;&lt;12 bytes&gt;&gt;,&lt;&lt;10 bytes&gt;&gt;,[],[{set_view,0,[],[],[],&lt;&lt;41 bytes&gt;&gt;,nil,[{&lt;&lt;6 bytes&gt;&gt;,&lt;&lt;6 bytes&gt;&gt;}],[]}],{[]},nil,nil,nil,nil,nil,nil,undefined,nil,nil}} <br/> y(5) &#39;$gen_call&#39; <br/> y(6) &lt;0.195.0&gt; <br/> <br/> 0x00007fc2cd3e32c8 Return addr 0x00007fc31de4d118 (couch_set_view:get_group_server/2 + 128) <br/> y(0) infinity <br/> y(1) {get_group_server,&lt;&lt;7 bytes&gt;&gt;,{set_view_group,&lt;&lt;16 bytes&gt;&gt;,nil,&lt;&lt;7 bytes&gt;&gt;,&lt;&lt;12 bytes&gt;&gt;,&lt;&lt;10 bytes&gt;&gt;,[],[{set_view,0,[],[],[],&lt;&lt;41 bytes&gt;&gt;,nil,[{&lt;&lt;6 bytes&gt;&gt;,&lt;&lt;6 bytes&gt;&gt;}],[]}],{[]},nil,nil,nil,nil,nil,nil,undefined,nil,nil}} <br/> y(2) couch_set_view <br/> y(3) Catch 0x00007fc32136d9a8 (gen_server:call/3 + 128) <br/> <br/> 0x00007fc2cd3e32f0 Return addr 0x00007fc31de4cb28 (couch_set_view:get_group/3 + 56) <br/> <br/> 0x00007fc2cd3e32f8 Return addr 0x00007fc31de4f390 (couch_set_view:get_reduce_view/5 + 88) <br/> y(0) ok <br/> <br/> 0x00007fc2cd3e3308 Return addr 0x00007fc2cc7731a8 (couch_view_merger:get_set_view/6 + 208) <br/> y(0) [99,100,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,125,126,128,129,148,149,166,167,168,175,177,178,191,192,193,200,201,218,219,243,244,255] <br/> y(1) &lt;&lt;6 bytes&gt;&gt; <br/> <br/> 0x00007fc2cd3e3320 Return addr 0x00007fc2cc773490 (couch_view_merger:prepare_set_view/4 + 352) <br/> y(0) [99,100,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,125,126,128,129,148,149,166,167,168,175,177,178,191,192,193,200,201,218,219,243,244,255] <br/> y(1) update_after <br/> y(2) &lt;&lt;6 bytes&gt;&gt; <br/> y(3) &lt;&lt;12 bytes&gt;&gt; <br/> y(4) &lt;&lt;7 bytes&gt;&gt; <br/> y(5) #Fun&lt;couch_set_view.get_reduce_view.5&gt; <br/> <br/> 0x00007fc2cd3e3358 Return addr 0x00007fc2cc7774c0 (couch_view_merger:reduce_set_view_folder/4 + 720) <br/> y(0) [] <br/> y(1) [] <br/> y(2) &lt;0.31777.0&gt; <br/> y(3) &lt;&lt;12 bytes&gt;&gt; <br/> y(4) &lt;&lt;7 bytes&gt;&gt; <br/> y(5) Catch 0x00007fc2cc773750 (couch_view_merger:prepare_set_view/4 + 1056) <br/> <br/> 0x00007fc2cd3e3390 Return addr 0x00000000008a4cc8 (&lt;terminate process normally&gt;) <br/> y(0) [] <br/> y(1) [] <br/> y(2) [] <br/> y(3) [] <br/> y(4) [] <br/> y(5) &lt;&lt;14 bytes&gt;&gt; <br/> y(6) &lt;0.31777.0&gt; <br/> y(7) {doc,&lt;&lt;12 bytes&gt;&gt;,{1,&lt;&lt;4 bytes&gt;&gt;},{[{&lt;&lt;5 bytes&gt;&gt;,{[{&lt;&lt;6 bytes&gt;&gt;,{[{&lt;&lt;3 bytes&gt;&gt;,&lt;&lt;41 bytes&gt;&gt;},{&lt;&lt;6 bytes&gt;&gt;,&lt;&lt;6 bytes&gt;&gt;}]}}]}}]},nil,false,[]} <br/> y(8) {index_merge,[{set_view_spec,&lt;&lt;7 bytes&gt;&gt;,&lt;&lt;12 bytes&gt;&gt;,&lt;&lt;6 bytes&gt;&gt;,[99,100,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,125,126,128,129,148,149,166,167,168,175,177,178,191,192,193,200,201,218,219,243,244,255],nil,nil}],#Fun&lt;couch_httpd_view_merger.0.27295029&gt;,{sender_acc,{httpd,{mochiweb_request,#Port&lt;0.29092&gt;,&#39;POST&#39;,&quot;/_view_merge/&quot;,{1,1},{4,{&quot;accept&quot;,{&#39;Accept&#39;,&quot;application/json&quot;},nil,{&quot;content-type&quot;,{&#39;Content-Type&#39;,&quot;application/json&quot;},{&quot;content-length&quot;,{&#39;Content-Length&#39;,&quot;272&quot;},nil,nil},{&quot;host&quot;,{&#39;Host&#39;,&quot;10.17.2.9:9504&quot;},nil,nil}}}}},&quot;10.17.2.9&quot;,&#39;POST&#39;,[&lt;&lt;11 bytes&gt;&gt;],[&lt;&lt;11 bytes&gt;&gt;],capi_frontend,{dict,8,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[[&lt;&lt;7 bytes&gt;&gt;|#Fun&lt;couch_httpd.8.26678597&gt;]],[[&lt;&lt;9 bytes&gt;&gt;|#Fun&lt;couch_httpd.8.26678597&gt;]],[[&lt;&lt;13 bytes&gt;&gt;|#Fun&lt;couch_httpd.8.26678597&gt;]],[],[],[[&lt;&lt;7 bytes&gt;&gt;|#Fun&lt;couch_httpd.8.26678597&gt;]],[[&lt;&lt;16 bytes&gt;&gt;|#Fun&lt;couch_httpd.8.26678597&gt;]],[],[[&lt;&lt;8 bytes&gt;&gt;|#Fun&lt;couch_httpd.8.26678597&gt;]],[],[],[[&lt;&lt;10 bytes&gt;&gt;|#Fun&lt;couch_httpd.8.26678597&gt;]],[[&lt;&lt;8 bytes&gt;&gt;|#Fun&lt;couch_httpd.8.26678597&gt;]],[],[],[]}}},{user_ctx,null,[&lt;&lt;6 bytes&gt;&gt;],&lt;&lt;50 bytes&gt;&gt;},undefined,{dict,11,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[[&lt;&lt;8 bytes&gt;&gt;|#Fun&lt;couch_httpd.10.12509357&gt;],[&lt;&lt;5 bytes&gt;&gt;|#Fun&lt;couch_httpd.10.12509357&gt;]],[[&lt;&lt;5 bytes&gt;&gt;|#Fun&lt;couch_httpd.10.12509357&gt;],[&lt;&lt;5 bytes&gt;&gt;|#Fun&lt;couch_httpd.10.12509357&gt;]],[[&lt;&lt;7 bytes&gt;&gt;|#Fun&lt;couch_httpd.10.12509357&gt;]],[],[[&lt;&lt;14 bytes&gt;&gt;|#Fun&lt;couch_httpd.10.12509357&gt;],[&lt;&lt;14 bytes&gt;&gt;|#Fun&lt;couch_httpd.10.12509357&gt;]],[],[],[],[[&lt;&lt;8 bytes&gt;&gt;|#Fun&lt;couch_httpd.10.12509357&gt;]],[],[],[],[[&lt;&lt;5 bytes&gt;&gt;|#Fun&lt;couch_httpd.10.12509357&gt;]],[],[[&lt;&lt;17 bytes&gt;&gt;|#Fun&lt;couch_httpd.10.12509357&gt;],[&lt;&lt;12 bytes&gt;&gt;|#Fun&lt;couch_httpd.10.12509357&gt;]]}}},undefined,#Fun&lt;couch_httpd.6.96738937&gt;,{dict,18,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[[&lt;&lt;8 bytes&gt;&gt;|#Fun&lt;couch_httpd.6.96738937&gt;],[&lt;&lt;10 bytes&gt;&gt;|#Fun&lt;couch_httpd.6.96738937&gt;]],[[&lt;&lt;13 bytes&gt;&gt;|#Fun&lt;couch_httpd.6.96738937&gt;]],[],[[&lt;&lt;14 bytes&gt;&gt;|#Fun&lt;couch_httpd.6.96738937&gt;]],[[&lt;&lt;6 bytes&gt;&gt;|#Fun&lt;couch_httpd.5.32618959&gt;],[&lt;&lt;1 byte&gt;&gt;|#Fun&lt;couch_httpd.5.32618959&gt;],[&lt;&lt;7 bytes&gt;&gt;|#Fun&lt;couch_httpd.6.96738937&gt;]],[[&lt;&lt;4 bytes&gt;&gt;|#Fun&lt;couch_httpd.6.96738937&gt;]],[[&lt;&lt;9 bytes&gt;&gt;|#Fun&lt;couch_httpd.6.96738937&gt;]],[],[[&lt;&lt;8 bytes&gt;&gt;|#Fun&lt;couch_httpd.6.96738937&gt;],[&lt;&lt;14 bytes&gt;&gt;|#Fun&lt;couch_httpd.6.96738937&gt;],[&lt;&lt;14 bytes&gt;&gt;|#Fun&lt;couch_httpd.6.96738937&gt;]],[],[[&lt;&lt;6 bytes&gt;&gt;|#Fun&lt;couch_httpd.6.96738937&gt;],[&lt;&lt;11 bytes&gt;&gt;|#Fun&lt;couch_httpd.5.32618959&gt;]],[[&lt;&lt;8 bytes&gt;&gt;|#Fun&lt;couch_httpd.6.96738937&gt;],[&lt;&lt;6 bytes&gt;&gt;|#Fun&lt;couch_httpd.6.96738937&gt;]],[[&lt;&lt;11 bytes&gt;&gt;|#Fun&lt;couch_httpd.6.96738937&gt;]],[],[[&lt;&lt;6 bytes&gt;&gt;|#Fun&lt;couch_httpd.6.96738937&gt;]],[]}}}},nil,continue,&lt;&lt;0 bytes&gt;&gt;,[]},60000,continue,{1,&lt;&lt;4 bytes&gt;&gt;},{view_query_args,undefined,undefined,&lt;&lt;0 bytes&gt;&gt;,&lt;&lt;1 byte&gt;&gt;,fwd,true,10000000000,0,0,reduce,false,false,update_after,false,nil,nil,true,nil,&lt;&lt;6 bytes&gt;&gt;},{user_ctx,null,[&lt;&lt;6 bytes&gt;&gt;],&lt;&lt;50 bytes&gt;&gt;},{view_merge,nil,nil,&lt;&lt;10 bytes&gt;&gt;,nil}} <br/> y(9) {view_query_args,undefined,undefined,&lt;&lt;0 bytes&gt;&gt;,&lt;&lt;1 byte&gt;&gt;,fwd,true,10000000000,0,0,reduce,false,false,update_after,false,nil,nil,true,nil,&lt;&lt;6 bytes&gt;&gt;} <br/> y(10) &lt;&lt;12 bytes&gt;&gt; <br/> y(11) &lt;&lt;7 bytes&gt;&gt; <br/> Attached also a full diag for the second case. Try with <a href="http://review.couchbase.org/#change,12767">http://review.couchbase.org/#change,12767</a> Was able to reproduce with all the latest fixes and disabled replica index. Attached verbose log for this issue. farshid hasn&#39;t seen this lately incremental rebalance out with reduce It&#39;s still reproducible. Aliaksey K is looking into it. Try querying now with ?debug=true. It will help diagnose. Integrated in github-couchdb-preview #339 (See [<a href="http://qa.hq.northscale.net/job/github-couchdb-preview/339/">http://qa.hq.northscale.net/job/github-couchdb-preview/339/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;made notify_db_updated consume exits only from parent.<a href="http://www.couchbase.com/issues/browse/MB-4686" title="View returns partial results"><strike>MB-4686</strike></a> (Revision df1e48bea718fc9d6f57feb00aa1a06a1d8e124c)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> Aliaksey Kandratsenka : <br/> Files : <br/> * src/couchdb/couch_db_updater.erl<br/> We believe it&#39;s fixed Date of First Response Tue, 24 Jan 2012 16:53:25 -0600 Rank 5772 Time In Status [MB-4678] couch_set_view_compactor dies with 'badarith' reason http://www.couchbase.com/issues/browse/MB-4678 Couchbase Server Just created a view and queried it. The query didn&#39;t returns anything because of died compactor: <br/> <br/> =========================CRASH REPORT========================= <br/> &nbsp;&nbsp;crasher: <br/> &nbsp;&nbsp;&nbsp;&nbsp;initial call: couch_set_view_group:init/1 <br/> &nbsp;&nbsp;&nbsp;&nbsp;pid: &lt;0.8655.0&gt; <br/> &nbsp;&nbsp;&nbsp;&nbsp;registered_name: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;exception exit: {compactor_died, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{badarith, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{couch_set_view_compactor,update_task,2}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_set_view_compactor, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;-compact_group/4-fun-2-&#39;,4}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_btree_copy,&#39;-before_leaf_write/2-fun-0-&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{lists,mapfoldl,3}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_btree_copy,before_leaf_write,2}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_btree_copy,flush_leaf,2}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_btree_copy,fold_copy,3}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_btree,stream_kv_node2,8}]}} <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in function gen_server:terminate/6 <br/> &nbsp;&nbsp;&nbsp;&nbsp;ancestors: [&lt;0.8644.0&gt;,&lt;0.8643.0&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;messages: [{&#39;EXIT&#39;,&lt;0.8664.0&gt;,normal}] <br/> &nbsp;&nbsp;&nbsp;&nbsp;links: [&lt;0.8658.0&gt;,&lt;0.8644.0&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;dictionary: [{random_seed,{3799,4537,12737}}] <br/> &nbsp;&nbsp;&nbsp;&nbsp;trap_exit: true <br/> &nbsp;&nbsp;&nbsp;&nbsp;status: running <br/> &nbsp;&nbsp;&nbsp;&nbsp;heap_size: 1597 <br/> &nbsp;&nbsp;&nbsp;&nbsp;stack_size: 24 <br/> &nbsp;&nbsp;&nbsp;&nbsp;reductions: 491655 <br/> &nbsp;&nbsp;neighbours: <br/> MB-4678 couch_set_view_compactor dies with 'badarith' reason Bug Blocker Resolved Fixed Public Filipe Manana Aliaksey Artamonau Mon, 23 Jan 2012 16:31:04 -0600 Wed, 3 Apr 2013 07:41:49 -0500 Tue, 24 Jan 2012 13:01:33 -0600 2.0-developer-preview-4 2.0-developer-preview-4 ns_server 0 1 Likely fixed by <a href="http://review.couchbase.org/#change,12612">http://review.couchbase.org/#change,12612</a>.<br/> No deterministic way to reproduce, so closing it until it&#39;s found again after the change mentioned was merged. Date of First Response Tue, 24 Jan 2012 13:01:33 -0600 Rank 5780 Time In Status [MB-4681] View queries hang during rebalance http://www.couchbase.com/issues/browse/MB-4681 Couchbase Server From time to time view requests performed during rebalance hangs forever. Though after rebalance&#39;s been completed everything works fine again. MB-4681 View queries hang during rebalance Bug Blocker Resolved Fixed Public Filipe Manana Aliaksey Artamonau Mon, 23 Jan 2012 18:55:02 -0600 Wed, 3 Apr 2013 07:47:42 -0500 Wed, 3 Apr 2013 07:47:42 -0500 2.0-developer-preview-4 2.0-developer-preview-4 ns_server 0 0 Investigating this on running cluster right now. On of the folders queue items to couch_view_merger_queue is hanging on request group call: <br/> <br/> Program counter: 0x00007feda83dfe08 (gen:do_call/4 + 576) <br/> CP: 0x0000000000000000 (invalid) <br/> arity = 0 <br/> <br/> 0x00007fed9d44c4b0 Return addr 0x00007feda84869a8 (gen_server:call/3 + 128) <br/> y(0) #Ref&lt;0.0.4.203975&gt; <br/> y(1) &#39;<a href='mailto:n_0@10.17.1.200'>n_0@10.17.1.200</a>&#39; <br/> y(2) [] <br/> y(3) infinity <br/> y(4) {request_group,false} <br/> y(5) &#39;$gen_call&#39; <br/> y(6) &lt;0.6187.0&gt; <br/> <br/> 0x00007fed9d44c4f0 Return addr 0x00007fed9c4956a0 (couch_set_view_group:request_replica_group/3 + 224) <br/> y(0) infinity <br/> y(1) {request_group,false} <br/> y(2) &lt;0.6187.0&gt; <br/> y(3) Catch 0x00007feda84869a8 (gen_server:call/3 + 128) <br/> <br/> 0x00007fed9d44c518 Return addr 0x00007fed9c495210 (couch_set_view_group:request_group/3 + 648) <br/> y(0) {set_view_group,&lt;&lt;16 bytes&gt;&gt;,&lt;0.6190.0&gt;,&lt;&lt;7 bytes&gt;&gt;,&lt;&lt;16 bytes&gt;&gt;,&lt;&lt;10 bytes&gt;&gt;,[],[{set_view,0,[{26,393},{27,417},{28,399},{52,416},{53,385},{54,384},{78,347},{79,191},{80,0},{104,0},{105,0},{106,0},{130,0},{131,0},{132,0},{156,0},{157,0},{158,0},{181,0},{182,0},{206,0},{207,0},{208,0},{233,0}],[{26,0},{27,0},{28,0},{52,0},{53,0},{54,0},{78,0},{79,0},{80,0},{104,0},{105,0},{106,0},{130,0},{131,0},{132,0},{156,0},{157,0},{158,0},{181,0},{182,0},{206,0},{207,0},{208,0},{233,0}],[],&lt;&lt;41 bytes&gt;&gt;,{btree,&lt;0.6190.0&gt;,{618032,{2932,[2932],906694396236169742385152},76817},#Fun&lt;couch_btree.3.75015339&gt;,#Fun&lt;couch_btree.4.130216060&gt;,#Fun&lt;couch_set_view.less_json_ids.2&gt;,#Fun&lt;couch_set_view_group.18.8055170&gt;,snappy,5120},[{&lt;&lt;6 bytes&gt;&gt;,&lt;&lt;6 bytes&gt;&gt;}],[]}],{[]},{btree,&lt;0.6190.0&gt;,{542081,{3705,10352619520185845681152133793415538178676663459162274228979956078084096},97005},#Fun&lt;couch_btree.3.75015339&gt;,#Fun&lt;couch_btree.4.130216060&gt;,#Fun&lt;couch_btree.5.75272376&gt;,#Fun&lt;couch_set_view_group.8.90432617&gt;,snappy,5120},nil,nil,&lt;0.6195.0&gt;,{set_view_index_header,1,256,13803492693581127574869511724554050904902217944340773110325048447598592,719908253023001528045108027695365486090234512698773541973131264,0,[{26,393},{27,417},{28,399},{52,416},{53,385},{54,384},{78,347},{79,191},{80,0},{104,0},{105,0},{106,0},{130,0},{131,0},{132,0},{156,0},{157,0},{158,0},{181,0},{182,0},{206,0},{207,0},{208,0},{233,0}],[{26,0},{27,0},{28,0},{52,0},{53,0},{54,0},{78,0},{79,0},{80,0},{104,0},{105,0},{106,0},{130,0},{131,0},{132,0},{156,0},{157,0},{158,0},{181,0},{182,0},{206,0},{207,0},{208,0},{233,0}],nil,[{nil,[],[]}],false,[]},&lt;0.6196.0&gt;,replica,nil,nil} <br/> <br/> 0x00007fed9d44c528 Return addr 0x00007feda52f8560 (couch_set_view:get_group/3 + 104) <br/> y(0) &lt;0.12361.0&gt; <br/> y(1) &lt;&lt;16 bytes&gt;&gt; <br/> y(2) &lt;&lt;7 bytes&gt;&gt; <br/> y(3) {set_view_group,&lt;&lt;16 bytes&gt;&gt;,&lt;0.12354.0&gt;,&lt;&lt;7 bytes&gt;&gt;,&lt;&lt;16 bytes&gt;&gt;,&lt;&lt;10 bytes&gt;&gt;,[],[{set_view,0,[{0,485},{1,479},{2,505},{3,467},{4,490},{5,491},{6,472},{7,438},{8,465},{9,460},{10,449},{11,424},{12,413},{13,427},{14,442},{15,401},{16,401},{17,384},{18,384},{19,375},{20,387},{21,409},{22,404},{23,399},{24,431},{25,398},{231,760},{232,786},{233,400}],[{0,0},{1,0},{2,0},{3,0},{4,0},{5,0},{6,0},{7,0},{8,0},{9,0},{10,0},{11,0},{12,0},{13,0},{14,0},{15,0},{16,0},{17,0},{18,0},{19,0},{20,0},{21,0},{22,0},{23,0},{24,0},{25,0},{231,0},{232,0},{233,0}],[],&lt;&lt;41 bytes&gt;&gt;,{btree,&lt;0.12354.0&gt;,{1322351,{11191,[11191],24156112213766973256021645517969589083578881402596352943068834850406399},307778},#Fun&lt;couch_btree.3.75015339&gt;,#Fun&lt;couch_btree.4.130216060&gt;,#Fun&lt;couch_set_view.less_json_ids.2&gt;,#Fun&lt;couch_set_view_group.18.8055170&gt;,snappy,5120},[{&lt;&lt;6 bytes&gt;&gt;,&lt;&lt;6 bytes&gt;&gt;}],[]}],{[]},{btree,&lt;0.12354.0&gt;,{1222634,{11191,24156112213766973256021645517969589083578881402596352943068834850406399},303075},#Fun&lt;couch_btree.3.75015339&gt;,#Fun&lt;couch_btree.4.130216060&gt;,#Fun&lt;couch_btree.5.75272376&gt;,#Fun&lt;couch_set_view_group.8.90432617&gt;,snappy,5120},nil,nil,&lt;0.12361.0&gt;,{set_view_index_header,1,256,10352619520185845681152133793415538178676663458255579832743786402807807,13803492693581127574869511724554050904902217944340773110325048447598592,0,[{0,485},{1,479},{2,505},{3,467},{4,490},{5,491},{6,472},{7,438},{8,465},{9,460},{10,449},{11,424},{12,413},{13,427},{14,442},{15,401},{16,401},{17,384},{18,384},{19,375},{20,387},{21,409},{22,404},{23,399},{24,431},{25,398},{231,760},{232,786},{233,400}],[{0,0},{1,0},{2,0},{3,0},{4,0},{5,0},{6,0},{7,0},{8,0},{9,0},{10,0},{11,0},{12,0},{13,0},{14,0},{15,0},{16,0},{17,0},{18,0},{19,0},{20,0},{21,0},{22,0},{23,0},{24,0},{25,0},{231,0},{232,0},{233,0}],{254917,{10018,67108863},254980},[{{506133,{10018,[10018],67108863},251282},[{0,485},{1,479},{2,505},{3,467},{4,490},{5,491},{6,472},{7,438},{8,465},{9,460},{10,449},{11,424},{12,413},{13,427},{14,442},{15,401},{16,401},{17,384},{18,384},{19,375},{20,387},{21,409},{22,404},{23,399},{24,431},{25,398}],[{0,0},{1,0},{2,0},{3,0},{4,0},{5,0},{6,0},{7,0},{8,0},{9,0},{10,0},{11,0},{12,0},{13,0},{14,0},{15,0},{16,0},{17,0},{18,0},{19,0},{20,0},{21,0},{22,0},{23,0},{24,0},{25,0}]}],true,&quot;&#xE9;&quot;},&lt;0.6185.0&gt;,main,{set_view_group,&lt;&lt;16 bytes&gt;&gt;,&lt;0.6190.0&gt;,&lt;&lt;7 bytes&gt;&gt;,&lt;&lt;16 bytes&gt;&gt;,&lt;&lt;10 bytes&gt;&gt;,[],[{set_view,0,[{26,393},{27,417},{28,399},{52,416},{53,385},{54,384},{78,347},{79,191},{80,0},{104,0},{105,0},{106,0},{130,0},{131,0},{132,0},{156,0},{157,0},{158,0},{181,0},{182,0},{206,0},{207,0},{208,0},{233,0}],[{26,0},{27,0},{28,0},{52,0},{53,0},{54,0},{78,0},{79,0},{80,0},{104,0},{105,0},{106,0},{130,0},{131,0},{132,0},{156,0},{157,0},{158,0},{181,0},{182,0},{206,0},{207,0},{208,0},{233,0}],[],&lt;&lt;41 bytes&gt;&gt;,{btree,&lt;0.6190.0&gt;,{618032,{2932,[2932],906694396236169742385152},76817},#Fun&lt;couch_btree.3.75015339&gt;,#Fun&lt;couch_btree.4.130216060&gt;,#Fun&lt;couch_set_view.less_json_ids.2&gt;,#Fun&lt;couch_set_view_group.18.8055170&gt;,snappy,5120},[{&lt;&lt;6 bytes&gt;&gt;,&lt;&lt;6 bytes&gt;&gt;}],[]}],{[]},{btree,&lt;0.6190.0&gt;,{542081,{3705,10352619520185845681152133793415538178676663459162274228979956078084096},97005},#Fun&lt;couch_btree.3.75015339&gt;,#Fun&lt;couch_btree.4.130216060&gt;,#Fun&lt;couch_btree.5.75272376&gt;,#Fun&lt;couch_set_view_group.8.90432617&gt;,snappy,5120},nil,nil,&lt;0.6195.0&gt;,{set_view_index_header,1,256,13803492693581127574869511724554050904902217944340773110325048447598592,719908253023001528045108027695365486090234512698773541973131264,0,[{26,393},{27,417},{28,399},{52,416},{53,385},{54,384},{78,347},{79,191},{80,0},{104,0},{105,0},{106,0},{130,0},{131,0},{132,0},{156,0},{157,0},{158,0},{181,0},{182,0},{206,0},{207,0},{208,0},{233,0}],[{26,0},{27,0},{28,0},{52,0},{53,0},{54,0},{78,0},{79,0},{80,0},{104,0},{105,0},{106,0},{130,0},{131,0},{132,0},{156,0},{157,0},{158,0},{181,0},{182,0},{206,0},{207,0},{208,0},{233,0}],nil,[{nil,[],[]}],false,[]},&lt;0.6196.0&gt;,replica,nil,nil},&lt;0.6187.0&gt;} <br/> y(4) [] <br/> y(5) [] <br/> y(6) &lt;0.6121.0&gt; <br/> y(7) false <br/> y(8) 1 <br/> <br/> 0x00007fed9d44c578 Return addr 0x00007feda52fad98 (couch_set_view:get_reduce_view/5 + 88) <br/> y(0) [] <br/> <br/> 0x00007fed9d44c588 Return addr 0x00007fed4f81e770 (couch_view_merger:prepare_set_view/4 + 352) <br/> y(0) [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,231,232] <br/> y(1) &lt;&lt;6 bytes&gt;&gt; <br/> <br/> 0x00007fed9d44c5a0 Return addr 0x00007fed4f8227a0 (couch_view_merger:reduce_set_view_folder/4 + 720) <br/> y(0) [] <br/> y(1) [] <br/> y(2) &lt;0.12632.0&gt; <br/> y(3) &lt;&lt;12 bytes&gt;&gt; <br/> y(4) &lt;&lt;7 bytes&gt;&gt; <br/> y(5) Catch 0x00007fed4f81ea30 (couch_view_merger:prepare_set_view/4 + 1056) <br/> <br/> 0x00007fed9d44c5d8 Return addr 0x00000000008a4cc8 (&lt;terminate process normally&gt;) <br/> y(0) [] <br/> y(1) [] <br/> y(2) [] <br/> y(3) [] <br/> y(4) [] <br/> y(5) &lt;&lt;14 bytes&gt;&gt; <br/> y(6) &lt;0.12632.0&gt; <br/> y(7) {doc,&lt;&lt;12 bytes&gt;&gt;,{1,&lt;&lt;4 bytes&gt;&gt;},{[{&lt;&lt;5 bytes&gt;&gt;,{[{&lt;&lt;6 bytes&gt;&gt;,{[{&lt;&lt;3 bytes&gt;&gt;,&lt;&lt;41 bytes&gt;&gt;},{&lt;&lt;6 bytes&gt;&gt;,&lt;&lt;6 bytes&gt;&gt;}]}}]}}]},nil,false,[]} <br/> y(8) {index_merge,[{merged_index_spec,&lt;&lt;35 bytes&gt;&gt;,{[{&lt;&lt;5 bytes&gt;&gt;,{[{&lt;&lt;4 bytes&gt;&gt;,{[{&lt;&lt;7 bytes&gt;&gt;,{[{&lt;&lt;4 bytes&gt;&gt;,&lt;&lt;19 bytes&gt;&gt;},{&lt;&lt;10 bytes&gt;&gt;,&quot;&#xE9;&#xEA;&#xEB;&#xEC;&#xED;&#xEE;&#xEF;&#xF0;&#xF1;&#xF2;&#xF3;&#xF4;&#xF5;&#xF6;&#xF7;&#xF8;&#xF9;&#xFA;&#xFB;&#xFC;&#xFD;&#xFE;&#xFF;&quot;}]}}]}}]}}]}},{merged_index_spec,&lt;&lt;35 bytes&gt;&gt;,{[{&lt;&lt;5 bytes&gt;&gt;,{[{&lt;&lt;4 bytes&gt;&gt;,{[{&lt;&lt;7 bytes&gt;&gt;,{[{&lt;&lt;4 bytes&gt;&gt;,&lt;&lt;19 bytes&gt;&gt;},{&lt;&lt;10 bytes&gt;&gt;,&quot;&#xCE;&#xCF;&#xD0;&#xD1;&#xD2;&#xD3;&#xD4;&#xD5;&#xD6;&#xD7;&#xD8;&#xD9;&#xDA;&#xDB;&#xDC;&#xDD;&#xDE;&#xDF;&#xE0;&#xE1;&#xE2;&#xE3;&#xE4;&#xE5;&#xE6;&quot;}]}}]}}]}}]}},{merged_index_spec,&lt;&lt;35 bytes&gt;&gt;,{[{&lt;&lt;5 bytes&gt;&gt;,{[{&lt;&lt;4 bytes&gt;&gt;,{[{&lt;&lt;7 bytes&gt;&gt;,{[{&lt;&lt;4 bytes&gt;&gt;,&lt;&lt;19 bytes&gt;&gt;},{&lt;&lt;10 bytes&gt;&gt;,&quot;&#xB5;&#xB6;&#xB7;&#xB8;&#xB9;&#xBA;&#xBB;&#xBC;&#xBD;&#xBE;&#xBF;&#xC0;&#xC1;&#xC2;&#xC3;&#xC4;&#xC5;&#xC6;&#xC7;&#xC8;&#xC9;&#xCA;&#xCB;&#xCC;&#xCD;&quot;}]}}]}}]}}]}},{merged_index_spec,&lt;&lt;35 bytes&gt;&gt;,{[{&lt;&lt;5 bytes&gt;&gt;,{[{&lt;&lt;4 bytes&gt;&gt;,{[{&lt;&lt;7 bytes&gt;&gt;,{[{&lt;&lt;4 bytes&gt;&gt;,&lt;&lt;19 bytes&gt;&gt;},{&lt;&lt;10 bytes&gt;&gt;,[156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,177,178,179,180]}]}}]}}]}}]}},{merged_index_spec,&lt;&lt;35 bytes&gt;&gt;,{[{&lt;&lt;5 bytes&gt;&gt;,{[{&lt;&lt;4 bytes&gt;&gt;,{[{&lt;&lt;7 bytes&gt;&gt;,{[{&lt;&lt;4 bytes&gt;&gt;,&lt;&lt;19 bytes&gt;&gt;},{&lt;&lt;10 bytes&gt;&gt;,[130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155]}]}}]}}]}}]}},{merged_index_spec,&lt;&lt;35 bytes&gt;&gt;,{[{&lt;&lt;5 bytes&gt;&gt;,{[{&lt;&lt;4 bytes&gt;&gt;,{[{&lt;&lt;7 bytes&gt;&gt;,{[{&lt;&lt;4 bytes&gt;&gt;,&lt;&lt;19 bytes&gt;&gt;},{&lt;&lt;10 bytes&gt;&gt;,[104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129]}]}}]}}]}}]}},{merged_index_spec,&lt;&lt;35 bytes&gt;&gt;,{[{&lt;&lt;5 bytes&gt;&gt;,{[{&lt;&lt;4 bytes&gt;&gt;,{[{&lt;&lt;7 bytes&gt;&gt;,{[{&lt;&lt;4 bytes&gt;&gt;,&lt;&lt;19 bytes&gt;&gt;},{&lt;&lt;10 bytes&gt;&gt;,&quot;NOPQRSTUVWXYZ[\]^_`abcdefg&quot;}]}}]}}]}}]}},{merged_index_spec,&lt;&lt;35 bytes&gt;&gt;,{[{&lt;&lt;5 bytes&gt;&gt;,{[{&lt;&lt;4 bytes&gt;&gt;,{[{&lt;&lt;7 bytes&gt;&gt;,{[{&lt;&lt;4 bytes&gt;&gt;,&lt;&lt;19 bytes&gt;&gt;},{&lt;&lt;10 bytes&gt;&gt;,&quot;456789:;&lt;=&gt;?@ABCDEFGHIJKLM&quot;}]}}]}}]}}]}},{merged_index_spec,&lt;&lt;35 bytes&gt;&gt;,{[{&lt;&lt;5 bytes&gt;&gt;,{[{&lt;&lt;4 bytes&gt;&gt;,{[{&lt;&lt;7 bytes&gt;&gt;,{[{&lt;&lt;4 bytes&gt;&gt;,&lt;&lt;19 bytes&gt;&gt;},{&lt;&lt;10 bytes&gt;&gt;,[26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51]}]}}]}}]}}]}},{set_view_spec,&lt;&lt;7 bytes&gt;&gt;,&lt;&lt;12 bytes&gt;&gt;,&lt;&lt;6 bytes&gt;&gt;,[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,231,232],nil,nil}],#Fun&lt;couch_httpd_view_merger.0.27295029&gt;,{sender_acc,{httpd,{mochiweb_request,#Port&lt;0.20203&gt;,&#39;GET&#39;,&quot;/default/_design/test/_view/reduce?stale=false&quot;,{1,1},{3,{&quot;user-agent&quot;,{&#39;User-Agent&#39;,&quot;curl/7.23.1 (x86_64-unknown-linux-gnu) libcurl/7.23.1 OpenSSL/1.0.0f zlib/1.2.5 libssh2/1.3.0&quot;},{&quot;host&quot;,{&#39;Host&#39;,&quot;127.0.0.1:9500&quot;},{&quot;accept&quot;,{&#39;Accept&#39;,&quot;*/*&quot;},nil,nil},nil},nil}}},&quot;127.0.0.1&quot;,&#39;GET&#39;,[&lt;&lt;7 bytes&gt;&gt;,&lt;&lt;7 bytes&gt;&gt;,&lt;&lt;4 bytes&gt;&gt;,&lt;&lt;5 bytes&gt;&gt;,&lt;&lt;18 bytes&gt;&gt;],[&lt;&lt;7 bytes&gt;&gt;,&lt;&lt;7 bytes&gt;&gt;,&lt;&lt;4 bytes&gt;&gt;,&lt;&lt;5 bytes&gt;&gt;,&lt;&lt;6 bytes&gt;&gt;],capi_frontend,{dict,8,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[[&lt;&lt;7 bytes&gt;&gt;|#Fun&lt;couch_httpd.8.26678597&gt;]],[[&lt;&lt;9 bytes&gt;&gt;|#Fun&lt;couch_httpd.8.26678597&gt;]],[[&lt;&lt;13 bytes&gt;&gt;|#Fun&lt;couch_httpd.8.26678597&gt;]],[],[],[[&lt;&lt;7 bytes&gt;&gt;|#Fun&lt;couch_httpd.8.26678597&gt;]],[[&lt;&lt;16 bytes&gt;&gt;|#Fun&lt;couch_httpd.8.26678597&gt;]],[],[[&lt;&lt;8 bytes&gt;&gt;|#Fun&lt;couch_httpd.8.26678597&gt;]],[],[],[[&lt;&lt;10 bytes&gt;&gt;|#Fun&lt;couch_httpd.8.26678597&gt;]],[[&lt;&lt;8 bytes&gt;&gt;|#Fun&lt;couch_httpd.8.26678597&gt;]],[],[],[]}}},{user_ctx,null,[&lt;&lt;6 bytes&gt;&gt;],&lt;&lt;50 bytes&gt;&gt;},undefined,{dict,11,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[],[[&lt;&lt;8 bytes&gt;&gt;|#Fun&lt;couch_httpd.10.12509357&gt;],[&lt;&lt;5 bytes&gt;&gt;|#Fun&lt;couch_httpd.10.12509357&gt;]],[[&lt;&lt;5 bytes&gt;&gt;|#Fun&lt;couch_httpd.10.12509357&gt;],[&lt;&lt;5 bytes&gt;&gt;|#Fun&lt;couch_httpd.10.12509357&gt;]],[[&lt;&lt;7 bytes&gt;&gt;|#Fun&lt;couch_httpd.10.12509357&gt;]],[],[[&lt;&lt;14 bytes&gt;&gt;|#Fun&lt;couch_httpd.10.12509357&gt;],[&lt;&lt;14 bytes&gt;&gt;|#Fun&lt;couch_httpd.10.12509357&gt;]],[],[],[],[[&lt;&lt;8 bytes&gt;&gt;|#Fun&lt;couch_httpd.10.12509357&gt;]],[],[],[],[[&lt;&lt;5 bytes&gt;&gt;|#Fun&lt;couch_httpd.10.12509357&gt;]],[],[[&lt;&lt;17 bytes&gt;&gt;|#Fun&lt;couch_httpd.10.12509357&gt;],[&lt;&lt;12 bytes&gt;&gt;|#Fun&lt;couch_httpd.10.12509357&gt;]]}}},undefined,#Fun&lt;couch_httpd.6.96738937&gt;,{dict,18,16,16,8,80,48,{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},{{[[&lt;&lt;8 bytes&gt;&gt;|#Fun&lt;couch_httpd.6.96738937&gt;],[&lt;&lt;10 bytes&gt;&gt;|#Fun&lt;couch_httpd.6.96738937&gt;]],[[&lt;&lt;13 bytes&gt;&gt;|#Fun&lt;couch_httpd.6.96738937&gt;]],[],[[&lt;&lt;14 bytes&gt;&gt;|#Fun&lt;couch_httpd.6.96738937&gt;]],[[&lt;&lt;6 bytes&gt;&gt;|#Fun&lt;couch_httpd.5.32618959&gt;],[&lt;&lt;1 byte&gt;&gt;|#Fun&lt;couch_httpd.5.32618959&gt;],[&lt;&lt;7 bytes&gt;&gt;|#Fun&lt;couch_httpd.6.96738937&gt;]],[[&lt;&lt;4 bytes&gt;&gt;|#Fun&lt;couch_httpd.6.96738937&gt;]],[[&lt;&lt;9 bytes&gt;&gt;|#Fun&lt;couch_httpd.6.96738937&gt;]],[],[[&lt;&lt;8 bytes&gt;&gt;|#Fun&lt;couch_httpd.6.96738937&gt;],[&lt;&lt;14 bytes&gt;&gt;|#Fun&lt;couch_httpd.6.96738937&gt;],[&lt;&lt;14 bytes&gt;&gt;|#Fun&lt;couch_httpd.6.96738937&gt;]],[],[[&lt;&lt;6 bytes&gt;&gt;|#Fun&lt;couch_httpd.6.96738937&gt;],[&lt;&lt;11 bytes&gt;&gt;|#Fun&lt;couch_httpd.5.32618959&gt;]],[[&lt;&lt;8 bytes&gt;&gt;|#Fun&lt;couch_httpd.6.96738937&gt;],[&lt;&lt;6 bytes&gt;&gt;|#Fun&lt;couch_httpd.6.96738937&gt;]],[[&lt;&lt;11 bytes&gt;&gt;|#Fun&lt;couch_httpd.6.96738937&gt;]],[],[[&lt;&lt;6 bytes&gt;&gt;|#Fun&lt;couch_httpd.6.96738937&gt;]],[]}}}},nil,continue,&lt;&lt;0 bytes&gt;&gt;,[]},60000,continue,auto,{view_query_args,undefined,undefined,&lt;&lt;0 bytes&gt;&gt;,&lt;&lt;1 byte&gt;&gt;,fwd,true,10000000000,0,0,reduce,false,false,false,false,nil,nil,true,nil,&lt;&lt;6 bytes&gt;&gt;},{user_ctx,null,[&lt;&lt;6 bytes&gt;&gt;],&lt;&lt;50 bytes&gt;&gt;},{view_merge,nil,nil,&lt;&lt;10 bytes&gt;&gt;,nil}} <br/> y(9) {view_query_args,undefined,undefined,&lt;&lt;0 bytes&gt;&gt;,&lt;&lt;1 byte&gt;&gt;,fwd,true,10000000000,0,0,reduce,false,false,false,false,nil,nil,true,nil,&lt;&lt;6 bytes&gt;&gt;} <br/> y(10) &lt;&lt;12 bytes&gt;&gt; <br/> y(11) &lt;&lt;7 bytes&gt;&gt; <br/> Still happens to me with all the latest fixes. With <a href="http://review.couchbase.org/#change,12711">http://review.couchbase.org/#change,12711</a> it became even worse. I see these hangs almost every rebalance. Closing this as fixed. Likely fixed by<br/> <br/> <a href="http://review.couchbase.org/#change,13032">http://review.couchbase.org/#change,13032</a><br/> <br/> and/or one of many other small changes that happened in the last 2 or 3 weeks.<br/> <br/> Reopen if it still happens. Date of First Response Thu, 9 Feb 2012 15:34:01 -0600 Rank 5786 Time In Status [MB-4664] mc-kvstore does not retry commands if mc_daemon crashes http://www.couchbase.com/issues/browse/MB-4664 Couchbase Server I was creating 100k items on a fresh cluster and some of couch_db:update_docs calls in mccouch failed thus causing mc_daemon to crash and thus to close a connection. After all sets where done there were 100k items available in memcached. But only about 90k of them were persisted and available from couchdb. Thus it seems that ep-engine or mc-kvstore does not attempt to retry sending commands if the connection to mccouch is being closed. MB-4664 mc-kvstore does not retry commands if mc_daemon crashes Bug Blocker Resolved Fixed Public Chiyoung Seo Aliaksey Artamonau Wed, 18 Jan 2012 21:27:09 -0600 Tue, 10 Apr 2012 21:03:21 -0500 Thu, 19 Jan 2012 13:49:27 -0600 2.0-developer-preview-4 2.0-developer-preview-4 couchbase-bucket 0 0 <a href="http://review.couchbase.org/#change,12517">http://review.couchbase.org/#change,12517</a> Integrated in github-ep-engine-2-0 #151 (See [<a href="http://qa.hq.northscale.net/job/github-ep-engine-2-0/151/">http://qa.hq.northscale.net/job/github-ep-engine-2-0/151/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.couchbase.com/issues/browse/MB-4664" title="mc-kvstore does not retry commands if mc_daemon crashes"><strike>MB-4664</strike></a> Requeue a mutation if a connection to mccouch is reset.<br/> <br/> Chiyoung Seo : <br/> Files : <br/> * mc-kvstore/mc-engine.cc<br/> Date of First Response Thu, 19 Jan 2012 13:49:27 -0600 Rank 5795 Time In Status [MB-4663] couch_db:update_docs crashes with reason {update_error,compaction_retry} http://www.couchbase.com/issues/browse/MB-4663 Couchbase Server I just tried to upload 100k items to a fresh cluster and saw several mccouch failures with this error. Attaching the log file. I also tried to reproduce it several times with no luck. MB-4663 couch_db:update_docs crashes with reason {update_error,compaction_retry} Bug Blocker Resolved Fixed Public Damien Katz Aliaksey Artamonau Wed, 18 Jan 2012 21:17:49 -0600 Thu, 19 Jan 2012 16:36:02 -0600 Thu, 19 Jan 2012 16:24:22 -0600 2.0-developer-preview-4 2.0-developer-preview-4 0 0 Change I33a9b089: Fix for <a href="http://www.couchbase.com/issues/browse/MB-4663" title="couch_db:update_docs crashes with reason {update_error,compaction_retry}"><strike>MB-4663</strike></a>, compaction retry race condition. Integrated in github-couchdb-preview #263 (See [<a href="http://qa.hq.northscale.net/job/github-couchdb-preview/263/">http://qa.hq.northscale.net/job/github-couchdb-preview/263/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;Fix for <a href="http://www.couchbase.com/issues/browse/MB-4663" title="couch_db:update_docs crashes with reason {update_error,compaction_retry}"><strike>MB-4663</strike></a>, compaction retry race condition.<br/> <br/> damien : <br/> Files : <br/> * src/couchdb/couch_db_updater.erl<br/> Date of First Response Thu, 19 Jan 2012 16:24:22 -0600 Rank 5801 Time In Status [MB-4651] Some race between insertion and delitions items to couch_setview_name_to_sig table http://www.couchbase.com/issues/browse/MB-4651 Couchbase Server Farshid got this while running test from this change set: <a href="http://review.couchbase.org/#change,12361">http://review.couchbase.org/#change,12361</a>. Among plenty of other errors in the log there is this one: <br/> <br/> =========================CRASH REPORT========================= <br/> &nbsp;&nbsp;crasher: <br/> &nbsp;&nbsp;&nbsp;&nbsp;initial call: couch_event_sup:init/1 <br/> &nbsp;&nbsp;&nbsp;&nbsp;pid: &lt;0.187.0&gt; <br/> &nbsp;&nbsp;&nbsp;&nbsp;registered_name: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;exception exit: {&#39;EXIT&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{{case_clause, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{&lt;&lt;&quot;default&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&lt;&lt;&quot;_design/dev_test_view_on_10k_docs-c5abd37&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;57,37,41,201,93,208,192,231,227,9,177,64, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;251,180,145,114&gt;&gt;}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&lt;&lt;&quot;default&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&lt;&lt;&quot;_design/dev_test_view_on_10k_docs-c5abd37&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;114,187,98,215,41,151,58,48,85,66,14,103, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;185,80,79,64&gt;&gt;}}]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{couch_set_view,handle_db_event,1}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_db_update_notifier,handle_event,2}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_event,server_update,4}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_event,server_notify,4}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_event,server_notify,4}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_event,handle_msg,5}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{proc_lib,init_p_do_apply,3}]}} <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in function gen_server:terminate/6 <br/> &nbsp;&nbsp;&nbsp;&nbsp;ancestors: [couch_set_view,couch_secondary_services,couch_server_sup, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;cb_couch_sup,ns_server_cluster_sup,&lt;0.41.0&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;messages: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;links: [&lt;0.186.0&gt;,&lt;0.153.0&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;dictionary: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;trap_exit: false <br/> &nbsp;&nbsp;&nbsp;&nbsp;status: running <br/> &nbsp;&nbsp;&nbsp;&nbsp;heap_size: 377 <br/> &nbsp;&nbsp;&nbsp;&nbsp;stack_size: 24 <br/> &nbsp;&nbsp;&nbsp;&nbsp;reductions: 152 <br/> &nbsp;&nbsp;neighbours: <br/> <br/> After a quick look at handle_db_event code it seems that this function does not expect to find several signatures for the same design document. So there must be some race between the code that calls add_to_ets and the one that calls delete_from_ets. MB-4651 Some race between insertion and delitions items to couch_setview_name_to_sig table Bug Blocker Resolved Fixed Public Filipe Manana Aliaksey Artamonau Mon, 16 Jan 2012 21:23:22 -0600 Wed, 3 Apr 2013 07:41:51 -0500 Wed, 18 Jan 2012 20:05:45 -0600 2.0-developer-preview-4 2.0-developer-preview-4 ns_server 0 0 Marking this resolved on news from Aliaksey A Date of First Response Wed, 18 Jan 2012 20:05:45 -0600 Rank 5807 Time In Status [MB-4650] Main set view group dies with reason {cleaner_died, {function_clause, [{couch_compress, decompress,[<<...]}]}} http://www.couchbase.com/issues/browse/MB-4650 Couchbase Server While running viewtests.ViewTests.test_load_10k_during_rebalance against 4 nodes cluster I encountered this error: <br/> <br/> &nbsp;Set view `default`, main group `_design/dev_test_view_on_10k_docs-1a0a615`, terminating with reason: {cleaner_died, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{function_clause, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{couch_compress, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;decompress, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&lt;&lt;... <br/> <br/> Three seconds before there is a message from compaction daemon: <br/> <br/> Compaction daemon - an error occurred while compacting the main view group `_design/dev_test_view_on_10k_docs-1a0a615` from bucket `default`: killed <br/> <br/> In parallel capi_set_view_manager was doing its usual job of setting partitions states in a view group. Surprisingly it was able to call couch_set_view:set_partition_states without errors but this call just hung and would seemingly never return. Attaching the log from this node. MB-4650 Main set view group dies with reason {cleaner_died, {function_clause, [{couch_compress, decompress,[<<...]}]}} Bug Blocker Resolved Fixed Public Filipe Manana Aliaksey Artamonau Mon, 16 Jan 2012 17:37:34 -0600 Wed, 3 Apr 2013 07:41:49 -0500 Thu, 19 Jan 2012 06:21:36 -0600 2.0-developer-preview-4 2.0-developer-preview-4 0 0 One more crash with similar reason. Another similar crash. Happened with <a href="http://review.couchbase.org/#change,12419">http://review.couchbase.org/#change,12419</a> applied. Filipe, <br/> Aliaksey A. also mentions this as most likely fixed. Can you confirm / resolve if so? <br/> Thanks! Apparently fixed by<br/> <br/> <a href="http://review.couchbase.org/#change,12491">http://review.couchbase.org/#change,12491</a><br/> <br/> and<br/> <br/> <a href="http://review.couchbase.org/#change,12487">http://review.couchbase.org/#change,12487</a> Date of First Response Wed, 18 Jan 2012 20:06:52 -0600 Rank 5810 Time In Status [MB-4644] couch_set_view_group dies with reason {read_loop_died,{badmatch,eof}} http://www.couchbase.com/issues/browse/MB-4644 Couchbase Server Encountered this while running viewtests.ViewTests.test_load_10k_during_rebalance testrunner test against 4 node cluster (started by cluster_run -n4). Attaching the log file from failing node. MB-4644 couch_set_view_group dies with reason {read_loop_died,{badmatch,eof}} Bug Blocker Resolved Fixed Public Filipe Manana Aliaksey Artamonau Sat, 14 Jan 2012 00:46:09 -0600 Wed, 3 Apr 2013 07:41:51 -0500 Thu, 19 Jan 2012 06:21:51 -0600 2.0-developer-preview-4 2.0-developer-preview-4 ns_server 0 1 Another similar error. Filipe, <br/> Aliaksey A. mentions this as most likely fixed. Can you confirm / resolve if so? <br/> Thanks! Apparently fixed by<br/> <br/> <a href="http://review.couchbase.org/#change,12491">http://review.couchbase.org/#change,12491</a><br/> <br/> and<br/> <br/> <a href="http://review.couchbase.org/#change,12487">http://review.couchbase.org/#change,12487</a> Date of First Response Wed, 18 Jan 2012 20:06:36 -0600 Rank 5818 Time In Status [MB-4560] Crash in set-index compaction code http://www.couchbase.com/issues/browse/MB-4560 Couchbase Server Not sure if it&#39;s easily reproducible. But I simply loaded 50k tiny items, defined nop index (the one our UI defines). Run it on full subset. Loaded 150k tiny items overwriting previously loaded 50k items. Then spotted that some compaction code is crashing. Attaching diag. Developer&#39;s 2 nodes cluster MB-4560 Crash in set-index compaction code Bug Blocker Resolved Fixed Public Filipe Manana Aleksey Kondratenko Tue, 20 Dec 2011 07:58:51 -0600 Wed, 3 Apr 2013 07:41:50 -0500 Thu, 19 Jan 2012 06:36:36 -0600 2.0-developer-preview-4 2.0-developer-preview-4 view-engine 0 0 Alk, for that very particular scenario, can you see if the following patch works: <br/> <br/> <a href="http://friendpaste.com/s3PjhxzMQeaboSa5dDEi0">http://friendpaste.com/s3PjhxzMQeaboSa5dDEi0</a> <br/> <br/> Also, if compaction goes successful, see if you can query the full view and the rows are properly sorted. <br/> Thanks Fixed and merged long time ago. Date of First Response Tue, 20 Dec 2011 08:34:12 -0600 Rank 5896 Time In Status [MB-4459] Deadlock in flusher thread and assertion failure http://www.couchbase.com/issues/browse/MB-4459 Couchbase Server Found this in user-visible logs: <br/> <br/> Port server memcached on node &#39;<a href='mailto:n_1@192.168.1.103'>n_1@192.168.1.103</a>&#39; exited with status 134. Restarting. Messages: Exception in flusher loop: MUTEX ERROR: Failed to acquire lock: Resource deadlock avoided <br/> <br/> memcached.orig: flusher.cc:275: bool Flusher::step(Dispatcher&amp;, TaskId): Assertion `false&#39; failed. MB-4459 Deadlock in flusher thread and assertion failure Bug Blocker Resolved Cannot Reproduce Public Mike Wiederhold Aleksey Kondratenko Mon, 21 Nov 2011 09:36:50 -0600 Tue, 10 Apr 2012 21:03:20 -0500 Thu, 16 Feb 2012 15:25:21 -0600 2.0-developer-preview-4 2.0 2.0-developer-preview-4 couchbase-bucket 0 0 Same assert seemingly on current code. Here&#39;s backtrace from core dump. <br/> <br/> <br/> GNU gdb (GDB) 7.3-debian <br/> Copyright (C) 2011 Free Software Foundation, Inc. <br/> License GPLv3+: GNU GPL version 3 or later &lt;<a href="http://gnu.org/licenses/gpl.html">http://gnu.org/licenses/gpl.html</a>&gt; <br/> This is free software: you are free to change and redistribute it. <br/> There is NO WARRANTY, to the extent permitted by law. Type &quot;show copying&quot; <br/> and &quot;show warranty&quot; for details. <br/> This GDB was configured as &quot;i486-linux-gnu&quot;. <br/> For bug reporting instructions, please see: <br/> &lt;<a href="http://www.gnu.org/software/gdb/bugs/">http://www.gnu.org/software/gdb/bugs/</a>&gt;... <br/> Reading symbols from /root/src/altoros/moxi/repo20/install/bin/memcached.orig...done. <br/> [New LWP 4087] <br/> [New LWP 4077] <br/> [New LWP 4088] <br/> [New LWP 4078] <br/> [New LWP 4080] <br/> [New LWP 4086] <br/> [New LWP 4082] <br/> [New LWP 4083] <br/> [New LWP 13267] <br/> [New LWP 4079] <br/> [New LWP 4081] <br/> [New LWP 4075] <br/> <br/> warning: Can&#39;t read pathname for load map: Input/output error. <br/> [Thread debugging using libthread_db enabled] <br/> Core was generated by `/root/src/altoros/moxi/repo20/install/bin/memcached.orig -u root -X /root/src/a&#39;. <br/> Program terminated with signal 6, Aborted. <br/> #0 0xf76fe430 in __kernel_vsyscall () <br/> (gdb) thread apply all bt <br/> <br/> Thread 12 (Thread 0xf74e36c0 (LWP 4075)): <br/> #0 0xf76fe430 in __kernel_vsyscall () <br/> #1 0xf75b5b96 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 <br/> #2 0xf76b238c in ?? () from /usr/lib/libevent-2.0.so.5 <br/> #3 0xf769cc77 in event_base_loop () from /usr/lib/libevent-2.0.so.5 <br/> #4 0x0804c1b8 in main (argc=17, argv=0xffec82b4) at daemon/memcached.c:7544 <br/> <br/> Thread 11 (Thread 0xf54a9b70 (LWP 4081)): <br/> #0 0xf76fe430 in __kernel_vsyscall () <br/> #1 0xf75b5b96 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 <br/> #2 0xf76b238c in ?? () from /usr/lib/libevent-2.0.so.5 <br/> #3 0xf769cc77 in event_base_loop () from /usr/lib/libevent-2.0.so.5 <br/> #4 0x0805a675 in worker_libevent (arg=0x9ca6418) at daemon/thread.c:304 <br/> #5 0xf7647c39 in start_thread (arg=0xf54a9b70) at pthread_create.c:304 <br/> #6 0xf75b512e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 <br/> Backtrace stopped: Not enough registers or memory available to unwind further <br/> <br/> Thread 10 (Thread 0xf64abb70 (LWP 4079)): <br/> #0 0xf76fe430 in __kernel_vsyscall () <br/> #1 0xf75b5b96 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 <br/> #2 0xf76b238c in ?? () from /usr/lib/libevent-2.0.so.5 <br/> #3 0xf769cc77 in event_base_loop () from /usr/lib/libevent-2.0.so.5 <br/> #4 0x0805a675 in worker_libevent (arg=0x9ca62f8) at daemon/thread.c:304 <br/> #5 0xf7647c39 in start_thread (arg=0xf64abb70) at pthread_create.c:304 <br/> #6 0xf75b512e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 <br/> Backtrace stopped: Not enough registers or memory available to unwind further <br/> <br/> Thread 9 (Thread 0xe69ffb70 (LWP 13267)): <br/> #0 0xf76fe430 in __kernel_vsyscall () <br/> #1 0xf757fbc6 in nanosleep () at ../sysdeps/unix/syscall-template.S:82 <br/> #2 0xf75aee2c in usleep (useconds=1000) at ../sysdeps/unix/sysv/linux/usleep.c:33 <br/> #3 0xf39ba3d4 in Flusher::wait (this=0x9d2e190) at flusher.cc:59 <br/> #4 0xf3986a8b in stopFlusher (this=0x9d1e978) at ep.cc:748 <br/> #5 EventuallyPersistentStore::~EventuallyPersistentStore (this=0x9d1e978, __in_chrg=&lt;optimized out&gt;) at ep.cc:714 <br/> #6 0xf39b5144 in EventuallyPersistentEngine::~EventuallyPersistentEngine (this=0x9d15a48, __in_chrg=&lt;optimized out&gt;) at ep_engine.h:509 <br/> #7 0xf39aadf6 in EvpDestroy (handle=0x9d15a48, force=false) at ep_engine.cc:85 <br/> #8 0xf76ed981 in engine_shutdown_thread (arg=0x9cbfbe8) at bucket_engine.c:1422 <br/> #9 engine_shutdown_thread (arg=0x9cbfbe8) at bucket_engine.c:1396 <br/> #10 0xf7647c39 in start_thread (arg=0xe69ffb70) at pthread_create.c:304 <br/> #11 0xf75b512e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 <br/> Backtrace stopped: Not enough registers or memory available to unwind further <br/> <br/> Thread 8 (Thread 0xf44a7b70 (LWP 4083)): <br/> #0 0xf76fe430 in __kernel_vsyscall () <br/> #1 0xf75b5b96 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 <br/> #2 0xf76b238c in ?? () from /usr/lib/libevent-2.0.so.5 <br/> #3 0xf769cc77 in event_base_loop () from /usr/lib/libevent-2.0.so.5 <br/> #4 0x0805a675 in worker_libevent (arg=0x9ca6538) at daemon/thread.c:304 <br/> #5 0xf7647c39 in start_thread (arg=0xf44a7b70) at pthread_create.c:304 <br/> #6 0xf75b512e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 <br/> Backtrace stopped: Not enough registers or memory available to unwind further <br/> <br/> Thread 7 (Thread 0xf4ca8b70 (LWP 4082)): <br/> #0 0xf76fe430 in __kernel_vsyscall () <br/> #1 0xf75b5b96 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 <br/> #2 0xf76b238c in ?? () from /usr/lib/libevent-2.0.so.5 <br/> #3 0xf769cc77 in event_base_loop () from /usr/lib/libevent-2.0.so.5 <br/> ---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit--- <br/> #4 0x0805a675 in worker_libevent (arg=0x9ca64a8) at daemon/thread.c:304 <br/> #5 0xf7647c39 in start_thread (arg=0xf4ca8b70) at pthread_create.c:304 <br/> #6 0xf75b512e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 <br/> Backtrace stopped: Not enough registers or memory available to unwind further <br/> <br/> Thread 6 (Thread 0xf385db70 (LWP 4086)): <br/> #0 0xf76fe430 in __kernel_vsyscall () <br/> #1 0xf764c20a in __pthread_cond_wait (cond=0x9d2c944, mutex=0x9d2c924) at pthread_cond_wait.c:153 <br/> #2 0xf397f6be in wait (this=&lt;optimized out&gt;) at syncobject.hh:31 <br/> #3 Dispatcher::run (this=0x9d2c918) at dispatcher.cc:85 <br/> #4 0xf39814b6 in launch_dispatcher_thread (arg=0x9d2c918) at dispatcher.cc:28 <br/> #5 0xf7647c39 in start_thread (arg=0xf385db70) at pthread_create.c:304 <br/> #6 0xf75b512e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 <br/> Backtrace stopped: Not enough registers or memory available to unwind further <br/> <br/> Thread 5 (Thread 0xf5caab70 (LWP 4080)): <br/> #0 0xf76fe430 in __kernel_vsyscall () <br/> #1 0xf75b5b96 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 <br/> #2 0xf76b238c in ?? () from /usr/lib/libevent-2.0.so.5 <br/> #3 0xf769cc77 in event_base_loop () from /usr/lib/libevent-2.0.so.5 <br/> #4 0x0805a675 in worker_libevent (arg=0x9ca6388) at daemon/thread.c:304 <br/> #5 0xf7647c39 in start_thread (arg=0xf5caab70) at pthread_create.c:304 <br/> #6 0xf75b512e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 <br/> Backtrace stopped: Not enough registers or memory available to unwind further <br/> <br/> Thread 4 (Thread 0xf6cacb70 (LWP 4078)): <br/> #0 0xf76fe430 in __kernel_vsyscall () <br/> #1 0xf757fbc6 in nanosleep () at ../sysdeps/unix/syscall-template.S:82 <br/> #2 0xf757f9f0 in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:138 <br/> #3 0x0805cb00 in check_isasl_db_thread (arg=0x806a380) at daemon/isasl.c:233 <br/> #4 0xf7647c39 in start_thread (arg=0xf6cacb70) at pthread_create.c:304 <br/> #5 0xf75b512e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 <br/> Backtrace stopped: Not enough registers or memory available to unwind further <br/> <br/> Thread 3 (Thread 0xf285bb70 (LWP 4088)): <br/> #0 0xf76fe430 in __kernel_vsyscall () <br/> #1 0xf764c703 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:236 <br/> #2 0xf3979644 in wait (tv=..., this=0x9d2d570) at syncobject.hh:42 <br/> #3 IdleTask::run (this=0x9d2e110, d=...) at dispatcher.cc:244 <br/> #4 0xf397f0d0 in Dispatcher::run (this=0x9d2d568) at dispatcher.cc:119 <br/> #5 0xf39814b6 in launch_dispatcher_thread (arg=0x9d2d568) at dispatcher.cc:28 <br/> #6 0xf7647c39 in start_thread (arg=0xf285bb70) at pthread_create.c:304 <br/> #7 0xf75b512e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 <br/> Backtrace stopped: Not enough registers or memory available to unwind further <br/> <br/> Thread 2 (Thread 0xf74e2b70 (LWP 4077)): <br/> #0 0xf76fe430 in __kernel_vsyscall () <br/> #1 0xf75a5e3b in read () at ../sysdeps/unix/syscall-template.S:82 <br/> #2 0xf754f0cb in _IO_new_file_underflow (fp=0xf763e440) at fileops.c:606 <br/> #3 0xf755096b in _IO_default_uflow (fp=0xf763e440) at genops.c:440 <br/> #4 0xf7551d98 in *__GI___uflow (fp=0xf763e440) at genops.c:394 <br/> #5 0xf754767c in _IO_getc (fp=0xf763e440) at getc.c:41 <br/> #6 0xf76f671c in check_stdin_thread (arg=0x804cc60) at extensions/daemon/stdin_check.c:19 <br/> #7 0xf7647c39 in start_thread (arg=0xf74e2b70) at pthread_create.c:304 <br/> #8 0xf75b512e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 <br/> Backtrace stopped: Not enough registers or memory available to unwind further <br/> <br/> Thread 1 (Thread 0xf305cb70 (LWP 4087)): <br/> ---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit--- <br/> #0 0xf76fe430 in __kernel_vsyscall () <br/> #1 0xf750f941 in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 <br/> #2 0xf7512d72 in *__GI_abort () at abort.c:92 <br/> #3 0xf7508b58 in *__GI___assert_fail (assertion=0xf3ab3f37 &quot;false&quot;, file=0xf3ab9941 &quot;flusher.cc&quot;, line=276, function=0xf3ab9c20 &quot;bool Flusher::step(Dispatcher&amp;, TaskId)&quot;) at assert.c:81 <br/> #4 0xf39bcf7d in Flusher::step (this=0x9d2e190, tid=..., d=&lt;optimized out&gt;) at flusher.cc:276 <br/> #5 0xf39bde42 in FlusherStepper::callback (this=0xa3bc890, d=..., t=...) at flusher.cc:23 <br/> #6 0xf3981823 in Task::run (this=0xc4baee0, d=..., t=...) at dispatcher.hh:139 <br/> #7 0xf397f0d0 in Dispatcher::run (this=0x9d269d8) at dispatcher.cc:119 <br/> #8 0xf39814b6 in launch_dispatcher_thread (arg=0x9d269d8) at dispatcher.cc:28 <br/> #9 0xf7647c39 in start_thread (arg=0xf305cb70) at pthread_create.c:304 <br/> #10 0xf75b512e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 <br/> Backtrace stopped: Not enough registers or memory available to unwind further <br/> (gdb) ^CQuit <br/> (gdb) ^CQuit <br/> Trond, any update on this? Mike, another one to look at as Trond&#39;s busy on windows fires. Hi Mike, any news on this? <br/> Thanks, <br/> Steve This is very hard to track down due to the way we handle mutex errors. I added code to help track down the issue when it happens again. I don&#39;t think it makes sense to continue until then. Date of First Response Fri, 3 Feb 2012 18:35:40 -0600 Rank 5996 Time In Status [MB-4801] Unfriendly error message for builtin reduce function _stats http://www.couchbase.com/issues/browse/MB-4801 Couchbase Server <br/> Items ~ 5M. <br/> <br/> Query (with negative connection timeout) :- <br/> <a href="http://10.2.1.64:8092/default/_design/dev_temp/_view/temp?stale=false&connection_timeout=-1&limit=10&skip=0">http://10.2.1.64:8092/default/_design/dev_temp/_view/temp?stale=false&amp;connection_timeout=-1&amp;limit=10&amp;skip=0</a> <br/> <br/> Design doc:- <br/> {&quot;_id&quot;:&quot;_design/dev_temp&quot;,&quot;_rev&quot;:&quot;4-a2116f64&quot;,&quot;views&quot;:{&quot;temp&quot;:{&quot;map&quot;:&quot;function (doc) {\n emit(doc._id, [doc.achievements, doc.coins]);\n}&quot;,&quot;reduce&quot;:&quot;_stats&quot;}}} <br/> <br/> <br/> Error:- <br/> <br/> &quot;from&quot;: &quot;local&quot;, <br/> &quot;reason&quot;: &quot;{function_clause,\n [{couch_query_servers,builtin_stats,\n [reduce,\n [[[&lt;&lt;\&quot;01b7575c38dac42f\&quot;&gt;&gt;,&lt;&lt;\&quot;01b7575c38dac42f\&quot;&gt;&gt;],[\&quot;\\r%m�-I\&quot;,4.39]],\n [[&lt;&lt;\&quot;02c29fad5f164d62\&quot;&gt;&gt;,&lt;&lt;\&quot;02c29fad5f164d62\&quot;&gt;&gt;],\n [[3,94,134,62,146,176],7.06]],\n [[&lt;&lt;\&quot;039b9f5c48fb1bbe\&quot;&gt;&gt;,&lt;&lt;\&quot;039b9f5c48fb1bbe\&quot;&gt;&gt;],[\&quot;Oo�m\&quot;,9.23]],\n [[&lt;&lt;\&quot;06866f020cb80790\&quot;&gt;&gt;,&lt;&lt;\&quot;06866f020cb80790\&quot;&gt;&gt;],[\&quot;=�\&quot;,16.7]],\n [[&lt;&lt;\&quot;07b1e246e902a97f\&quot;&gt;&gt;,&lt;&lt;\&quot;07b1e246e902a97f\&quot;&gt;&gt;],\n [[76,157,157,179,14,239],19.69]],\n [[&lt;&lt;\&quot;094bc4aa468a6cb4\&quot;&gt;&gt;,&lt;&lt;\&quot;094bc4aa468a6cb4\&quot;&gt;&gt;],\n [[48,80,134,214,52,206],23.79]],\n [[&lt;&lt;\&quot;0a1e1d8da51f5dd7\&quot;&gt;&gt;,&lt;&lt;\&quot;0a1e1d8da51f5dd7\&quot;&gt;&gt;],\n [[62,142,187,197,91],25.9]],\n [[&lt;&lt;\&quot;0a797fd202caabdf\&quot;&gt;&gt;,&lt;&lt;\&quot;0a797fd202caabdf\&quot;&gt;&gt;],[\&quot; ..@�9�\&quot;,26.81]],\n [[&lt;&lt;\&quot;0a862b7f915e2f79\&quot;&gt;&gt;,&lt;&lt;\&quot;0a862b7f915e2f79\&quot;&gt;&gt;],\n [[54,126,135,185,58,156],26.94]],\n [[&lt;&lt;\&quot;0ac60fb446a982fc\&quot;&gt;&gt;,&lt;&lt;\&quot;0ac60fb446a982fc\&quot;&gt;&gt;],\n [[21,53,107,207,138],27.58]],\n [[&lt;&lt;\&quot;0b0b963ba25b9d38\&quot;&gt;&gt;,&lt;&lt;\&quot;0b0b963ba25b9d38\&quot;&gt;&gt;],\n [[5,85,103,153,51,93,213],28.27]],\n [[&lt;&lt;\&quot;0b96459020caf4e1\&quot;&gt;&gt;,&lt;&lt;\&quot;0b96459020caf4e1\&quot;&gt;&gt;],\n [[78,188,116,131],29.66]]]]},\n {couch_query_servers,builtin_reduce,4},\n {couch_query_servers,reduce,3},\n {couch_view_group,&#39;-init_group/4-fun-1-&#39;,4},\n {couch_btree,&#39;-write_node/3-lc$^0/1-0-&#39;,5},\n {couch_btree,write_node,3},\n {couch_btree,modify_node,8},\n {couch_btree,query_modify,6}]}&quot; <br/> } <br/> <br/> Attaching the diags 1 node cluster. <br/> Ubuntu 10.04 64 bit <br/> 2.0.0r-665-ga56327e MB-4801 Unfriendly error message for builtin reduce function _stats Bug Blocker Resolved Fixed Public Filipe Manana Karan Kumar Tue, 14 Feb 2012 15:06:27 -0600 Wed, 3 Apr 2013 07:47:37 -0500 Wed, 3 Apr 2013 07:47:37 -0500 2.0 2.0-developer-preview-4 view-engine 0 0 See the following error messages when querying with:- <br/> <br/> <a href="http://10.2.1.64:8092/default/_design/dev_temp/_view/temp?full_set=true&stale=false&connection_timeout=60000&limit=10&skip=0">http://10.2.1.64:8092/default/_design/dev_temp/_view/temp?full_set=true&amp;stale=false&amp;connection_timeout=60000&amp;limit=10&amp;skip=0</a> <br/> <br/> {&quot;error&quot;:&quot;{nocatch,{error,function_clause}}&quot;,&quot;reason&quot;:&quot;[{couch_set_view,get_group,3},\n {couch_set_view,get_reduce_view,5},\n {couch_view_merger,prepare_set_view,4},\n {couch_view_merger,reduce_set_view_folder,4}]&quot;} This does not seem related to connection_timeout at all. Was able to reproduce this with even default connection_timeout param. CouchDB&#39;s builtin reduce function _stats expects values emitted by the map function to be numbers. <br/> I&#39;ve updated in CouchDB the error handling, so for a 2 nodes cluster for e.g., you get: <br/> <br/> $ curl <a href="http://localhost:9500/default/_design/test/_view/view1?stale=false">http://localhost:9500/default/_design/test/_view/view1?stale=false</a> <br/> {&quot;rows&quot;:[ <br/> ], <br/> &quot;errors&quot;:[ <br/> {&quot;from&quot;:&quot;local&quot;,&quot;reason&quot;:&quot;builtin _stats function requires map values to be numbers&quot;}, <br/> {&quot;from&quot;:&quot;<a href="http://192.168.1.80:9501/_view_merge/">http://192.168.1.80:9501/_view_merge/</a>&quot;,&quot;reason&quot;:&quot;builtin _stats function requires map values to be numbers&quot;} <br/> ] <br/> } <br/> <br/> In <a href="http://review.couchbase.org/#change,13253">http://review.couchbase.org/#change,13253</a> Applied to branch-20dp4 and master:<br/> <br/> <a href="https://github.com/couchbase/couchdb/commit/015e9129aef5231c091ecc3514111bbfbee27dfc">https://github.com/couchbase/couchdb/commit/015e9129aef5231c091ecc3514111bbfbee27dfc</a> Date of First Response Wed, 15 Feb 2012 08:44:47 -0600 Rank 5670 Time In Status [MB-4773] Rebalance failing on latest 2.0 build due to wait_for_memcached failures (ns_rebalancer:wait_for_memcached) http://www.couchbase.com/issues/browse/MB-4773 Couchbase Server The latest test run, failed with rebalance failures. This is a regression from the previous test runs. No cores files generated. <br/> <br/> <a href="http://qa.hq.northscale.net/job/centos-64-view-tests/317/">http://qa.hq.northscale.net/job/centos-64-view-tests/317/</a> <br/> <br/> From the logs <br/> [rebalance:info] [2012-02-07 2:01:50] [<a href='mailto:ns_1@10.1.2.30'>ns_1@10.1.2.30</a>:&lt;0.4598.3&gt;:ns_rebalancer:wait_for_memcached:296] Waiting for [&#39;<a href='mailto:ns_1@10.1.2.30'>ns_1@10.1.2.30</a>&#39;] <br/> [user:info] [2012-02-07 2:01:50] [<a href='mailto:ns_1@10.1.2.30'>ns_1@10.1.2.30</a>:&lt;0.317.0&gt;:ns_orchestrator:handle_info:234] Rebalance exited with reason {wait_for_memcached_failed, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;default&quot;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&#39;<a href='mailto:ns_1@10.1.2.30'>ns_1@10.1.2.30</a>&#39;]} <br/> <br/> [ns_server:info] [2012-02-07 2:01:50] [<a href='mailto:ns_1@10.1.2.30'>ns_1@10.1.2.30</a>:ns_log:ns_log:handle_cast:115] suppressing duplicate log ns_orchestrator:2(&quot;Rebalance exited with reason {wait_for_memcached_failed,\&quot;default\&quot;,\n [&#39;<a href='mailto:ns_1@10.1.2.30'>ns_1@10.1.2.30</a>&#39;]}\n&quot;) because it&#39;s been seen 1 times in the past 254.737081 secs (last seen 254.737081 secs ago <br/> [ns_server:info] [2012-02-07 2:01:50] [<a href='mailto:ns_1@10.1.2.30'>ns_1@10.1.2.30</a>:ns_config_events:ns_config_log:handle_event:60] config change: <br/> counters -&gt; <br/> [{rebalance_fail,2},{rebalance_start,11},{rebalance_success,9}] <br/> [ns_server:info] [2012-02-07 2:01:50] [<a href='mailto:ns_1@10.1.2.30'>ns_1@10.1.2.30</a>:ns_config_events:ns_node_disco_conf_events:handle_event:56] ns_node_disco_conf_events config all <br/> [ns_server:info] [2012-02-07 2:01:50] [<a href='mailto:ns_1@10.1.2.30'>ns_1@10.1.2.30</a>:ns_config_rep:ns_config_rep:handle_info:181] Pushing config <br/> [ns_server:info] [2012-02-07 2:01:50] [<a href='mailto:ns_1@10.1.2.30'>ns_1@10.1.2.30</a>:ns_config_rep:ns_config_rep:handle_info:183] Pushing config done <br/> [ns_server:info] [2012-02-07 2:01:50] [<a href='mailto:ns_1@10.1.2.30'>ns_1@10.1.2.30</a>:ns_config_events:ns_config_log:handle_event:60] config change: <br/> rebalance_status -&gt; <br/> {none,&lt;&lt;&quot;Rebalance failed. See logs for detailed reason. You can try rebalance again.&quot;&gt;&gt;} <br/> <br/> Centos 64 bit. 6 node cluster <br/> <br/> 2.0.0r-651-g02c3154 MB-4773 Rebalance failing on latest 2.0 build due to wait_for_memcached failures (ns_rebalancer:wait_for_memcached) Bug Blocker Resolved Fixed Public Filipe Manana Karan Kumar Tue, 7 Feb 2012 11:38:59 -0600 Wed, 3 Apr 2013 07:47:39 -0500 Wed, 3 Apr 2013 07:47:39 -0500 2.0 2.0-developer-preview-4 bucket-engine 0 0 Damien, this looks exactly like shutdown race we&#39;ve discussed yesterday <br/> <br/> ns_memcached is stuck deleting one of databases during bucket deletion. <br/> <br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&lt;0.20901.2&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{registered_name,[]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{status,waiting}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{initial_call,{proc_lib,init_p,5}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{backtrace, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&lt;&lt;&quot;Program counter: 0x00002b89521a8ef0 (gen:do_call/4 + 576)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;CP: 0x0000000000000000 (invalid)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;arity = 0&quot;&gt;&gt;,&lt;&lt;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;0x00002aaaaf7bd1a0 Return addr 0x00002b8952226498 (gen_server:call/3 + 128)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(0) #Ref&lt;0.0.101.52421&gt;&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(1) &#39;<a href='mailto:ns_1@10.1.2.30'>ns_1@10.1.2.30</a>&#39;&quot;&gt;&gt;,&lt;&lt;&quot;y(2) []&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(3) infinity&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(4) {delete,&lt;&lt;10 bytes&gt;&gt;,[]}&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(5) &#39;$gen_call&#39;&quot;&gt;&gt;,&lt;&lt;&quot;y(6) &lt;0.161.0&gt;&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;0x00002aaaaf7bd1e0 Return addr 0x00002aaaaea06290 (ns_storage_conf:delete_database/1 + 72)&quot;&gt;&gt;, <br/> <br/> it is part of 10.1.2.30&#39;s diag <br/> <br/> then look at <br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&lt;0.28202.2&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{registered_name,[]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{status,waiting}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{initial_call,{proc_lib,init_p,5}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{backtrace, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&lt;&lt;&quot;Program counter: 0x00002aaaac4c7a18 (couch_util:shutdown_sync/1 + 352)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;CP: 0x0000000000000000 (invalid)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;arity = 0&quot;&gt;&gt;,&lt;&lt;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;0x00002aaaae939a08 Return addr 0x00002aaaadf092e0 (couch_db:terminate/2 + 160)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(0) #Ref&lt;0.0.101.52424&gt;&quot;&gt;&gt;,&lt;&lt;&quot;y(1) []&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(2) Catch 0x00002aaaac4c7b30 (couch_util:shutdown_sync/1 + 632)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(3) []&quot;&gt;&gt;,&lt;&lt;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;0x00002aaaae939a30 Return addr 0x00002b895222c788 (gen_server:terminate/6 + 184)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;0x00002aaaae939a38 Return addr 0x00002b89521adfe8 (proc_lib:init_p_do_apply/3 + 56)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(0) []&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;(1) {db,&lt;0.28202.2&gt;,&lt;0.28203.2&gt;,&lt;0.32369.2&gt;,&lt;&lt;16 bytes&gt;&gt;,&lt;0.28197.2&gt;,&lt;0.28204.2&gt;,{db_h&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(2) couch_db&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(3) {&#39;EXIT&#39;,&lt;0.161.0&gt;,shutdown}&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(4) &lt;0.28202.2&gt;&quot;&gt;&gt;,&lt;&lt;&quot;y(5) shutdown&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(6) Catch 0x00002b895222c788 (gen_server:terminate/6 + 184)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;0x00002aaaae939a78 Return addr 0x000000000088e318 (&lt;terminate process normally&gt;)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(0) Catch 0x00002b89521ae008 (proc_lib:init_p_do_apply/3 + 88)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&gt;&gt;]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{error_handler,error_handler}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{garbage_collection, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{min_bin_vheap_size,46368}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{min_heap_size,233}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{fullsweep_after,0}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{minor_gcs,0}]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{heap_size,987}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{total_heap_size,987}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{links,[]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{memory,9184}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{message_queue_len,1}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{reductions,2868}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{trap_exit,true}]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{&lt;0.28203.2&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{registered_name,[]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{status,waiting}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{initial_call,{proc_lib,init_p,5}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{backtrace, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[&lt;&lt;&quot;Program counter: 0x00002b89521a8ef0 (gen:do_call/4 + 576)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;CP: 0x0000000000000000 (invalid)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;arity = 0&quot;&gt;&gt;,&lt;&lt;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;0x00002aaaabcda110 Return addr 0x00002b8952226498 (gen_server:call/3 + 128)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(0) #Ref&lt;0.0.101.52434&gt;&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(1) &#39;<a href='mailto:ns_1@10.1.2.30'>ns_1@10.1.2.30</a>&#39;&quot;&gt;&gt;,&lt;&lt;&quot;y(2) []&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(3) infinity&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;(4) {db_updated,{db,&lt;0.28202.2&gt;,&lt;0.28203.2&gt;,nil,&lt;&lt;16 bytes&gt;&gt;,&lt;0.32412.2&gt;,&lt;0.32420.2&gt;,{&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(5) &#39;$gen_call&#39;&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(6) &lt;0.28202.2&gt;&quot;&gt;&gt;,&lt;&lt;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;0x00002aaaabcda150 Return addr 0x00002aaaadf0fb50 (couch_db_updater:handle_call/3 + 5368)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(0) infinity&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;(1) {db_updated,{db,&lt;0.28202.2&gt;,&lt;0.28203.2&gt;,nil,&lt;&lt;16 bytes&gt;&gt;,&lt;0.32412.2&gt;,&lt;0.32420.2&gt;,{&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(2) &lt;0.28202.2&gt;&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(3) Catch 0x00002b8952226498 (gen_server:call/3 + 128)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;0x00002aaaabcda178 Return addr 0x00002b895222a618 (gen_server:handle_msg/5 + 272)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(0) []&quot;&gt;&gt;,&lt;&lt;&quot;y(1) []&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(2) &lt;0.32412.2&gt;&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;(3) {db,&lt;0.28202.2&gt;,&lt;0.28203.2&gt;,&lt;0.32369.2&gt;,&lt;&lt;16 bytes&gt;&gt;,&lt;0.28197.2&gt;,&lt;0.28204.2&gt;,{db_h&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(4) []&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(5) \&quot;/opt/couchbase/var/lib/couchdb/default/27.couch.1\&quot;&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(6) &lt;&lt;10 bytes&gt;&gt;&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(7) \&quot;/opt/couchbase/var/lib/couchdb/default/27.couch.2\&quot;&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(8) []&quot;&gt;&gt;,&lt;&lt;&quot;y(9) &lt;0.28197.2&gt;&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;(10) {db,&lt;0.28202.2&gt;,&lt;0.28203.2&gt;,nil,&lt;&lt;16 bytes&gt;&gt;,&lt;0.32412.2&gt;,&lt;0.32420.2&gt;,{db_header,8,&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(11) []&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(12) \&quot;/opt/couchbase/var/lib/couchdb\&quot;&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;0x00002aaaabcda1e8 Return addr 0x00002b89521adfe8 (proc_lib:init_p_do_apply/3 + 56)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(0) couch_db_updater&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;(1) {db,&lt;0.28202.2&gt;,&lt;0.28203.2&gt;,&lt;0.32369.2&gt;,&lt;&lt;16 bytes&gt;&gt;,&lt;0.28197.2&gt;,&lt;0.28204.2&gt;,{db_h&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(2) &lt;0.28203.2&gt;&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(3) &lt;0.28202.2&gt;&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(4) {compact_done,\&quot;/opt/couchbase/var/lib/couchdb/default/27.couch.1.compact\&quot;}&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(5) {&lt;0.32369.2&gt;,#Ref&lt;0.0.101.52370&gt;}&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(6) Catch 0x00002b895222a618 (gen_server:handle_msg/5 + 272)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;0x00002aaaabcda228 Return addr 0x000000000088e318 (&lt;terminate process normally&gt;)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;y(0) Catch 0x00002b89521ae008 (proc_lib:init_p_do_apply/3 + 88)&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&gt;&gt;]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{error_handler,error_handler}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{garbage_collection, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{min_bin_vheap_size,46368}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{min_heap_size,233}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{fullsweep_after,0}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{minor_gcs,0}]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{heap_size,1597}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{total_heap_size,1597}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{links,[&lt;0.32369.2&gt;]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{memory,14104}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{message_queue_len,1}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{reductions,12954}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{trap_exit,true}]}, <br/> <br/> <br/> changing to blocker because it&#39;s deadlock Filipe fixed it by this: <a href="http://review.couchbase.org/13032">http://review.couchbase.org/13032</a> Date of First Response Tue, 7 Feb 2012 12:15:44 -0600 Rank 5696 Time In Status [MB-4401] cli commands including cbstats and couchbase-cli don't work in Mac OSX http://www.couchbase.com/issues/browse/MB-4401 Couchbase Server #! /bin/sh <br/> prefix=/Users/dustin/work/dev/membase/171/install <br/> exec_prefix=${prefix} <br/> root=${exec_prefix}/lib/python <br/> <br/> if test -z &quot;${PYTHONPATH}&quot;; then <br/> &nbsp;&nbsp;&nbsp;PYTHONPATH=$root <br/> else <br/> &nbsp;&nbsp;&nbsp;PYTHONPATH=$root:${PYTHONPATH} <br/> fi <br/> export PYTHONPATH <br/> exec $root/`basename $0` &quot;$@&quot; Mac OSX 1.7.1 MB-4401 cli commands including cbstats and couchbase-cli don't work in Mac OSX Bug Blocker Resolved Fixed Public Unassigned Mike Wiederhold Wed, 2 Nov 2011 19:18:59 -0500 Sun, 18 Nov 2012 20:13:21 -0600 Mon, 4 Jun 2012 02:36:50 -0500 1.7.2 1.8.0 1.8.1 2.0-developer-preview-3 recent-builds-2.0 2.0 installer 0 1 the path needs to be fixed before creating the dmg <br/> <br/> right now the path is hardcoded in all the scripts as mentioned above Is this really fixed? Prefix is set to &quot;../&quot; but it still doesn&#39;t work: <br/> <br/> bin/couchbase-cli: line 13: /Applications/Couchbase Server.app/Contents/Resources/couchbase-core/..//lib/python/couchbase-cli: No such file or directory <br/> bin/couchbase-cli: line 13: exec: /Applications/Couchbase Server.app/Contents/Resources/couchbase-core/..//lib/python/couchbase-cli: cannot execute: No such file or directory <br/> <br/> Setting prefix to &quot;/Applications/Couchbase Server.app/Contents/Resources/couchbase-core&quot; fixes it. Looks like this was fixed for 2.0. I just downloaded 1.8.1 and verified that this is an issue, but it looks like things function okay in the 2.0 recent builds Date of First Response Fri, 6 Jan 2012 08:39:23 -0600 Rank 6046 Time In Status [MB-5794] View parser returns JSON strings unescaped http://www.couchbase.com/issues/browse/MB-5794 Couchbase Server Document editing is using the following url <a href="http://lh:9000/couchBase/default/_all_docs?skip=0&include_docs=true&limit=5&_=1341509643238">http://lh:9000/couchBase/default/_all_docs?skip=0&amp;include_docs=true&amp;limit=5&amp;_=1341509643238</a> <br/> <br/> it normally works except Steve has broken it on my box by uploading this: <br/> <br/> <br/> # wget -O- -q &#39;<a href="http://lh:9000/couchBase/default/00000001&#39;">http://lh:9000/couchBase/default/00000001&amp;#39;</a> <br/> {&quot;_id&quot;:&quot;00000001&quot;,&quot;_rev&quot;:&quot;2-00027be31890c78a0000000000000000&quot;,&quot;name&quot;:&quot;steve \&quot;yen\&quot;&quot;,&quot;job&quot;:1,&quot;$expiration&quot;:0,&quot;$flags&quot;:0} <br/> <br/> And those quotes break all docs output massively: <br/> wget -O- -q &#39;<a href="http://lh:9000/couchBase/default/_all_docs?skip=0&include_docs=true&limit=5&_=1341509643238&#39;">http://lh:9000/couchBase/default/_all_docs?skip=0&amp;include_docs=true&amp;limit=5&amp;_=1341509643238&amp;#39;</a> <br/> {&quot;total_rows&quot;:998000,&quot;rows&quot;:[ <br/> {&quot;id&quot;:&quot;00000001&quot;,&quot;key&quot;:&quot;00000001&quot;,&quot;value&quot;:{&quot;rev&quot;:&quot;2-00027be31890c78a0000000000000000&quot;},&quot;doc&quot;:{&quot;_id&quot;:&quot;00000001&quot;,&quot;_rev&quot;:&quot;2-00027be31890c78a0000000000000000&quot;,&quot;name&quot;:&quot;steve &quot;yen&quot;&quot;,&quot;job&quot;:1,&quot;$expiration&quot;:0,&quot;$flags&quot;:0}}, <br/> {&quot;id&quot;:&quot;00000002&quot;,&quot;key&quot;:&quot;00000002&quot;,&quot;value&quot;:{&quot;rev&quot;:&quot;1-00027b84bccda3f10000000000000000&quot;},&quot;doc&quot;:{&quot;_id&quot;:&quot;00000002&quot;,&quot;_rev&quot;:&quot;1-00027b84bccda3f10000000000000000&quot;,&quot;$expiration&quot;:0,&quot;$flags&quot;:0}}, <br/> {&quot;id&quot;:&quot;00000004&quot;,&quot;key&quot;:&quot;00000004&quot;,&quot;value&quot;:{&quot;rev&quot;:&quot;1-00027b84bcc993a90000000000000000&quot;},&quot;doc&quot;:{&quot;_id&quot;:&quot;00000004&quot;,&quot;_rev&quot;:&quot;1-00027b84bcc993a90000000000000000&quot;,&quot;$expiration&quot;:0,&quot;$flags&quot;:0}}, <br/> {&quot;id&quot;:&quot;00000006&quot;,&quot;key&quot;:&quot;00000006&quot;,&quot;value&quot;:{&quot;rev&quot;:&quot;1-00027b84bcc9f7420000000000000000&quot;},&quot;doc&quot;:{&quot;_id&quot;:&quot;00000006&quot;,&quot;_rev&quot;:&quot;1-00027b84bcc9f7420000000000000000&quot;,&quot;$expiration&quot;:0,&quot;$flags&quot;:0}}, <br/> {&quot;id&quot;:&quot;00000007&quot;,&quot;key&quot;:&quot;00000007&quot;,&quot;value&quot;:{&quot;rev&quot;:&quot;1-00027b84bccc52730000000000000000&quot;},&quot;doc&quot;:{&quot;_id&quot;:&quot;00000007&quot;,&quot;_rev&quot;:&quot;1-00027b84bccc52730000000000000000&quot;,&quot;$expiration&quot;:0,&quot;$flags&quot;:0}} <br/> ] <br/> } <br/> <br/> As you can see quotes inside steve &quot;yen&quot; are not escaped properly <br/> MB-5794 View parser returns JSON strings unescaped Bug Blocker Resolved Fixed Public Filipe Manana Aleksey Kondratenko Thu, 5 Jul 2012 12:39:03 -0500 Wed, 18 Jul 2012 11:24:12 -0500 Fri, 6 Jul 2012 10:50:51 -0500 recent-builds-2.0 recent-builds-2.0 view-engine 0 2 Verified, thanks for this excellent find. <br/> <br/> It turns out that YAJL&#39;s stream parser unescapes strings before passing them to the callback. <br/> The only solution (inefficient) is call YAJL&#39;s helpers to explicitly escape the string again. <br/> <br/> Just sent a patch upstream to make it not unescape the string, and avoiding re-escaping it: <br/> <br/> <a href="https://github.com/lloyd/yajl/pull/81">https://github.com/lloyd/yajl/pull/81</a> <br/> <br/> Will close this soon. Fix merged to master.<br/> <a href="http://review.couchbase.org/#change,17979">http://review.couchbase.org/#change,17979</a> Integrated in github-couchdb-preview #441 (See [<a href="http://qa.hq.northscale.net/job/github-couchdb-preview/441/">http://qa.hq.northscale.net/job/github-couchdb-preview/441/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.couchbase.com/issues/browse/MB-5794" title="View parser returns JSON strings unescaped"><strike>MB-5794</strike></a> View parser didn&#39;t preserve escaping in strings (Revision 588f33dc74366d0e601fed50e705cae788b0c098)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> Filipe David Borba Manana : <br/> Files : <br/> * src/couch_view_parser/yajl/yajl_parse.h<br/> * src/couch_view_parser/yajl/yajl_parser.c<br/> * src/couch_view_parser/yajl/yajl.c<br/> * src/couch_view_parser/test/01-map-view.t<br/> * src/couch_view_parser/couch_view_parser.cc<br/> Date of First Response Fri, 6 Jul 2012 09:45:32 -0500 Rank 4753 Story Points 0.5 Time In Status [MB-6239] membase-cli breaks due to missing timeout module http://www.couchbase.com/issues/browse/MB-6239 Couchbase Server ________________________________________ <br/> From: Sharon Barr <br/> Sent: Wednesday, August 15, 2012 1:55 PM <br/> To: Steve Yen; Farshid Ghods <br/> Cc: Peter Wansch <br/> Subject: build 1588 clustering is broken on rightscale <br/> <br/> I am afraid that this is just a symptom. <br/> Clustering on my right scale templates stopped working on build 1588. It worked on 1579. Here are the logs: <br/> <br/> What&#39;s the source of it? <br/> <br/> Sharon <br/> <br/> <br/> Build 1579 works well. Here are the logs of the clustering phase: <br/> <br/> ******************************************************************************** <br/> *RS&gt; RightScript: &#39;DEV CBS 2.0.x - Clustering&#39; **** <br/> 20:21:54: MB_CLI_PATH in use is: /opt/couchbase/bin/couchbase-cli <br/> 20:22:04: Running Clustering Script <br/> 20:22:04: Attempting cluster-init: MEMBASE CLI=/opt/couchbase/bin/couchbase-cli <br/> 20:22:04: Attempted to cluster-init: SUCCESS: init 127.0.0.1 <br/> 20:22:06: Setting my own tags: <br/> 20:22:07: Request to add tag &#39;membase_cluster_tag:steve01=20120815202206-10.176.30.201&#39; sent successfully. <br/> 20:22:07: Checking whether I am clustered already: <br/> 20:22:07: Searching for other members of cluster: steve01 <br/> 20:22:07: Attempting to join node:10.177.2.38 <br/> 20:22:09: Joined <br/> 20:22:09: Checking whether we want to rebalance... <br/> 20:22:09: Skipping rebalance since there are only 2 nodes in the cluster <br/> 20:22:09: Script exit status: 0 <br/> 20:22:09: Script duration: 16.145578 <br/> ******************************************************************************** <br/> <br/> <br/> On a medium VM. <br/> Build 1588 failes to cluster nodes. Here are the logs: <br/> ******************************************************************************** <br/> *RS&gt; RightScript: &#39;DEV CBS 2.0.x - Clustering&#39; **** <br/> 20:36:50: MB_CLI_PATH in use is: /opt/couchbase/bin/couchbase-cli <br/> 20:37:00: Running Clustering Script <br/> 20:37:00: Attempting cluster-init: MEMBASE CLI=/opt/couchbase/bin/couchbase-cli <br/> 20:37:01: Traceback (most recent call last): <br/> 20:37:01: File &quot;/opt/couchbase/lib/python/couchbase-cli&quot;, line 12, in ? <br/> 20:37:01: import buckets <br/> 20:37:01: File &quot;/opt/couchbase/lib/python/buckets.py&quot;, line 7, in ? <br/> 20:37:01: from timeout import timed_out <br/> 20:37:01: ImportError <br/> 20:37:01: : <br/> 20:37:01: No module named timeout <br/> 20:37:01: Attempted to cluster-init: <br/> 20:37:11: Running Clustering Script <br/> 20:37:11: Attempting cluster-init: MEMBASE CLI=/opt/couchbase/bin/couchbase-cli <br/> 20:37:11: Traceback (most recent call last): <br/> 20:37:11: File &quot;/opt/couchbase/lib/python/couchbase-cli&quot;, line 12, in ? <br/> 20:37:11: import buckets <br/> 20:37:11: File &quot;/opt/couchbase/lib/python/buckets.py&quot;, line 7, in ? <br/> 20:37:11: from timeout import timed_out <br/> 20:37:11: ImportError <br/> 20:37:11: : <br/> 20:37:11: No module named timeout <br/> 20:37:11: Attempted to cluster-init: <br/> 20:37:21: Running Clustering Script <br/> 20:37:21: Attempting cluster-init: MEMBASE CLI=/opt/couchbase/bin/couchbase-cli <br/> 20:37:21: Traceback (most recent call last): <br/> 20:37:21: File &quot;/opt/couchbase/lib/python/couchbase-cli&quot;, line 12, in ? <br/> 20:37:21: import buckets <br/> 20:37:21: File &quot;/opt/couchbase/lib/python/buckets.py&quot;, line 7, in ? <br/> 20:37:21: from timeout import timed_out <br/> 20:37:21: ImportError <br/> 20:37:21: : <br/> 20:37:21: No module named timeout <br/> 20:37:21: Attempted to cluster-init: <br/> 20:37:31: Running Clustering Script <br/> 20:37:31: Attempting cluster-init: MEMBASE CLI=/opt/couchbase/bin/couchbase-cli <br/> 20:37:31: Traceback (most recent call last): <br/> 20:37:31: File &quot;/opt/couchbase/lib/python/couchbase-cli&quot;, line 12, in ? <br/> 20:37:31: import buckets <br/> 20:37:31: File &quot;/opt/couchbase/lib/python/buckets.py&quot;, line 7, in ? <br/> 20:37:31: from timeout import timed_out <br/> 20:37:31: ImportError <br/> 20:37:31: : <br/> 20:37:31: No module named timeout <br/> 20:37:31: Attempted to cluster-init: <br/> 20:37:41: Running Clustering Script <br/> 20:37:41: Attempting cluster-init: MEMBASE CLI=/opt/couchbase/bin/couchbase-cli <br/> 20:37:41: Traceback (most recent call last): <br/> 20:37:41: File &quot;/opt/couchbase/lib/python/couchbase-cli&quot;, line 12, in ? <br/> 20:37:41: import buckets <br/> 20:37:41: File &quot;/opt/couchbase/lib/python/buckets.py&quot;, line 7, in ? <br/> 20:37:41: from timeout import timed_out <br/> 20:37:41: ImportError <br/> 20:37:41: : <br/> 20:37:41: No module named timeout <br/> 20:37:41: Attempted to cluster-init: <br/> 20:37:41: Failed to set username/pass: <br/> 20:37:41: Script exit status: 255 <br/> 20:37:41: Script duration: 51.576187 <br/> *ERROR&gt; Chef converge failed <br/> *ERROR&gt; An external command returned an error during the execution of Chef: <br/> <br/> RightScript &lt; DEV CBS 2.0.x - Clustering &gt; returned 255 <br/> *ERROR&gt; Chef process failure <br/> *ERROR&gt; Chef process failed with return code 1 <br/> *RS&gt; boot failed: SYS Timezone set v1, SYS Monitoring install v9, SYS Lvm <br/> *RS&gt; install - 11H1, SYS Add Swap Partition, RB rubygems 1.3.1 + quickinstall <br/> *RS&gt; v4, EBS stripe Rightscale tools install v2, Membase Server - EBS stripe <br/> *RS&gt; volume create v2, Membase Server - EBS stripe volume restore v2, Membase <br/> *RS&gt; Server - EBS create backup scripts v2, Membase Server - EBS continuous <br/> *RS&gt; backups v2, Membase Server - enable login, DEV CBS 2.0.x - Install, DEV CBS <br/> *RS&gt; 2.0.x - EBS Setup, DEV CBS 2.0.x - DNS Setup, Couchbase Server - Enterprise <br/> *RS&gt; Edition Support Setup, DEV CBS 2.0.x - Clustering, DEV CBS 2.0.x - <br/> *RS&gt; Configure, Membase Server - Delay Decommission <br/> *ERROR&gt; Failed to run boot sequence: Failed to run boot bundle <br/> MB-6239 membase-cli breaks due to missing timeout module Bug Blocker Resolved Fixed Public Pavel Paulau Steve Yen Wed, 15 Aug 2012 16:01:34 -0500 Thu, 16 Aug 2012 17:17:00 -0500 Thu, 16 Aug 2012 00:06:50 -0500 2.0-beta recent-builds-2.0 tools 0 0 <a href="http://review.couchbase.org/#change,19689">http://review.couchbase.org/#change,19689</a> Date of First Response Wed, 15 Aug 2012 23:58:53 -0500 Rank 4344 Time In Status [MB-5607] docloader fails to run at all in 2.0 builds leading to DP5 http://www.couchbase.com/issues/browse/MB-5607 Couchbase Server Running the docloader without any arguments I expected to get a usage or something nice. Instead, I received: <br/> <br/> # /opt/couchbase/bin/docloader <br/> &nbsp;&nbsp;File &quot;/opt/couchbase/lib/python/docloader&quot;, line 90 <br/> &nbsp;&nbsp;&nbsp;&nbsp;with open(os.path.join(basedir, item), &#39;r&#39;) as fp: <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;^ <br/> SyntaxError: invalid syntax <br/> Version: 2.0.0 community edition (build-1269) <br/> CentOS 5.x, 64-bit MB-5607 docloader fails to run at all in 2.0 builds leading to DP5 Bug Blocker Resolved Fixed Public Pavel Paulau Matt Ingenthron Tue, 19 Jun 2012 11:29:12 -0500 Mon, 23 Jul 2012 14:50:50 -0500 Fri, 6 Jul 2012 17:58:46 -0500 2.0-beta recent-builds-2.0 2.0-beta tools 0 1 It appears the syntax is valid with python26 after installing that package from EPEL, but then I run into missing dependencies: <br/> <br/> # python26 /opt/couchbase/lib/python/docloader -u Administrator -p password -n 127.0.0.1:8091 -b beer-sample -s 256 /root/beer-sample.zip <br/> Traceback (most recent call last): <br/> &nbsp;&nbsp;File &quot;/opt/couchbase/lib/python/docloader&quot;, line 10, in &lt;module&gt; <br/> &nbsp;&nbsp;&nbsp;&nbsp;from couchbase import client <br/> &nbsp;&nbsp;File &quot;/opt/couchbase/lib/python/couchbase/__init__.py&quot;, line 1, in &lt;module&gt; <br/> &nbsp;&nbsp;&nbsp;&nbsp;from couchbase.client import Server, Bucket <br/> &nbsp;&nbsp;File &quot;/opt/couchbase/lib/python/couchbase/client.py&quot;, line 30, in &lt;module&gt; <br/> &nbsp;&nbsp;&nbsp;&nbsp;from rest_client import RestConnection <br/> &nbsp;&nbsp;File &quot;/opt/couchbase/lib/python/couchbase/rest_client.py&quot;, line 24, in &lt;module&gt; <br/> &nbsp;&nbsp;&nbsp;&nbsp;import httplib2 <br/> ImportError: No module named httplib2 <br/> After installation of python 2.6 and adding httplib2 manually, I was able to run the docloader with the beer-sample.zip as described in <a href="http://www.couchbase.com/issues/browse/MB-4989" title="using views after loading data using docloader with sample data ( beer-sample.zip ) on dp4 Mac OSX fails with &quot;from&quot;: &quot;local&quot;, &quot;reason&quot;: &quot;epipe&quot; error"><strike>MB-4989</strike></a>. However, it takes 2m8s of runtime and still outputs debug messages to the console that I added back in September/October of last year. Bumping to critical, as this blocks the UI from installing the current gamesim sample at install time. Selecting the sample database at configuration time leads to &quot;There was an unexpected error&quot;. I faced other dependency related issue: <br/> <br/> Traceback (most recent call last): <br/> &nbsp;&nbsp;File &quot;/opt/couchbase/lib/python/docloader&quot;, line 10, in &lt;module&gt; <br/> &nbsp;&nbsp;&nbsp;&nbsp;from couchbase import client <br/> &nbsp;&nbsp;File &quot;/opt/couchbase/lib/python/couchbase/__init__.py&quot;, line 1, in &lt;module&gt; <br/> &nbsp;&nbsp;&nbsp;&nbsp;from couchbase.client import Server, Bucket <br/> &nbsp;&nbsp;File &quot;/opt/couchbase/lib/python/couchbase/client.py&quot;, line 27, in &lt;module&gt; <br/> &nbsp;&nbsp;&nbsp;&nbsp;import requests <br/> <br/> It can be fixed either by installing this package or by adding it couchbase-python-client source lib. <br/> The second option is much better but may have problems with license. Or not? btw, what python version must be supported? python 2.4 if possible. The docloader code is here (in case you hadn&#39;t found it already)... <br/> <br/> &nbsp;&nbsp;<a href="https://github.com/couchbase/couchbase-examples">https://github.com/couchbase/couchbase-examples</a> <br/> Basically you can&#39;t install requests package with python 2.4 (this version is not supported). <br/> <br/> Can we add ISC licensed library to our sdk? It&#39;s possible, but we&#39;d need to review it. <br/> <br/> What is the package, and where is the license? Oh, and this may not be an issue for the SDK, but rather for Couchbase Server&#39;s fork of stuff to work. The SDK is not planning to support 2.4, so it wouldn&#39;t be added there. <a href="https://github.com/kennethreitz/requests">https://github.com/kennethreitz/requests</a> <br/> <a href="https://raw.github.com/kennethreitz/requests/develop/LICENSE">https://raw.github.com/kennethreitz/requests/develop/LICENSE</a> <br/> <br/> If you are not going to support python 2.4 then it would be very hard to support it in docloader which relies on sdk. Yes, I definitely understand that. As you&#39;re aware, there are discussions about how we&#39;ll approach common Python components. We should have resolution on that very soon, but it&#39;s not resolved yet. There are conflicting requirements at a product level, but I still (at the moment) believe they&#39;re resolvable. blocks testing <a href="http://www.couchbase.com/issues/browse/MB-4989">http://www.couchbase.com/issues/browse/MB-4989</a> <br/> Pavel, <br/> this version of python sdk works with python 2.4 <br/> <br/> can you try docloader with this revision ? <br/> &lt;project name=&quot;couchbase-python-client&quot; path=&quot;couchbase-python-client&quot; revision=&quot;couchbase-181&quot; <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;remote=&quot;couchbase&quot;/&gt; This version of Python SDK doesn&#39;t work with Python 2.4 because of this: <br/> <br/> <a href="https://github.com/couchbase/couchbase-python-client/blob/couchbase-181/couchbase/rest_client.py#L207">https://github.com/couchbase/couchbase-python-client/blob/couchbase-181/couchbase/rest_client.py#L207</a> <br/> <br/> However these changes may help: <br/> <a href="http://review.couchbase.org/#change,17924">http://review.couchbase.org/#change,17924</a> <br/> <a href="http://review.couchbase.org/#change,17925">http://review.couchbase.org/#change,17925</a> <br/> <a href="http://review.couchbase.org/#change,17929">http://review.couchbase.org/#change,17929</a> This also must be merged to branch &quot;couchbase-181&quot;: <br/> <a href="https://github.com/couchbase/couchbase-python-client/commit/b31ab60ff79601683d25c0c2feead75507060894">https://github.com/couchbase/couchbase-python-client/commit/b31ab60ff79601683d25c0c2feead75507060894</a> the fix to python sdk code you mentioned is one line change to use .join or % instead of .format <br/> <br/> you can merge that code to branch-1.8.1 and still have the doc loader run the view once <br/> <br/> the other three changes you posted look good to me so let&#39;s merge them before recent builds Done. <br/> <br/> <a href="http://review.couchbase.org/#change,17933">http://review.couchbase.org/#change,17933</a> <br/> <a href="http://review.couchbase.org/#change,17934">http://review.couchbase.org/#change,17934</a> Currently using sdk branch &quot;couchbase-181&quot; it works: <br/> -- via cli <br/> -- via setting menu <br/> <br/> It doesn&#39;t work before cluster initialization. <br/> <br/> @Farshid <br/> Is it ok? yes that works. Date of First Response Thu, 21 Jun 2012 21:16:42 -0500 Rank 4926 Time In Status [MB-5938] UI needs to deal with very slow docloader (alternatively, docloader is too slow) http://www.couchbase.com/issues/browse/MB-5938 Couchbase Server It works fine with old samples (gamesim-sample), but recent archives (beer-sample) contains much more data and usually fail due to timeout. <br/> <br/> Now it takes at least 40 seconds to load documents and create views. MB-5938 UI needs to deal with very slow docloader (alternatively, docloader is too slow) Bug Blocker Resolved Fixed Public Aleksey Kondratenko Pavel Paulau Wed, 18 Jul 2012 13:17:44 -0500 Fri, 26 Oct 2012 18:14:36 -0500 Mon, 17 Sep 2012 10:14:18 -0500 recent-builds-2.0 2.0-beta-2 ns_server 1 3 Hm. Users are not generally prepared to wait that much. In theory there should be no problem loading thousands of docs per second. I don&#39;t think beers data is mega docs. 1. before loading these docs we have to create new bucket (if it doesn&#39;t exist). <br/> 2. every document is a separate file (for beers we have more then 6000 files in archive). and current approach to read these files is pretty weird. <br/> 3. we also create new ddocs/views and perform queries on new indices. <br/> <br/> if there is any problem with increasing timeout, I can fix point #2. Increasing timeout is simple. But waiting 40 seconds is a bit too much for users to tolerate. <br/> ok, fixing #2 doesn&#39;t help sometimes. <br/> <br/> python client have extremely low throughput (regardless environment). and this is behind cbdocloader. <br/> thus we need ~15-20 seconds to load these sample items. <br/> <br/> I understand that 40 seconds are too long. but nobody is going to fix/improve old sdk branch, which is only compatible with python 2.4. Trying longer timeout. That sucks. We don&#39;t have good feedback for users. Can we just make the load asynchronous and log problems? Integrated in github-ns-server-2-0 #412 (See [<a href="http://qa.hq.northscale.net/job/github-ns-server-2-0/412/">http://qa.hq.northscale.net/job/github-ns-server-2-0/412/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.couchbase.com/issues/browse/MB-5938" title="UI needs to deal with very slow docloader (alternatively, docloader is too slow)"><strike>MB-5938</strike></a>: bumped up docloader timeout to 60 seconds (Revision f6d28497ddb2270962b8ba73284da66237120be8)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> Aliaksey Artamonau : <br/> Files : <br/> * src/menelaus_web.erl<br/> Assigning to Pavel for UI-only attempt to fix I had tried to POST on <a href="http://localhost:9001/sampleBuckets/install">http://localhost:9001/sampleBuckets/install</a> and after 2.8min of pending I got error 400 from server with the following response <br/> {&quot;error&quot;:true,&quot;type&quot;:&quot;docloader_failed&quot;,&quot;reason&quot;:&quot;There was an unexpected error.&quot;} <br/> <br/> In this case SAMPLES_LOADING_TIMEOUT was 12000000. There&#39;s blocker level bug on docloader for that Also look at <br/> <a href="http://www.couchbase.com/issues/browse/MB-6274">http://www.couchbase.com/issues/browse/MB-6274</a> <br/> <br/> for other docloader improvements. Not actually fixed. Sorry. Meant to assign to Pavel <a href="http://review.couchbase.org/20834">http://review.couchbase.org/20834</a> I&#39;ve uploaded alternative commit to gerrit. Sample loading call will now return as soon as buckets are created, then data will be loaded in background and task will be reported via list of tasks (without progress report however) in right top corner. Sample data will now be asynchronously loaded. Loading sample data will be <br/> executed as a background task then later reported in web console <br/> upon completion. Date of First Response Wed, 18 Jul 2012 15:08:50 -0500 Rank 4617 Sprint Priority 2.0 Sprint Status Time In Status [MB-1225] moxi semantics: upstream conns should be associated to a preferred downstream http://www.couchbase.com/issues/browse/MB-1225 Couchbase Server this allows for more performance without increasing the number of threads in moxi. <br/> <br/> the upstream to downstream assoication could be a simple hash. <br/> <br/> this allows moxi to still honor memcached semantics. <br/> <br/> but it more complex than just a config change to increase the number of threads that moxi uses. Operating System: All <br/> Platform: All MB-1225 moxi semantics: upstream conns should be associated to a preferred downstream Bug Blocker Resolved Won't Fix Unassigned Steve Yen Sat, 19 Jun 2010 16:33:32 -0500 Tue, 3 Jul 2012 15:46:02 -0500 Thu, 7 Jun 2012 17:55:23 -0500 1.6.0 beta4 2.0-beta moxi 0 0 A Pivotal Tracker story has been created for this Issue: <a href="http://www.pivotaltracker.com/story/show/27629589">http://www.pivotaltracker.com/story/show/27629589</a> Steve Yen deleted the linked story in Pivotal Tracker Duplicate MB-968 Date of First Response Thu, 5 Apr 2012 22:25:13 -0500 Rank 845 Time In Status [MB-5304] stats for key are not defined in documentation, many stats are not defined http://www.couchbase.com/issues/browse/MB-5304 Couchbase Server To work through the implementation of OBSERVE[1], we were looking for the details on STATS KEY, only to find it&#39;s not documented. It exists in the server implementation, but it&#39;s not even in the stats.org[2] associated with the code repository. <br/> <br/> This statistic and all others that are public interface should be documented in the manual. <br/> <br/> Implementation of OBSERVE operations in the various client libraries will need this information. <br/> <br/> 1. <a href="http://www.couchbase.com/wiki/display/couchbase/Observe">http://www.couchbase.com/wiki/display/couchbase/Observe</a> <br/> 2. <a href="https://github.com/couchbase/ep-engine/blob/master/docs/stats.org">https://github.com/couchbase/ep-engine/blob/master/docs/stats.org</a> MB-5304 stats for key are not defined in documentation, many stats are not defined Bug Blocker Resolved Fixed Public Mike Wiederhold Matt Ingenthron Tue, 15 May 2012 10:26:04 -0500 Tue, 3 Jul 2012 15:46:04 -0500 Mon, 4 Jun 2012 15:43:43 -0500 1.7.2 1.8.0 2.0-developer-preview-4 2.0-beta documentation 1 5 Chiyoung, can you provide details on stats key so that documentation can be added? Jin, <br/> <br/> We should provide the &quot;stats key&quot; documentation for the client library team, so that they can refer to this documentation to implement OBSERVE command in the client libraries. Assign it to Mike as he was mainly involved in the initial OBSERVE design. Integrated in github-ep-engine-2-0 #301 (See [<a href="http://qa.hq.northscale.net/job/github-ep-engine-2-0/301/">http://qa.hq.northscale.net/job/github-ep-engine-2-0/301/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.couchbase.com/issues/browse/MB-5304" title="stats for key are not defined in documentation, many stats are not defined"><strike>MB-5304</strike></a>: Document stats key/vkey in stats.org (Revision df4dd0f9aa65b69a36a304ca76a8389090bd3297)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> Mike Wiederhold : <br/> Files : <br/> * docs/stats.org<br/> Documentation mixes stats key and stats vkey, though the return values for each need to be understood. Also, see <a href="http://www.couchbase.com/issues/browse/MB-5435" title="stats key returns a stat as non-key value in violation of the protocol"><strike>MB-5435</strike></a>. This documentation appears to be wrong with respect to the key valid field, or there&#39;s some subtlety that is not explained by the table. Stats key and vkey return the same data. The only difference is the key_valid field returned by stats vkey. I have marked each stat with the command that returns it in the column on the left. I made a few minor wording changes to the documentation, but I disagree that the documentation is wrong. Please provide an explanation of what is confusing or contact me over Skype to discuss anything you think is incorrect. <br/> <br/> <br/> ** Stats Key and Vkey <br/> | key_cas | The keys current cas value |KV| <br/> | key_data_age | How long the key has waited for its |KV| <br/> | | value to be persisted (0 if clean) |KV| <br/> | key_exptime | Expiration time from the epoch |KV| <br/> | key_flags | Flags for this key |KV| <br/> | key_is_dirty | If the value is not yet persisted |KV| <br/> | key_last_modified_time | Last updated time |KV| <br/> | key_valid | See description below | V| <br/> | key_vb_state | The vbucket state of this key |KV| <br/> <br/> =key_valid= can have the following responses: <br/> <br/> this_is_a_bug - Some case we didn&#39;t take care of. <br/> dirty - The value in memory has not been persisted yet. <br/> length_mismatch - Data length in memory doesn&#39;t match the data length on disk. <br/> data_mismatch - The data in memory doesn&#39;t match the data on disk. <br/> flags_mismatch - The flags in memory don&#39;t match the flags on disk. <br/> valid - The data on disk and in memory match <br/> ram_but_not_disk - The value doesn&#39;t exist yet on disk. <br/> item_deleted - The item has been deleted. <br/> What is KV in the last column? I think it should be KV instead of V for key_valid K = Key <br/> V = VKey <br/> <br/> And key_valid is not included in stats key. What does mean &quot;is not included in stats&quot;? I can have key &quot;x&quot; with value &quot;test&quot; in the default bucket on couchbase 2.0. Here is the output of the command line tools: <br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;$ cbc cat x <br/> &nbsp;&nbsp;&nbsp;&nbsp;&quot;x&quot; Size 5 Flags 0x0 CAS 0x69cf70ff8de90000 <br/> &nbsp;&nbsp;&nbsp;&nbsp;test <br/> &nbsp;&nbsp;&nbsp;&nbsp;$ curl -s <a href="http://localhost:8091/pools/default/buckets/default">http://localhost:8091/pools/default/buckets/default</a> | vbuckettool - x <br/> &nbsp;&nbsp;&nbsp;&nbsp;key: x master: 127.0.0.1:11210 vBucketId: 220 couchApiBase: <a href="http://127.0.0.1:8092/default">http://127.0.0.1:8092/default</a> replicas: <br/> &nbsp;&nbsp;&nbsp;&nbsp;$ cbc stats &quot;key x 220&quot; <br/> &nbsp;&nbsp;&nbsp;&nbsp;127.0.0.1:11210 key_is_dirty 0 <br/> &nbsp;&nbsp;&nbsp;&nbsp;127.0.0.1:11210 key_exptime 0 <br/> &nbsp;&nbsp;&nbsp;&nbsp;127.0.0.1:11210 key_flags 0 <br/> &nbsp;&nbsp;&nbsp;&nbsp;127.0.0.1:11210 key_cas 256796085243753 <br/> &nbsp;&nbsp;&nbsp;&nbsp;127.0.0.1:11210 key_dirtied 0 <br/> &nbsp;&nbsp;&nbsp;&nbsp;127.0.0.1:11210 key_data_age 0 <br/> &nbsp;&nbsp;&nbsp;&nbsp;127.0.0.1:11210 key_last_modification_time 1338826758 <br/> &nbsp;&nbsp;&nbsp;&nbsp;$ cbc stats &quot;vkey x 220&quot; <br/> &nbsp;&nbsp;&nbsp;&nbsp;127.0.0.1:11210 key_is_dirty 0 <br/> &nbsp;&nbsp;&nbsp;&nbsp;127.0.0.1:11210 key_exptime 0 <br/> &nbsp;&nbsp;&nbsp;&nbsp;127.0.0.1:11210 key_flags 0 <br/> &nbsp;&nbsp;&nbsp;&nbsp;127.0.0.1:11210 key_cas 256796085243753 <br/> &nbsp;&nbsp;&nbsp;&nbsp;127.0.0.1:11210 key_dirtied 0 <br/> &nbsp;&nbsp;&nbsp;&nbsp;127.0.0.1:11210 key_data_age 0 <br/> &nbsp;&nbsp;&nbsp;&nbsp;127.0.0.1:11210 key_last_modification_time 1338826854 <br/> &nbsp;&nbsp;&nbsp;&nbsp;127.0.0.1:11210 key_valid valid <br/> <br/> You can see the last line says there is key &quot;key_valid&quot; and value &quot;valid&quot; :) I see now sorry. VKey. So I have no objections any more Sergey appears to be okay with the documentation. PLease reopen if any other issues come up. My confusion too was based on the last column. Could we clarify that (if not done already)? I thought you were saying whether or not each stat returned a key and value for the stat. Date of First Response Tue, 15 May 2012 11:41:15 -0500 Flagged Rank 5344 Sprint Priority 1.0 Time In Status [MB-6072] ns_log cannot deal with damaged data file http://www.couchbase.com/issues/browse/MB-6072 Couchbase Server SUBJ. James showed me how exhausted disk space lead to empty ns_log which lead us to being unable to start <br/> MB-6072 ns_log cannot deal with damaged data file Bug Blocker Resolved Fixed Public Aleksey Kondratenko Aleksey Kondratenko Tue, 31 Jul 2012 16:43:24 -0500 Wed, 1 Aug 2012 10:42:53 -0500 Wed, 1 Aug 2012 10:42:53 -0500 1.8.1 2.0-beta ns_server 0 0 Integrated in github-ns-server-2-0 #421 (See [<a href="http://qa.hq.northscale.net/job/github-ns-server-2-0/421/">http://qa.hq.northscale.net/job/github-ns-server-2-0/421/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.couchbase.com/issues/browse/MB-6072" title="ns_log cannot deal with damaged data file"><strike>MB-6072</strike></a>: made ns_log deal with corrupted logs file (Revision 5e990197ca5b940575c19ee8d9061707dbdecded)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> Aliaksey Artamonau : <br/> Files : <br/> * src/ns_log.erl<br/> Date of First Response Wed, 1 Aug 2012 08:54:53 -0500 Rank 4488 Time In Status [MB-5610] view access must enforce bucket level authentication/authorization http://www.couchbase.com/issues/browse/MB-5610 Couchbase Server Currently, all view requests go to the base URI specified in the cluster configuration. Client libraries will assert authentication headers, and the server should verify these authentication headers for all but the default bucket. <br/> <br/> Note: I could not find an issue for this, though I know it&#39;s been discussed. MB-5610 view access must enforce bucket level authentication/authorization Bug Blocker Resolved Duplicate Public Peter Wansch Matt Ingenthron Tue, 19 Jun 2012 13:19:47 -0500 Tue, 3 Jul 2012 15:46:07 -0500 Tue, 19 Jun 2012 13:38:48 -0500 2.0-developer-preview-3 2.0-developer-preview-4 2.0-beta 2.0-beta 0 1 I believe it&#39;s part of generic CAPI needs to be password protected. <br/> <br/> BTW AFAIK latest master already should work correctly w.r.t. Basic auth and views access. this is not a bug. It has not yet been implemented and is in the works being tracked by engineering. tracked via development tasks. we are not feature complete yet. functionality is currently in the works. <br/> <br/> Peter, FYI. Date of First Response Tue, 19 Jun 2012 13:35:47 -0500 Rank 4927 Time In Status [MB-4542] timeout and connection failures under moderate load http://www.couchbase.com/issues/browse/MB-4542 Couchbase Server Running initially with a PHP client, but then simplifying the test and running both clustered and unclustered basic workload tests shows that on both micro instances and large instances basic workload tests timeout and/or fail. On micro instances, this is shows up particularly quickly even in a single node localhost situation. <br/> <br/> Steps: <br/> - Set up EC2 instance <br/> - Install Couchbase Server 2.0 DP3 <br/> - Run /opt/couchbase/bin/memcachetest -l -t 32 -h localhost:11211 <br/> <br/> Observed behavior: <br/> Many errors such as: <br/> ASCII set failure: SERVER_ERROR proxy downstream timeout <br/> ASCII set failure: SERVER_ERROR proxy downstream timeout 10.48.43.67 <br/> ASCII set failure: SERVER_ERROR proxy out of downstreams <br/> ASCII set failure: SERVER_ERROR proxy downstream timeout <br/> &lt;3944&gt; isn&#39;t there anymore <br/> ... <br/> <br/> Also note that the observed throughput would start high, slow down, speed up, etc. <br/> <br/> Expected behavior: <br/> Workload should degrade or possibly timeout in an extreme workload case, but not be issuing out of downstreams. The server should definitely handle this workload without errors. 2.0 DP3 <br/> originally reported on Debian GNU/Linux 6.0 <br/> reproduced with slight differences on Ubuntu 11.10 64 bit MB-4542 timeout and connection failures under moderate load Bug Blocker Resolved Cannot Reproduce Public Unassigned Matt Ingenthron Fri, 16 Dec 2011 13:36:15 -0600 Mon, 20 Feb 2012 16:37:12 -0600 Mon, 20 Feb 2012 16:37:12 -0600 2.0-developer-preview-3 2.0-beta 0 4 Output courtesy buzzy&#39;s Debian system. <br/> <br/> <a href='mailto:root@ip-10-48-43-67'>root@ip-10-48-43-67</a>:/opt/couchbase/bin# ./memcachetest -l -t 32 -h localhost:11211 <br/> Average with 32 threads <br/> Get operations: <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#of ops. min max avg max90th max95th max99th <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6655 108 us 18 ms 4457 us 9023 us 9384 us 11 ms <br/> <br/> Set operations: <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#of ops. min max avg max90th max95th max99th <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;3345 116 us 18 ms 4284 us 8986 us 9311 us 11 ms <br/> <br/> Average with 32 threads <br/> Get operations: <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#of ops. min max avg max90th max95th max99th <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6758 114 us 26 ms 4995 us 9600 us 10 ms 13 ms <br/> <br/> Set operations: <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#of ops. min max avg max90th max95th max99th <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;3242 116 us 26 ms 5053 us 9586 us 10 ms 14 ms <br/> <br/> Average with 32 threads <br/> Get operations: <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#of ops. min max avg max90th max95th max99th <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6696 109 us 20 ms 4878 us 9280 us 9811 us 12 ms <br/> <br/> Set operations: <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#of ops. min max avg max90th max95th max99th <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;3304 117 us 15 ms 4585 us 9256 us 9691 us 12 ms <br/> <br/> Average with 32 threads <br/> Get operations: <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#of ops. min max avg max90th max95th max99th <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6720 133 us 20 ms 4946 us 9562 us 10 ms 13 ms <br/> <br/> Set operations: <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#of ops. min max avg max90th max95th max99th <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;3280 115 us 16 ms 4872 us 9520 us 10 ms 13 ms <br/> <br/> Average with 32 threads <br/> Get operations: <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#of ops. min max avg max90th max95th max99th <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6752 125 us 14 ms 4581 us 9192 us 10 ms 12 ms <br/> <br/> Set operations: <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#of ops. min max avg max90th max95th max99th <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;3248 156 us 14 ms 4387 us 9103 us 10 ms 12 ms <br/> <br/> Average with 32 threads <br/> Get operations: <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#of ops. min max avg max90th max95th max99th <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6694 116 us 13 ms 4501 us 9088 us 9431 us 11 ms <br/> <br/> Set operations: <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#of ops. min max avg max90th max95th max99th <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;3306 127 us 13 ms 4357 us 9069 us 9365 us 11 ms <br/> <br/> Average with 32 threads <br/> Get operations: <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#of ops. min max avg max90th max95th max99th <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6749 110 us 15 ms 4365 us 8978 us 9269 us 11 ms <br/> <br/> Set operations: <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#of ops. min max avg max90th max95th max99th <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;3251 114 us 15 ms 4255 us 8973 us 9236 us 10 ms <br/> <br/> Average with 32 threads <br/> Get operations: <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#of ops. min max avg max90th max95th max99th <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6766 109 us 23 ms 4538 us 9143 us 9691 us 11 ms <br/> <br/> Set operations: <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#of ops. min max avg max90th max95th max99th <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;3234 117 us 23 ms 4557 us 9154 us 9773 us 12 ms <br/> <br/> &lt;9749&gt; isn&#39;t there anymore <br/> &lt;3328&gt; isn&#39;t there anymore <br/> &lt;1644&gt; isn&#39;t there anymore <br/> &lt;638&gt; isn&#39;t there anymore <br/> &lt;9351&gt; isn&#39;t there anymore <br/> &lt;2917&gt; isn&#39;t there anymore <br/> ASCII set failure: SERVER_ERROR proxy downstream timeout 10.48.43.67 <br/> ASCII set failure: SERVER_ERROR proxy out of downstreams <br/> &lt;3944&gt; isn&#39;t there anymore <br/> &lt;6808&gt; isn&#39;t there anymore <br/> ASCII set failure: SERVER_ERROR proxy out of downstreams <br/> &lt;3570&gt; isn&#39;t there anymore <br/> ASCII set failure: SERVER_ERROR proxy out of downstreams <br/> &lt;8189&gt; isn&#39;t there anymore <br/> &lt;2863&gt; isn&#39;t there anymore <br/> ASCII set failure: SERVER_ERROR proxy out of downstreams <br/> ASCII set failure: SERVER_ERROR proxy out of downstreams <br/> &lt;3757&gt; isn&#39;t there anymore <br/> &lt;2961&gt; isn&#39;t there anymore <br/> &lt;1153&gt; isn&#39;t there anymore <br/> &lt;5042&gt; isn&#39;t there anymore <br/> &lt;8506&gt; isn&#39;t there anymore <br/> ASCII set failure: SERVER_ERROR proxy out of downstreams <br/> &lt;5451&gt; isn&#39;t there anymore <br/> &lt;9111&gt; isn&#39;t there anymore <br/> &lt;7916&gt; isn&#39;t there anymore <br/> ASCII set failure: SERVER_ERROR proxy out of downstreams <br/> &lt;1854&gt; isn&#39;t there anymore <br/> &lt;5432&gt; isn&#39;t there anymore <br/> &lt;9347&gt; isn&#39;t there anymore <br/> &lt;7558&gt; isn&#39;t there anymore <br/> &lt;5836&gt; isn&#39;t there anymore <br/> ASCII set failure: SERVER_ERROR proxy out of downstreams <br/> &lt;5681&gt; isn&#39;t there anymore <br/> &lt;2271&gt; isn&#39;t there anymore I have found this: <br/> <br/> <a href="http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/concepts_micro_instances.html">http://docs.amazonwebservices.com/AWSEC2/latest/UserGuide/concepts_micro_instances.html</a> <br/> <br/> This might be the cause of how micro-instances at Amazon EC2 works. Seems like they limit the CPU after some time of high load. <br/> <br/> I will try to do the same thing on a &quot;small&quot;-instance and see if it gives the same result. So far I have not been able to reproduce this with a large instance. Was this seen with the same memcachetest on both the micro and large? I have tried it with a &quot;small&quot; instance at Amazon EC2 now and it was even worse :( <br/> <br/> 1. Create a &quot;small&quot; instance in EC2 with ami-6bb5821f in the EU-region <br/> 2. Download the 32bit .deb package from couchbase and install with &quot;dpkg -i&quot; <br/> 3. Go to the HTTP admin and configure a 500MB membase bucket. <br/> 4. Run a test from another instance, like: ./memcachetest -l -t 32 -h 10.227.7.34:11211 <br/> <br/> When I do this, I see the numbers go up to 4-5k/sec for a couple of seconds, then errors: <br/> <br/> (<a href='mailto:root@ip-10-48-125-234'>root@ip-10-48-125-234</a>)-(/opt/couchbase/bin)# ./memcachetest -l -t 32 -h 10.227.7.34:11211 <br/> ASCII set failure: SERVER_ERROR proxy downstream timeout <br/> Failed to set [8876]: during populate. <br/> ASCII set failure: SERVER_ERROR proxy downstream timeout <br/> Failed to set [1074]: during populate. <br/> ASCII set failure: SERVER_ERROR proxy downstream timeout <br/> Failed to set [4521]: during populate. <br/> ASCII set failure: SERVER_ERROR proxy downstream timeout <br/> Failed to set [6076]: during populate. <br/> Tried to run the test locally on the Couchbase server itself. It seems to work a little longer. <br/> <br/> I get about 3500 ops/sec for 2-3 min, then errors start. <br/> <br/> Maybe connections are not closed fast enough? <br/> <br/> ./memcachetest -l -t 32 -h localhost:11211 <br/> <br/> Set operations: <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#of ops. min max avg max90th max95th max99th <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;3377 103 us 501 ms 7934 us 7156 us 36 ms 214 ms <br/> <br/> Average with 32 threads <br/> Get operations: <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#of ops. min max avg max90th max95th max99th <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6720 90 us1266874885 2745051 s 7031 us 43 ms 239 ms <br/> <br/> Set operations: <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#of ops. min max avg max90th max95th max99th <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;3280 101 us1266874885 5624007 s 6580 us 37 ms 188 ms <br/> <br/> Average with 32 threads <br/> Get operations: <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#of ops. min max avg max90th max95th max99th <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;6613 92 us1266874885 2789466 s 14 ms 40 ms 168 ms <br/> <br/> Set operations: <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;#of ops. min max avg max90th max95th max99th <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;3387 101 us1266874885 5446337 s 12 ms 42 ms 281 ms <br/> <br/> ASCII set failure: SERVER_ERROR proxy downstream timeout <br/> ASCII set failure: SERVER_ERROR proxy downstream timeout <br/> &lt;1656&gt; isn&#39;t there anymore <br/> Average with 32 threads <br/> I have done some more tests. This time with &quot;memslap&quot; <br/> <br/> ./memslap -s 127.0.0.1:11211 -S 5s -c 128 <br/> &lt;84 SERVER_ERROR proxy downstream timeout <br/> &lt;80 SERVER_ERROR proxy downstream timeout <br/> &lt;92 SERVER_ERROR proxy downstream timeout <br/> &lt;100 SERVER_ERROR proxy downstream timeout <br/> &lt;104 SERVER_ERROR proxy downstream timeout <br/> <br/> ./memslap -s 127.0.0.1:11211 -S 5s -c 64 <br/> This gives less errors <br/> <br/> ./memslap -s 127.0.0.1:11211 -S 5s -c 16 <br/> This one works fine for hours <br/> <br/> It is pretty obvious that this has something to do with concurrent connections. The Proxy cannot take more than somewhere between 16-64 connection. <br/> <br/> Is someone looking at this? I would say this is kind of a big deal :) <br/> small instance also has 1 ECU as well. couch base server needs at least 2 real physical cores. <br/> <br/> we couldn&#39;t reproduce this on a 64-bit large instance. have you seen the same issue there ? Yes, I see the same issue on &quot;micro&quot;, &quot;small&quot; and &quot;large&quot; instances. <br/> <br/> But now I found this: <br/> <a href="http://www.couchbase.org/wiki/display/membase/Moxi+Performance+Tuning">http://www.couchbase.org/wiki/display/membase/Moxi+Performance+Tuning</a> <br/> <br/> &quot;To simplify, let&#39;s say you just have a single bucket (such as the &quot;default&quot; bucket), that downstream_max is 4 (the default, circa 1.6.0 moxi) and that NUM_MOXI_WORKER_THREADS is 4 (which is also the default -t command-line flag value, circa 1.6.0 moxi). So, in this example, there are 4 x 1 x 4 = 16 connections that a single moxi will make to memcached/membase.&quot; <br/> <br/> Is it true that the default connection-limit is only 16? That is waaaaaay too low in my opinion. yes thats moxi&#39;s limitation. <br/> <br/> do you see the same behavior when running the load directly against memcached instead of maxi? I cannot run against memcached directly. I see that couchbase installed memcached on port 11210, but running memslap againt that port fails. <br/> <br/> <br/> (<a href='mailto:root@ip-10-49-37-17'>root@ip-10-49-37-17</a>)-(/opt/couchbase/bin)# netstat -nlp|grep memcache <br/> tcp 0 0 0.0.0.0:11210 0.0.0.0:* LISTEN 1488/memcached <br/> tcp6 0 0 :::11210 :::* LISTEN 1488/memcached <br/> udp 0 0 0.0.0.0:11210 0.0.0.0:* 1488/memcached <br/> udp6 0 0 :::11210 :::* 1488/memcached <br/> (<a href='mailto:root@ip-10-49-37-17'>root@ip-10-49-37-17</a>)-(/opt/couchbase/bin)# ./memslap -s 127.0.0.1:11210 -S 5s -c 16 <br/> servers : 127.0.0.1:11210 <br/> threads count: 1 <br/> concurrency: 16 <br/> run time: 600s <br/> windows size: 10k <br/> set proportion: set_prop=0.10 <br/> get proportion: get_prop=0.90 <br/> memslap: clients/memslap.c:793: ms_print_memslap_stats: Assertion `pos &lt;= buf&#39; failed. <br/> Aborted <br/> Daniele, this is one of the many reasons why we recommend deploying a client-side Moxi as opposed to using the included server-side one. Why was Moxi added if it cripples memcached that much? Normal memcached has like 1024 connection limit by default. Moxi limits this to 16? Would make sense to follow memcached default limit to 1024 then. Perry, but I am running all this on one machine. What difference would a &quot;client-side&quot; moxi do then? Do you have a link how to just turned the moxi off all together and rely on the clients hashing instead? Daniele, <br/> <br/> memslap is not a supported tool or a binary against recent membase or couchbase servers. It would b best to try out with a smart client or use the client side moxi where you can change its configurations and increase the number of connections and etc as Perry mentioned. <br/> <br/> you should be able to use memcachetest to run smaple load against memcached directly if you have one node. The &quot;client-side&quot; Moxi gives you the ability to change these parameters...the server-side has hardcoded defaults which are obviously too low for certain scenarios. <br/> <br/> There is no way to turn off Moxi altogether. You don&#39;t need to turn off the server-side one if you&#39;re using a client-side Moxi and/or smart client, the server-side one will just be sitting idle. Unless you are using a smart client (currently available for Java and .NET, soon to be released for PHP and Ruby), you must use Moxi. <br/> <br/> Ok, understood. I was looking for a way to change the config for the &quot;server-side&quot; moxi without success. <br/> <br/> I guess in my case, I will install another moxi beside the existing one and put it on another port and run my tests again. <br/> <br/> Btw, the &quot;memslap&quot; binary is included in your package so I assumed it was ok to use it :) <br/> I have also used memcachetest with the same result, but at least now I know why. <br/> <br/> I will let you know if the client-side moxi solves my issues. Thank you. Date of First Response Sun, 18 Dec 2011 02:27:28 -0600 Rank 5907 Time In Status [MB-5925] Broken ejson encoder http://www.couchbase.com/issues/browse/MB-5925 Couchbase Server The JSON encoder/decoder used in Erlang (inside couchdb component), ejson, is currently broken. It encodes invalid UTF-8 strings when it should throw/raise/return an error, on the other hand it&#39;s unable to decode the invalid UTF-8 encoded strings (which is correct). Mochijson2, another Erlang-based JSON encoder/decoder, has the correct behaviour on raising an error if a string contains invalid UTF-8. See below example in an Erlang shell: <br/> <br/> 1&gt; Result = ejson:encode(&lt;&lt;255&gt;&gt;). <br/> &lt;&lt;&quot;\&quot;&#xFF;\&quot;&quot;&gt;&gt; <br/> 2&gt; ejson:decode(Result). <br/> ** exception throw: {invalid_json,{{error,{2, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&quot;lexical error: invalid bytes in UTF8 string.\n&quot;}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;\&quot;&#xFF;\&quot;&quot;&gt;&gt;}} <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in function ejson:nif_decode/1 (ejson.erl, line 57) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in call from ejson:decode/1 (ejson.erl, line 38) <br/> 3&gt; <br/> 3&gt; mochijson2:encode(&lt;&lt;255&gt;&gt;). <br/> ** exception exit: {ucs,{bad_utf8_character_code}} <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in function xmerl_ucs:from_utf8/1 (xmerl_ucs.erl, line 185) <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in call from mochijson2:json_encode_string/2 (mochijson2.erl, line 186) <br/> <br/> <br/> In Python, encoding a string with byte 255 (invalid UTF-8) is also not allowed (which is correct): <br/> <br/> $ python <br/> Python 2.7.1 (r271:86832, Jun 16 2011, 16:59:05) <br/> [GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin <br/> Type &quot;help&quot;, &quot;copyright&quot;, &quot;credits&quot; or &quot;license&quot; for more information. <br/> &gt;&gt;&gt; import json <br/> &gt;&gt;&gt; json.dumps(&quot;\xff&quot;) <br/> Traceback (most recent call last): <br/> &nbsp;&nbsp;File &quot;&lt;stdin&gt;&quot;, line 1, in &lt;module&gt; <br/> &nbsp;&nbsp;File &quot;/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py&quot;, line 231, in dumps <br/> &nbsp;&nbsp;&nbsp;&nbsp;return _default_encoder.encode(obj) <br/> &nbsp;&nbsp;File &quot;/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/encoder.py&quot;, line 195, in encode <br/> &nbsp;&nbsp;&nbsp;&nbsp;return encode_basestring_ascii(o) <br/> UnicodeDecodeError: &#39;utf8&#39; codec can&#39;t decode byte 0xff in position 0: invalid start byte <br/> &gt;&gt;&gt; MB-5925 Broken ejson encoder Bug Blocker Resolved Won't Fix Public Damien Katz Filipe Manana Tue, 17 Jul 2012 06:19:15 -0500 Thu, 30 Aug 2012 15:48:52 -0500 Fri, 24 Aug 2012 13:12:54 -0500 2.0-developer-preview-4 2.0-beta storage-engine 0 2 At the very least, there&#39;s the inconsistency of being able to encode such string but being unable to decode the produced string. <br/> This can break view-engine, cross data center replicaton, etc. I think this is related to <a href="http://www.couchbase.com/issues/browse/MB-6138">http://www.couchbase.com/issues/browse/MB-6138</a> which has additional client code to reproduce We will be dropping non-utf8 keys per CBD-453 Date of First Response Mon, 6 Aug 2012 20:54:25 -0500 Rank 4632 Time In Status [MB-5676] Meta data stat is incorrect in ui http://www.couchbase.com/issues/browse/MB-5676 Couchbase Server <br/> The stat you are using is the overhead of the hashtable and doesn&#39;t include any stored value object. Please use &quot;vb_active_meta_data_memory&quot; to properly display the meta data in use. MB-5676 Meta data stat is incorrect in ui Bug Blocker Resolved Fixed Public Mike Wiederhold Mike Wiederhold Mon, 25 Jun 2012 14:48:31 -0500 Tue, 10 Jul 2012 12:26:41 -0500 Mon, 9 Jul 2012 16:05:25 -0500 2.0-developer-preview-4 2.0-beta UI 0 0 <a href="http://review.couchbase.com/#change,18099">http://review.couchbase.com/#change,18099</a> Integrated in github-ns-server-2-0 #397 (See [<a href="http://qa.hq.northscale.net/job/github-ns-server-2-0/397/">http://qa.hq.northscale.net/job/github-ns-server-2-0/397/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.couchbase.com/issues/browse/MB-5676" title="Meta data stat is incorrect in ui"><strike>MB-5676</strike></a>: Use the correct meta data stat in the UI (Revision 0739cd1a3be01508a520a9ad3b6144dcfbcca3e2)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> Aliaksey Kandratsenka : <br/> Files : <br/> * src/stats_collector.erl<br/> * src/menelaus_stats.erl<br/> * priv/public/js/hooks.js<br/> * include/ns_stats.hrl<br/> Date of First Response Tue, 10 Jul 2012 01:14:10 -0500 Rank 4868 Time In Status [MB-5426] Unable to retrieve documents after node reboot http://www.couchbase.com/issues/browse/MB-5426 Couchbase Server We are testing DP4 on a 3 node Couchbase cluster. After stopping and starting any number of nodes we are unable to retrieve a large number of documents on the database. <br/> <br/> At that point we are able to see errors such as these in the logs: <br/> <br/> Control connection to memcached on &#39;<a href='mailto:ns_1@172.x.x.x'>ns_1@172.x.x.x</a>&#39; disconnected: {{badmatch, <br/> {error, <br/> closed}}, <br/> [{mc_client_binary, <br/> stats_recv, <br/> 4}, <br/> {mc_client_binary, <br/> stats, <br/> 4}, <br/> {ns_memcached, <br/> do_handle_call, <br/> 3}, <br/> {ns_memcached, <br/> handle_call, <br/> 3}, <br/> {gen_server, <br/> handle_msg, <br/> 5}, <br/> {proc_lib, <br/> init_p_do_apply, <br/> 3}]} (repeated 6 times) 64 Bit Debian 6 Linux. Installed from couchbase-server-community_x86_64_2.0.0-dev-preview-4.deb MB-5426 Unable to retrieve documents after node reboot Bug Blocker Resolved Duplicate Public Dipti Borkar Shain Miley Fri, 1 Jun 2012 12:18:38 -0500 Tue, 3 Jul 2012 15:45:54 -0500 Fri, 1 Jun 2012 13:05:39 -0500 2.0-developer-preview-4 2.0-beta 0 1 from the logs : <br/> memcached: stored-value.hh:281: size_t StoredValue::valLength(): Assertion `value-&gt;length() == sizeof(uval)&#39; failed. <br/> 2012-06-01 20:22:49.140 ns_port_server:0:info:message(<a href='mailto:ns_1@172.31.2.132'>ns_1@172.31.2.132</a>) - Port server memcached on node &#39;<a href='mailto:ns_1@172.31.2.132'>ns_1@172.31.2.132</a>&#39; exited with status 134. Restarting. Messages: Trying to connect to mccouch: &quot;localhost:11213&quot; <br/> <br/> <br/> dupe of <a href="http://www.couchbase.com/issues/browse/MB-5021">http://www.couchbase.com/issues/browse/MB-5021</a> <br/> <br/> fixed in 2.0 dp5 <a href="http://www.couchbase.com/issues/browse/MB-4985" title="memcached crash ( stored-value.hh:284:) when view merger is updating partition states and user tries to delete the bucket"><strike>MB-4985</strike></a> Date of First Response Fri, 1 Jun 2012 13:05:29 -0500 Rank 5074 Time In Status [MB-5423] data goes missing, nodes drop into pending state with cluster in steady state http://www.couchbase.com/issues/browse/MB-5423 Couchbase Server There are reports of at least the Web Console showing nodes in a good size cluster (&gt;5 nodes) going into pending state at the web console. <br/> <br/> From the reporter: <br/> <br/> 1. A node will fill it&#39;s RAM, then go into a PEND state. <br/> 2. I reboot the service on the node. <br/> 3. The node starts up and loads its docs. <br/> 4. It drops all it&#39;s documents and goes back into PEND state. <br/> 5. Loads it&#39;s docs up again then goes UP state. <br/> 6. Then repeats the loss of docs and PEND/UP state. <br/> <br/> See the thread here: <br/> <a href="http://www.couchbase.com/forums/thread/losing-data-when-restarting-node">http://www.couchbase.com/forums/thread/losing-data-when-restarting-node</a> At least on CentOS 6.2, and possibly one other platform MB-5423 data goes missing, nodes drop into pending state with cluster in steady state Bug Blocker Resolved Duplicate Public Dipti Borkar Matt Ingenthron Fri, 1 Jun 2012 11:32:36 -0500 Tue, 3 Jul 2012 15:45:54 -0500 Tue, 12 Jun 2012 18:27:14 -0500 2.0-developer-preview-4 2.0-beta 0 1 Having some logs would help here. This looks like memcached crash. And even user-level logs would confirm that. &nbsp;&nbsp;&nbsp;My setup is single node and pretty much what you get &quot;out of the box&quot; so cloning another cluster is not a possible cause. It is not apparent that RAM is filled as after the node comes up RAM is often &lt;10M per bucket <br/> <br/> &nbsp;I believe there are two issues: <br/> <br/> 1) the node dropping into pending state. The log messages I get are: <br/> <br/> Module Code: ns_port_server000 <br/> <br/> Port server memcached on node &#39;<a href='mailto:ns_1@127.0.0.1'>ns_1@127.0.0.1</a>&#39; exited with status 134. Restarting. Messages: Failed to connect to: &quot;localhost:11213&quot; <br/> Connection closed by mccouch <br/> Preloaded 136076 keys (with metadata) <br/> Trying to connect to mccouch: &quot;localhost:11213&quot; <br/> Failed to connect to: &quot;localhost:11213&quot; <br/> Failed to connect to: &quot;localhost:11213&quot; <br/> Connection closed by mccouch <br/> Connection closed by mccouch <br/> Preloaded 3998 keys (with metadata) <br/> Preloaded 102201 keys (with metadata) <br/> Trying to connect to mccouch: &quot;localhost:11213&quot; <br/> Trying to connect to mccouch: &quot;localhost:11213&quot; <br/> .... <br/> <br/> Followed by <br/> Module Code: ns_memcached004 <br/> <br/> Control connection to memcached on &#39;<a href='mailto:ns_1@127.0.0.1'>ns_1@127.0.0.1</a>&#39; disconnected: {{badmatch, <br/> {error, <br/> closed}}, <br/> [{mc_client_binary, <br/> stats_recv, <br/> 4}, <br/> {mc_client_binary, <br/> stats,4}, <br/> {ns_memcached, <br/> do_handle_call, <br/> 3}, <br/> {ns_memcached, <br/> handle_call, <br/> 3}, <br/> {gen_server, <br/> handle_msg, <br/> 5}, <br/> {proc_lib, <br/> init_p_do_apply, <br/> 3}]} (repeated 15 times) <br/> <br/> <br/> 2) The UI agent is either not collecting or not displaying summary stats correctly. The item count is almost always incorrect for all buckets when view from the Data Buckets page but the view of the bucket on port 8092 seems to show the correct item count <br/> Steven, <br/> <br/> do you see something like this in the diags ? <br/> <br/> memcached: stored-value.hh:281: size_t StoredValue::valLength(): Assertion `value-&gt;length() == sizeof(uval)&#39; failed. <a href="http://www.couchbase.com/issues/browse/MB-5021" title="memcached: stored-value.hh:281 under load on dp4 build"><strike>MB-5021</strike></a> Date of First Response Fri, 1 Jun 2012 11:39:28 -0500 Rank 5283 Time In Status [MB-5168] Loading couchbase crashes web console http://www.couchbase.com/issues/browse/MB-5168 Couchbase Server web server crashed while inserting JSON records. Incoming records are &quot;bursty&quot; but not incredibly large or numerous, i.e. couple hundred records per sec at a couple kb each. <br/> <br/> I can make the coredump available if it would be useful. <br/> <br/> <br/> /var/log/messages: <br/> Apr 26 07:25:09 asitri2 abrt[4431]: saved core dump of pid 2071 (/mnt/couch/couchbase/lib/erlang/erts-5.8.4/bin/beam.smp) to /var/spool/abrt/ccpp-2012-04-26-07:17:54-2071.new/coredump (3163418624 bytes) <br/> Apr 26 07:25:10 asitri2 abrtd: Directory &#39;ccpp-2012-04-26-07:17:54-2071&#39; creation detected <br/> Apr 26 07:25:17 asitri2 abrtd: Package &#39;couchbase-server&#39; isn&#39;t signed with proper key <br/> Apr 26 07:25:17 asitri2 abrtd: Corrupted or bad dump /var/spool/abrt/ccpp-2012-04-26-07:17:54-2071 (res:2), deleting Linux 2.6.32-220.7.1.el6.i686 #1 SMP Tue Mar 6 21:21:22 GMT 2012 i686 i686 i386 GNU/Linux <br/> java 1.1-dp client MB-5168 Loading couchbase crashes web console Bug Blocker Resolved Incomplete Public Aleksey Kondratenko Steven Cooke Thu, 26 Apr 2012 06:57:29 -0500 Tue, 3 Jul 2012 15:45:46 -0500 Tue, 12 Jun 2012 18:25:39 -0500 2.0-developer-preview-4 2.0-beta ns_server 0 2 Is there any way we can get this core dump? <br/> Date of First Response Fri, 27 Apr 2012 12:00:17 -0500 Rank 5425 Time In Status [MB-4970] Undefined set view error when running views after rebalancing out a node which was restarted http://www.couchbase.com/issues/browse/MB-4970 Couchbase Server Given: <br/> - A cluster with 3 boxes (2 Windows 2008 R2, 1 Ubuntu 11.04) and 5 data buckets with no replication. <br/> <br/> When: <br/> - Ubuntu&#xB4;s box goes down. <br/> - Restart the couchbase-server service in that box. <br/> <br/> Then: <br/> - Can&#xB4;t execute any view. <br/> - The admin console shows: <br/> <br/> Subset of nodes failed with the following error: <br/> [ <br/> { <br/> &quot;from&quot;: &quot;<a href="http://10.230.58.238:8092/_view_merge/">http://10.230.58.238:8092/_view_merge/</a>&quot;, <br/> &quot;reason&quot;: &quot;Undefined set view `test` for `_design/dev_CategoryTree` design document.&quot; <br/> } <br/> ] <br/> <br/> Logs: <br/> [ns_server:info] [2012-03-28 17:11:53] [<a href='mailto:ns_1@10.230.58.221'>ns_1@10.230.58.221</a>:&lt;0.17585.189&gt;:ns_vbm_sup:spawn_mover:198] Spawned mover &quot;dev&quot; 125 &#39;<a href='mailto:ns_1@10.230.58.221'>ns_1@10.230.58.221</a>&#39; -&gt; &#39;<a href='mailto:ns_1@10.230.58.238'>ns_1@10.230.58.238</a>&#39;: &lt;0.17586.189&gt; <br/> [ns_server:info] [2012-03-28 17:11:53] [<a href='mailto:ns_1@10.230.58.221'>ns_1@10.230.58.221</a>:&lt;0.738.0&gt;:ns_port_server:log:161] memcached&lt;0.738.0&gt;: Vbucket &lt;121&gt; is going dead. <br/> memcached&lt;0.738.0&gt;: Vbucket &lt;122&gt; is going dead. <br/> memcached&lt;0.738.0&gt;: Vbucket &lt;123&gt; is going dead. <br/> memcached&lt;0.738.0&gt;: Vbucket &lt;124&gt; is going dead. <br/> <br/> [rebalance:info] [2012-03-28 17:11:53] [<a href='mailto:ns_1@10.230.58.221'>ns_1@10.230.58.221</a>:&lt;0.17586.189&gt;:ebucketmigrator_srv:init:135] CheckpointIdsDict: <br/> {dict,128,26,32,16,130,78, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{{[[0|1],[32|1],[64|1],[96|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[19|1],[51|1],[83|1],[115|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[6|1],[38|1],[70|1],[102|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[25|1],[57|1],[89|1],[121|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[12|1],[44|1],[76|1],[108|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[31|1],[63|1],[95|1],[127|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[18|1],[50|1],[82|1],[114|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[5|2],[37|1],[69|1],[101|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[24|1],[56|1],[88|1],[120|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[11|1],[43|1],[75|1],[107|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[14|1],[30|1],[46|1],[62|1],[78|1],[94|1],[110|1],[126|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[1|1],[17|1],[33|1],[49|1],[65|1],[81|1],[97|1],[113|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[4|1],[20|1],[36|1],[52|1],[68|1],[84|1],[100|1],[116|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[7|1],[23|1],[39|1],[55|1],[71|1],[87|1],[103|1],[119|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[10|1],[26|1],[42|1],[58|1],[74|1],[90|1],[106|1],[122|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[13|1],[29|1],[45|1],[61|1],[77|1],[93|1],[109|1],[125|1]]}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{[[16|1],[48|1],[80|1],[112|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[3|1],[35|1],[67|1],[99|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[22|1],[54|1],[86|1],[118|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[9|1],[41|1],[73|1],[105|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[28|1],[60|1],[92|1],[124|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[15|1],[47|1],[79|1],[111|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[2|1],[34|1],[66|1],[98|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[21|1],[53|1],[85|1],[117|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[8|1],[40|1],[72|1],[104|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[[27|1],[59|1],[91|1],[123|1]], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[],[],[],[],[],[]}}} <br/> <br/> [rebalance:info] [2012-03-28 17:11:53] [<a href='mailto:ns_1@10.230.58.221'>ns_1@10.230.58.221</a>:&lt;0.17586.189&gt;:ebucketmigrator_srv:init:166] Starting tap stream: <br/> [{vbuckets,&quot;}&quot;}, <br/> &nbsp;{checkpoints,[{125,0}]}, <br/> &nbsp;{name,&quot;rebalance_125&quot;}, <br/> &nbsp;{takeover,true}] <br/> <br/> [error_logger:error] [2012-03-28 17:11:53] [<a href='mailto:ns_1@10.230.58.221'>ns_1@10.230.58.221</a>:error_logger:ale_error_logger_handler:log_msg:76] ** Generic server auto_failover terminating <br/> ** Last message in was tick <br/> ** When Server state == {state, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{state, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{node_state,&#39;<a href='mailto:ns_1@10.230.58.221'>ns_1@10.230.58.221</a>&#39;,0,up,false}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{node_state,&#39;<a href='mailto:ns_1@10.230.58.238'>ns_1@10.230.58.238</a>&#39;,1, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;nearly_down,false}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{node_state,&#39;<a href='mailto:ns_1@10.230.58.37'>ns_1@10.230.58.37</a>&#39;,0,up,false}], <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;0,3}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{interval,#Ref&lt;0.0.3282.124225&gt;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;30,0} <br/> ** Reason for termination == <br/> ** {{badmatch,rebalancing}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;[{ns_cluster_membership,failover,1}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{auto_failover,&#39;-handle_info/2-fun-0-&#39;,2}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{lists,foldl,3}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{auto_failover,handle_info,2}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server,handle_msg,5}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{proc_lib,init_p_do_apply,3}]} <br/> <br/> [ns_server:info] [2012-03-28 17:11:53] [<a href='mailto:ns_1@10.230.58.221'>ns_1@10.230.58.221</a>:&lt;0.17586.189&gt;:ebucketmigrator_srv:init:175] upstream_sender pid: &lt;0.17589.189&gt; <br/> [error_logger:error] [2012-03-28 17:11:53] [<a href='mailto:ns_1@10.230.58.221'>ns_1@10.230.58.221</a>:error_logger:ale_error_logger_handler:log_report:72] <br/> =========================CRASH REPORT========================= <br/> &nbsp;&nbsp;crasher: <br/> &nbsp;&nbsp;&nbsp;&nbsp;initial call: auto_failover:init/1 <br/> &nbsp;&nbsp;&nbsp;&nbsp;pid: &lt;0.14760.180&gt; <br/> &nbsp;&nbsp;&nbsp;&nbsp;registered_name: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;exception exit: {{badmatch,rebalancing}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{ns_cluster_membership,failover,1}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{auto_failover,&#39;-handle_info/2-fun-0-&#39;,2}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{lists,foldl,3}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{auto_failover,handle_info,2}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server,handle_msg,5}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{proc_lib,init_p_do_apply,3}]} <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;in function gen_server:terminate/6 <br/> &nbsp;&nbsp;&nbsp;&nbsp;ancestors: [mb_master_sup,mb_master,ns_server_sup,ns_server_cluster_sup, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;0.51.0&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;messages: [] <br/> &nbsp;&nbsp;&nbsp;&nbsp;links: [&lt;0.696.0&gt;,&lt;0.160.0&gt;] <br/> &nbsp;&nbsp;&nbsp;&nbsp;dictionary: [{random_seed,{6571,6892,26285}}] <br/> &nbsp;&nbsp;&nbsp;&nbsp;trap_exit: false <br/> &nbsp;&nbsp;&nbsp;&nbsp;status: running <br/> &nbsp;&nbsp;&nbsp;&nbsp;heap_size: 75025 <br/> &nbsp;&nbsp;&nbsp;&nbsp;stack_size: 24 <br/> &nbsp;&nbsp;&nbsp;&nbsp;reductions: 4294650 <br/> &nbsp;&nbsp;neighbours: <br/> <br/> [error_logger:error] [2012-03-28 17:11:53] [<a href='mailto:ns_1@10.230.58.221'>ns_1@10.230.58.221</a>:error_logger:ale_error_logger_handler:log_report:72] <br/> =========================SUPERVISOR REPORT========================= <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Supervisor: {local,mb_master_sup} <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Context: child_terminated <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Reason: {{badmatch,rebalancing}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{ns_cluster_membership,failover,1}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{auto_failover,&#39;-handle_info/2-fun-0-&#39;,2}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{lists,foldl,3}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{auto_failover,handle_info,2}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server,handle_msg,5}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{proc_lib,init_p_do_apply,3}]} <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Offender: [{pid,&lt;0.14760.180&gt;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{name,auto_failover}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{mfargs,{auto_failover,start_link,[]}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{restart_type,permanent}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{shutdown,10}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{child_type,worker}] <br/> <br/> <br/> [rebalance:info] [2012-03-28 17:11:53] [<a href='mailto:ns_1@10.230.58.221'>ns_1@10.230.58.221</a>:&lt;0.17586.189&gt;:ebucketmigrator_srv:terminate:202] Skipping close ack for successfull takover <br/> Windows 2008 R2 64Bit, Ubuntu 11.04. MB-4970 Undefined set view error when running views after rebalancing out a node which was restarted Bug Blocker Resolved Fixed Public Farshid Ghods francares Wed, 28 Mar 2012 15:27:49 -0500 Tue, 3 Jul 2012 15:46:06 -0500 Tue, 12 Jun 2012 18:42:32 -0500 2.0-developer-preview-4 2.0-beta view-engine 0 1 can you also attach the diags from :8091/diag <br/> <br/> did you wait until the node came back up and the bucket status was ready (green) ? Yep, I did. If I do not wait until it becomes green It can&#xB4;t make a rebalance operation I think. <br/> <br/> Could you tell me what other info in the diag file do you need? Its size is about 300Mb. I&#xB4;ve added latest log entries when the issue was manifested. <br/> In my opinion you should improve the logs functionality because it&#xB4;s not good to have big logs files. <br/> Maybe a listener for Splunk or other tool should be fine. <br/> can you please try to bzip or zip the file and see if the size is less than 20 MB . <br/> <br/> jira allows uploading attachments &lt;= 20 MB also can you please update the &quot;when&quot; part : <br/> <br/> you didn&#39;t mention rebalancing or possible failover int he bug description Added log file. <br/> <br/> I don&#xB4;t have permissions to modify the field description. So here it go: <br/> <br/> When: <br/> - Ubuntu&#xB4;s box goes down. <br/> - Restart the couchbase-server service in that box. <br/> - Add server again. <br/> - Do a rebalance. <br/> <br/> Let me know if you need more info... Hi Filipe,<br/> <br/> Can you please have a look at this and see if there is anything obvious in the logs <br/> <br/> <br/> Farshid Can&#39;t see anything related to my side. <br/> <br/> There&#39;s the following however repeated several times (ep-engine/mccouch): <br/> <br/> <br/> [ns_server:info] [2012-03-28 17:32:53] [<a href='mailto:ns_1@10.230.58.221'>ns_1@10.230.58.221</a>:&lt;0.11597.191&gt;:menelaus_web:handle_streaming:947] menelaus_web streaming socket closed by client <br/> [couchdb:info] [2012-03-28 17:32:54] [<a href='mailto:ns_1@10.230.58.221'>ns_1@10.230.58.221</a>:&lt;0.11891.190&gt;:couch_log:info:39] FLUSHING ALL THE THINGS! <br/> [views:info] [2012-03-28 17:32:54] [<a href='mailto:ns_1@10.230.58.221'>ns_1@10.230.58.221</a>:&#39;capi_set_view_manager-qa&#39;:capi_set_view_manager:apply_map:445] Applying map to bucket qa: <br/> [{active,[]},{passive,[]},{replica,[]},{ignore,[]}] <br/> [views:info] [2012-03-28 17:32:54] [<a href='mailto:ns_1@10.230.58.221'>ns_1@10.230.58.221</a>:&#39;capi_set_view_manager-qa&#39;:capi_set_view_manager:apply_map:450] Classified vbuckets for qa: <br/> Active: [] <br/> Passive: [] <br/> Cleanup: [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85] <br/> Replica: [] <br/> ReplicaCleanup: [] <br/> [couchdb:info] [2012-03-28 17:32:55] [<a href='mailto:ns_1@10.230.58.221'>ns_1@10.230.58.221</a>:&#39;capi_set_view_manager-qa&#39;:couch_log:error:42] MC daemon: Error opening vb 0 in &lt;&lt;&quot;qa&quot;&gt;&gt;: {not_found, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;no_db_file} <br/> [couchdb:info] [2012-03-28 17:32:55] [<a href='mailto:ns_1@10.230.58.221'>ns_1@10.230.58.221</a>:&#39;capi_set_view_manager-qa&#39;:couch_log:error:42] MC daemon: Error opening vb 1 in &lt;&lt;&quot;qa&quot;&gt;&gt;: {not_found, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;no_db_file} <br/> <br/> <br/> Isn&#39;t there a known issue with bucket flush not working ok? <br/> <br/> <br/> Also seeing this (ns_server) repeated many many times: <br/> <br/> {error,rest_error, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;Could not connect to \&quot;10.230.58.37\&quot; on port 8091. This could be due to an incorrect host/port combination or a firewall in place between the servers.&quot;&gt;&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{error,econnrefused}} <br/> <br/> [cluster:debug] [2012-03-26 17:13:14] [<a href='mailto:ns_1@10.230.58.221'>ns_1@10.230.58.221</a>:ns_cluster:ns_cluster:handle_call:120] add_node(&quot;10.230.58.37&quot;, 8091, ..) -&gt; {error, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;engage_cluster, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&lt;&lt;&quot;Prepare join failed. Could not connect to \&quot;10.230.58.37\&quot; on port 8091. This could be due to an incorrect host/port combination or a firewall in place between the servers.&quot;&gt;&gt;, <br/> <br/> <br/> And finally a file permission error when trying to open a vbucket database file: <br/> <br/> [couchdb:info] [2012-03-27 12:29:08] [<a href='mailto:ns_1@10.230.58.221'>ns_1@10.230.58.221</a>:&lt;0.9685.158&gt;:couch_log:error:42] Set view `test`, main group `_design/dev_appsByCategory`, doc loader error <br/> error: {case_clause,{error,eacces}} <br/> stacktrace: [{couch_db,fast_reads,2}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_set_view_updater,&#39;-load_changes/7-fun-2-&#39;,8}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{lists,foldl,3}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_set_view_updater,load_changes,7}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_set_view_updater,&#39;-update/7-fun-4-&#39;,10}] <br/> <br/> <br/> The later can be kind of fixed so that it doesn&#39;t hard crash, but it won&#39;t solve the issue. please try out the latest dp4 refresh build available <a href="http://www.couchbase.com/download">http://www.couchbase.com/download</a> Date of First Response Wed, 28 Mar 2012 15:31:10 -0500 Rank 5240 Time In Status [MB-4946] View binary documents shows "There was an unexpected error" on the UI http://www.couchbase.com/issues/browse/MB-4946 Couchbase Server Insert documnets vis mcsoda and json=0: <br/> pytests/performance/mcsoda.py <a href='mailto:default@localhost'>default@localhost</a> json=1 max-creates=1000000 ratio-creates=1.0 ratio-sets=1 json=0 <br/> <br/> go to &quot;data buckets&quot;-&gt;Documents, select a random document shows &quot;There was an unexpected error&quot;. <br/> <br/> getting the same document via telnet shows: <br/> get 0000104cd168386a <br/> VALUE 0000104cd168386a 0 237 <br/> *&quot;key&quot;:&quot;0000104cd168386a&quot;, <br/> &nbsp;&quot;key_num&quot;:848775, <br/> &nbsp;&quot;name&quot;:&quot;0000 386&quot;, <br/> &nbsp;&quot;email&quot;:&quot;<a href='mailto:0000@01.com'>0000@01.com</a>&quot;, <br/> &nbsp;&quot;city&quot;:&quot;104&quot;, <br/> &nbsp;&quot;country&quot;:&quot;cd&quot;, <br/> &nbsp;&quot;realm&quot;:&quot;16&quot;, <br/> &nbsp;&quot;coins&quot;:0.0, <br/> &nbsp;&quot;achievements&quot;:[16, 25, 85, 173, 209, 47],&quot;body&quot;:&quot;xc4ca4238a0b923820dcc509a6f75849b&quot;} <br/> END <br/> <br/> MB-4946 View binary documents shows "There was an unexpected error" on the UI Bug Blocker Resolved Fixed Public Aleksey Kondratenko Sharon Barr Thu, 22 Mar 2012 18:29:52 -0500 Wed, 12 Sep 2012 16:24:08 -0500 Mon, 27 Aug 2012 16:39:24 -0500 2.0-developer-preview-4 2.0-beta UI 0 1 there&#39;s a duplicate somewhere. CAPI CRUD is currently quite broken for non-json values AFAIK Remeber, this is a pure k/v object, it&#39;s non json. Notice the &quot;*&quot; as the first character of the value. i would guess that the UI tries to parse it as json. Let&#39;s investigate once we understand how to display non-JSON docs on the UI. Chris, can you add this to your design document and come up with a recommendation. The current state (after my patch is applied) is that binary documents show up as the raw byte array and are not editable (only view-able). <br/> <br/> I would prefer that the line break is not after each line number (perhaps at 40 chars wide instead...) <br/> <br/> And it would be cool if editing worked. That&#39;s the sort of clean up I hope we get to do once my changes are in. <br/> Hm. What if binary value is 20 megs ?<br/> <br/> Dipti said she is ok about not displaying binaries at all. Alk, given that Dipti gave her ok, let&#39;s proceed with this simple approach to not display binary documents. Since we landed Chris&#39;s stuff we&#39;re not crashing anymore. I&#39;ll create separate bug to hide blobs on UI and not avoid sending them back via CRUD API Date of First Response Thu, 22 Mar 2012 18:32:58 -0500 Rank 5254 Time In Status [MB-4854] Change the default data path for views and database to /opt/couchbase/var/lib/couchbase http://www.couchbase.com/issues/browse/MB-4854 Couchbase Server MB-4854 Change the default data path for views and database to /opt/couchbase/var/lib/couchbase Bug Blocker Resolved Fixed Public Aleksey Kondratenko Sharon Barr Wed, 29 Feb 2012 20:32:09 -0600 Tue, 3 Jul 2012 15:45:35 -0500 Wed, 25 Apr 2012 14:19:26 -0500 2.0-developer-preview-4 2.0-beta UI 0 1 This is likely a better fit for one of the Aleksey&#39;s as it&#39;s not set in the UI, but served up via the REST API and only displayed to the user by the UI. Integrated in github-ns-server-2-0 #338 (See [<a href="http://qa.hq.northscale.net/job/github-ns-server-2-0/338/">http://qa.hq.northscale.net/job/github-ns-server-2-0/338/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;changed default db dir to couchbase data directory. <a href="http://www.couchbase.com/issues/browse/MB-4854" title="Change the default data path for views and database to /opt/couchbase/var/lib/couchbase"><strike>MB-4854</strike></a> (Revision 35ec3078084d6dfce511b8f3c108b4ae2d6449ab)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> Srinivas Vadlamani : <br/> Files : <br/> * Makefile<br/> * etc/capi.ini.in<br/> Date of First Response Wed, 4 Apr 2012 14:20:45 -0500 Rank 5625 Time In Status [MB-6413] 7 empty buckets is nearly killing server [was: Couchbase causes unreasonable high CPU utilization without any connected clients] http://www.couchbase.com/issues/browse/MB-6413 Couchbase Server After setup the Couchbase and creating buckets the load increases to 100% (beam.smp and memcahed causing this, see screenshots). <br/> <br/> All the time there are no clientd connected to this instance. All load was caused from config tasks in the web interface. <br/> <br/> I put also a diagnostic report inside the attachment. <br/> <br/> There where no reports in dmesg output or anything else on system level. Ubuntu 11.10, 12.04 LTS and CentOS 5.3 - Just plain setup and updated. <br/> <br/> For Ubuntu we&#39;re used this package: <br/> couchbase-server-community_x86_64_2.0.0-1554-rel.deb MB-6413 7 empty buckets is nearly killing server [was: Couchbase causes unreasonable high CPU utilization without any connected clients] Bug Blocker Resolved Fixed Public Aleksey Kondratenko Alexander Finger Fri, 24 Aug 2012 04:37:08 -0500 Thu, 6 Sep 2012 15:11:04 -0500 Thu, 6 Sep 2012 15:11:04 -0500 recent-builds-2.0 2.0-beta build couchbase-bucket performance 0 2 Thanks for filing this. I&#39;ll take a look Thank you very much for reporting this.<br/> <br/> I&#39;m able to reproduce this on my box. This seems to be caused by larger than average number of buckets you have.<br/> <br/> For now consider lowering number of buckets to 2-3 as a workaround We&#39;re indeed consuming more that 20% of CPU gathering couch stats if there&#39;s 7 buckets on my quite quick hardware. On slower box that indeed can be quite a bit higher. moving this to 2.0 GA Integrated in github-ns-server-2-0 #459 (See [<a href="http://qa.hq.northscale.net/job/github-ns-server-2-0/459/">http://qa.hq.northscale.net/job/github-ns-server-2-0/459/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.couchbase.com/issues/browse/MB-6413" title="7 empty buckets is nearly killing server [was: Couchbase causes unreasonable high CPU utilization without any connected clients]"><strike>MB-6413</strike></a>: re-implemented bucket_databases using efficient API (Revision 8ba83d2df2b35ebd8fbc621dcad193aec2255fe5)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> pwansch : <br/> Files : <br/> * src/capi_set_view_manager.erl<br/> * src/ns_storage_conf.erl<br/> * src/compaction_daemon.erl<br/> Integrated in github-couchdb-preview #502 (See [<a href="http://qa.hq.northscale.net/job/github-couchdb-preview/502/">http://qa.hq.northscale.net/job/github-couchdb-preview/502/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.couchbase.com/issues/browse/MB-6413" title="7 empty buckets is nearly killing server [was: Couchbase causes unreasonable high CPU utilization without any connected clients]"><strike>MB-6413</strike></a> More reliable server startup when testing (Revision 9419f75d63ac4df9222df2d55d64991ea5efbb0d)<br/> <a href="http://www.couchbase.com/issues/browse/MB-6413" title="7 empty buckets is nearly killing server [was: Couchbase causes unreasonable high CPU utilization without any connected clients]"><strike>MB-6413</strike></a>: Fix race condition in test 12-errors.t (Revision 39ae383c157c8c6e0df11baf8522540655bbb9cc)<br/> <a href="http://www.couchbase.com/issues/browse/MB-6413" title="7 empty buckets is nearly killing server [was: Couchbase causes unreasonable high CPU utilization without any connected clients]"><strike>MB-6413</strike></a>: open all databases on startup (Revision 7db41d630c782db9eee6cd9e74b3b45324bcf463)<br/> <a href="http://www.couchbase.com/issues/browse/MB-6413" title="7 empty buckets is nearly killing server [was: Couchbase causes unreasonable high CPU utilization without any connected clients]"><strike>MB-6413</strike></a>: re-implemented all_known_databases/0 (Revision c4ba56ae22ec01bdd3453450f56a4a6e4ece6edb)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> pwansch : <br/> Files : <br/> * src/couchdb/couch_server_sup.erl<br/> * src/couch_set_view/test/couch_set_view_test_util.erl<br/> * src/couchdb/couch_server.erl<br/> <br/> pwansch : <br/> Files : <br/> * src/couch_set_view/test/12-errors.t<br/> <br/> pwansch : <br/> Files : <br/> * src/couchdb/couch_server.erl<br/> <br/> pwansch : <br/> Files : <br/> * src/couchdb/couch_server.erl<br/> We&#39;re still far from being good enough on multiple buckets, but worst problem was recently fixed. We were quadratic on total number of vbuckets (from all buckets) on certain periodic operations and that&#39;s now fixed and 7 buckets is far from being as bad as it was Date of First Response Fri, 24 Aug 2012 13:07:44 -0500 Rank 4192 Time In Status [MB-6309] Query throughput significantly drops after rebalance http://www.couchbase.com/issues/browse/MB-6309 Couchbase Server See attachments for details. Both physical and virtual machines, CentOS 6.2(6.3). MB-6309 Query throughput significantly drops after rebalance Bug Blocker Resolved Cannot Reproduce Public Pavel Paulau Pavel Paulau Sun, 19 Aug 2012 13:58:14 -0500 Wed, 22 Aug 2012 00:12:38 -0500 Wed, 22 Aug 2012 00:12:38 -0500 recent-builds-2.0 2.0-beta view-engine 0 2 Double check if replica indexes are still being updated. I&#39;ve seen them to be built long after rebalance is done. @Pavel, for how long did the test performed queries after rebalance? <br/> <br/> One theory I have is that because rebalance keeps aborting compactions while it&#39;s in progress, the view files grew a lot, resulting in too unbalanced btrees and greater disk seek distances between btree nodes. <br/> It&#39;s perhaps useful to start attaching all logs from all nodes as well. <br/> <br/> thanks The scenario: <br/> -- Do 4M queries (takes ~ 40 minutes in both cases) <br/> -- Rebalance from 3 to 4 nodes (cisco build is almost 2 times faster) <br/> -- Do 4M queries (~1 hour build 1505, ~5 hours build 704) <br/> <br/> Logs will be attached later. View data/disk sizes after rebalance look strange to me.<br/> <br/> Especially decreasing data size. That&#39;s not really strange, data size corresponds to the size of user data + all supporting metadata (btree nodes, checksums, etc). Compactions rebalances btrees and results in everything going into different file offsets, so it&#39;s not really unexpected. <br/> <br/> Same applies to data size of databases. Yes, that is expected. <br/> <br/> But when I (as user) rebalance cluster from 3 nodes to 4 nodes I expect rather equal disk usage on first 3 nodes after rebalance. <br/> 62GB-68GB-6GB-0GB is not something I expect. Please, attach the server logs from all nodes. <br/> Only having the report pdf is not that useful. <br/> <br/> Technically the view engine has no knowledge of rebalances, only ns_server knows about that is triggers/aborts view compactions. <br/> <br/> If you don&#39;t have the logs anymore, just retest and save the new logs. I tried to restart the test but it fails because of insufficient disk space. So I can&#39;t gather server logs. <br/> <br/> I&#39;ll collect all required info once we upgrade server configuration. I surprised that test managed to finish first time. Can&#39;t reproduce in recent builds. Date of First Response Sun, 19 Aug 2012 14:00:30 -0500 Rank 4287 Time In Status [MB-6096] Index building is seemingly taking too much ram http://www.couchbase.com/issues/browse/MB-6096 Couchbase Server I have a crash dump that shows that simple index build over 1.2 million of small docs eats tons of ram. I&#39;m going to attach erlang crash dump where you can see second biggest process (0.7277.2 or sort by stack+heap) is index updater. <br/> <br/> NOTE: crash itself was caused by attempt to grab diag so biggest process died because of that, but real cause seems to be index building. MB-6096 Index building is seemingly taking too much ram Bug Blocker Resolved Fixed Public Filipe Manana Aleksey Kondratenko Thu, 2 Aug 2012 13:53:45 -0500 Thu, 16 Aug 2012 09:36:33 -0500 Thu, 16 Aug 2012 09:36:12 -0500 recent-builds-2.0 2.0-beta performance storage-engine view-engine 0 0 Crash dump can be downloaded here: <a href="https://docs.google.com/open?id=0B6NtGsLhIcf7QVpwVldNcVlZdkk">https://docs.google.com/open?id=0B6NtGsLhIcf7QVpwVldNcVlZdkk</a> Integrated in github-couchdb-preview #474 (See [<a href="http://qa.hq.northscale.net/job/github-couchdb-preview/474/">http://qa.hq.northscale.net/job/github-couchdb-preview/474/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.couchbase.com/issues/browse/MB-6096" title="Index building is seemingly taking too much ram"><strike>MB-6096</strike></a> Fix too high memory consumption during indexing (Revision c8003e55b9dc8edbee51e5d047c0317aaf192560)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> Filipe David Borba Manana : <br/> Files : <br/> * src/mapreduce/mapreduce.h<br/> * src/couchdb/couch_util.erl<br/> * src/couchdb/couch_db_updater.erl<br/> * src/mapreduce/mapreduce.cc<br/> * src/couch_set_view/src/couch_set_view_util.erl<br/> * src/couchdb/couch_doc.erl<br/> * src/couchdb/couch_btree.erl<br/> * src/couch_set_view/src/couch_set_view_updater.erl<br/> * src/mapreduce/mapreduce_nif.cc<br/> * src/mapreduce/mapreduce.erl<br/> Integrated in github-couchdb-preview #475 (See [<a href="http://qa.hq.northscale.net/job/github-couchdb-preview/475/">http://qa.hq.northscale.net/job/github-couchdb-preview/475/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.couchbase.com/issues/browse/MB-6096" title="Index building is seemingly taking too much ram"><strike>MB-6096</strike></a> Use enif_alloc()/enif_free() in ejson decoder (Revision 2c4b88c46e1ce4500d987552e600eaa200546fe6)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> Filipe David Borba Manana : <br/> Files : <br/> * src/ejson/decode.c<br/> Re-opened to assign to myself, since I did all the investigation and code work Closing again. Date of First Response Fri, 10 Aug 2012 13:28:51 -0500 Rank 4470 Story Points 2.0 Time In Status [MB-5893] Installed build-1416 on 3 node cluster, vbucket stats is showing unusual values for # of buckets, breaking client libraries http://www.couchbase.com/issues/browse/MB-5893 Couchbase Server Looking at 3 of the buckets on this server, I see the following values for total # of vbuckets: <br/> <br/> default - 1.36K <br/> events - 682 <br/> profiles - 0 <br/> <br/> Attached are screenshots of these numbers, as well as server diagnostics. <br/> <br/> This ultimately causes a problem in the ruby client library: <br/> <br/> irb(main):021:0* Article.ensure_design_document! <br/> Couchbase::Error::Protocol: Number of buckets must be a power of two &gt; 0 and &lt;= MAX_BUCKETS (error=0x15) <br/> from /app/vendor/bundle/ruby/1.9.1/gems/couchbase-1.2.0.dp5/lib/couchbase.rb:51:in `initialize&#39; <br/> from /app/vendor/bundle/ruby/1.9.1/gems/couchbase-1.2.0.dp5/lib/couchbase.rb:51:in `new&#39; <br/> from /app/vendor/bundle/ruby/1.9.1/gems/couchbase-1.2.0.dp5/lib/couchbase.rb:51:in `connect&#39; <br/> from /app/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.4.0/lib/new_relic/agent/method_tracer.rb:491:in `block in connect_with_trace_Couchbase_connect&#39; <br/> from /app/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.4.0/lib/new_relic/agent/method_tracer.rb:242:in `trace_execution_scoped&#39; <br/> from /app/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.4.0/lib/new_relic/agent/method_tracer.rb:486:in `connect_with_trace_Couchbase_connect&#39; <br/> from /app/vendor/bundle/ruby/1.9.1/gems/couchbase-1.2.0.dp5/lib/couchbase.rb:82:in `bucket&#39; <br/> from /app/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.4.0/lib/new_relic/agent/method_tracer.rb:491:in `block in bucket_with_trace_Couchbase_bucket&#39; <br/> from /app/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.4.0/lib/new_relic/agent/method_tracer.rb:242:in `trace_execution_scoped&#39; <br/> from /app/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.4.0/lib/new_relic/agent/method_tracer.rb:486:in `bucket_with_trace_Couchbase_bucket&#39; <br/> from /app/vendor/bundle/ruby/1.9.1/bundler/gems/ruby-couchbase-model-b52f33e7b93d/lib/couchbase/model.rb:532:in `bucket&#39; <br/> from /app/vendor/bundle/ruby/1.9.1/bundler/gems/ruby-couchbase-model-b52f33e7b93d/lib/couchbase/model.rb:210:in `ensure_design_document!&#39; <br/> from (irb):21 <br/> from /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.6/lib/rails/commands/console.rb:47:in `start&#39; <br/> from /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.6/lib/rails/commands/console.rb:8:in `start&#39; <br/> from /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.6/lib/rails/commands.rb:41:in `&lt;top (required)&gt;&#39; <br/> from script/rails:6:in `require&#39; <br/> from script/rails:6:in `&lt;main&gt;&#39; <br/> <br/> irb(main):022:0&gt; Couch.client.set(&quot;thing&quot;, {:one =&gt; &quot;1&quot;, :two =&gt; &quot;2&quot;, :three =&gt; &quot;3&quot;}) <br/> Couchbase::Error::Protocol: Number of buckets must be a power of two &gt; 0 and &lt;= MAX_BUCKETS (error=0x15) <br/> from /app/vendor/bundle/ruby/1.9.1/gems/couchbase-1.2.0.dp5/lib/couchbase.rb:51:in `initialize&#39; <br/> from /app/vendor/bundle/ruby/1.9.1/gems/couchbase-1.2.0.dp5/lib/couchbase.rb:51:in `new&#39; <br/> from /app/vendor/bundle/ruby/1.9.1/gems/couchbase-1.2.0.dp5/lib/couchbase.rb:51:in `connect&#39; <br/> from /app/app/models/couch.rb:20:in `client&#39; <br/> from (irb):22 <br/> from /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.6/lib/rails/commands/console.rb:47:in `start&#39; <br/> from /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.6/lib/rails/commands/console.rb:8:in `start&#39; <br/> from /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.6/lib/rails/commands.rb:41:in `&lt;top (required)&gt;&#39; <br/> from script/rails:6:in `require&#39; <br/> from script/rails:6:in `&lt;main&gt;&#39; <br/> irb(main):023:0&gt; <br/> Linux ip-10-80-63-134 2.6.35.14-97.44.amzn1.x86_64 #1 SMP Mon Oct 24 16:03:08 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux MB-5893 Installed build-1416 on 3 node cluster, vbucket stats is showing unusual values for # of buckets, breaking client libraries Bug Blocker Resolved Fixed Public Farshid Ghods Marty Schoch Thu, 12 Jul 2012 18:47:40 -0500 Tue, 24 Jul 2012 18:48:36 -0500 Tue, 24 Jul 2012 18:22:21 -0500 recent-builds-2.0 2.0-beta couchbase-bucket 0 3 some observations <br/> <br/> 1. in fact, vBucketServerMap conforms to expectations for 1024 vBuckets <a href="http://ec2-174-129-165-213.compute-1.amazonaws.com:8091/pools/default/saslBucketsStreaming">http://ec2-174-129-165-213.compute-1.amazonaws.com:8091/pools/default/saslBucketsStreaming</a> (see attached file) <br/> 2. there are a lot of errors in logs like as: <br/> <br/> memcached&lt;0.867.0&gt;: Total memory in memoryDeallocated() &gt;= GIGANTOR !!! Disable the memory tracker... <br/> <br/> [stats:error] [2012-07-12 19:20:20] [<a href='mailto:ns_1@10.80.63.134'>ns_1@10.80.63.134</a>:&lt;0.11015.22&gt;:stats_collector:handle_info:96] Exception in stats collector: {exit, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{timeout, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server,call, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{&#39;couch_stats_reader-views&#39;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&#39;<a href='mailto:ns_1@10.80.63.134'>ns_1@10.80.63.134</a>&#39;}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;fetch_stats]}}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[{gen_server,call,2}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{couch_stats_reader,fetch_stats,1}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{stats_collector,grab_all_stats,1}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{stats_collector,handle_info,2}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{gen_server,handle_msg,5}, <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;{proc_lib,init_p_do_apply,3}]} <br/> <br/> UI console displays CPU usage 100% (screenshot) <br/> <br/> 3. quite interesting logs <br/> <br/> <br/> memcached&lt;0.867.0&gt;: Connected to mccouch: &quot;localhost:11213&quot; <br/> memcached&lt;0.867.0&gt;: Failed to load mutation log, falling back to key dump <br/> memcached&lt;0.867.0&gt;: metadata loaded in 867 usec <br/> memcached&lt;0.867.0&gt;: warmup completed in 1003 usec <br/> memcached&lt;0.867.0&gt;: Extension support isn&#39;t implemented in this version of bucket_engine <br/> memcached&lt;0.867.0&gt;: Trying to connect to mccouch: &quot;localhost:11213&quot; <br/> memcached&lt;0.867.0&gt;: Connected to mccouch: &quot;localhost:11213&quot; <br/> memcached&lt;0.867.0&gt;: Failed to load mutation log, falling back to key dump <br/> memcached&lt;0.867.0&gt;: Extension support isn&#39;t implemented in this version of bucket_engine <br/> memcached&lt;0.867.0&gt;: metadata loaded in 7762 usec <br/> memcached&lt;0.867.0&gt;: warmup completed in 7883 usec <br/> There&#39;s evidence of massive CPU usage. But I&#39;m not sure what&#39;s eating it. Can somebody confirm it&#39;s erlang or memcached ? If erlang I&#39;d like another diag to try to see who is eating CPU in erlang. Here is the CPU as reported by top on the 3 boxes: <br/> <br/> &nbsp;&nbsp;PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <br/> 10556 couchbas 20 0 552m 198m 3280 S 191.5 2.7 3047:49 memcached <br/> 10499 couchbas 20 0 5854m 700m 9364 S 47.5 9.4 829:28.14 beam.smp <br/> <br/> &nbsp;&nbsp;PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <br/> 25340 couchbas 20 0 554m 199m 3228 S 220.1 2.7 3058:41 memcached <br/> 25281 couchbas 20 0 4681m 669m 9352 S 24.7 9.0 825:04.05 beam.smp <br/> <br/> &nbsp;&nbsp;PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND <br/> 26292 couchbas 20 0 544m 197m 3232 S 134.6 2.6 3043:41 memcached <br/> 26233 couchbas 20 0 5987m 764m 9324 S 59.5 10.2 840:53.10 beam.smp Nice. So that&#39;s ep-engine&#39;s problem then, not ns_server&#39;s. <br/> <br/> BTW somebody mentioned &#39;upgrade&#39;. Did you guys attempted online/offline upgrade between 2.0 builds ? This was a clean install. I was sure to kill all Couchbase processes and rm -rf /opt/couchbase before installing. After this scripts were rerun to set up buckets and load data. Marty, <br/> can you specify the details of this installation, it will help debug this. <br/> AFAIR, there were 6 buckets. was there any data in them, etc.. It is 3 nodes, each is an amazon m1.large instance. There are 6 buckets. Currently I only see data in one of the buckets. I&#39;ll try to get the main developers to comment on this issue with more details, but they are offline at the moment. looks like there few bucket creation/deletion happening and few bugs were fixed related to bucket deletion that memcached/ns-server were not cleaning up data files from disk. its just a guess . <br/> <br/> the memcached cpu percentage seems too high so looking at the logs again. <br/> <br/> Marty can you give me access to one of these vms ? and i think the memory underflow issue had sth to do with the bucket deletion/creation. <br/> <br/> assigning this to couchbase bucket team I&#39;ve added the QAkey provided by Farshid to the ssh authorized_keys. <br/> <br/> The hostnames of all the nodes are: <br/> <br/> ec2-174-129-165-213.compute-1.amazonaws.com <br/> ec2-23-22-51-21.compute-1.amazonaws.com <br/> ec2-184-73-64-246.compute-1.amazonaws.com backtrace from memcached: <br/> <br/> Thread 38 (Thread 0x7f28eef85700 (LWP 10565)): <br/> #0 0x00007f28ef77b9cd in read () from /lib64/libc.so.6 <br/> #1 0x00007f28ef714128 in _IO_new_file_underflow () from /lib64/libc.so.6 <br/> #2 0x00007f28ef715c2e in _IO_default_uflow_internal () from /lib64/libc.so.6 <br/> #3 0x00007f28ef71111b in getc () from /lib64/libc.so.6 <br/> #4 0x00007f28eef86879 in check_stdin_thread (arg=0x403420) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at extensions/daemon/stdin_check.c:19 <br/> #5 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #6 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 37 (Thread 0x7f28ee57a700 (LWP 10566)): <br/> #0 0x00007f28ef7892c3 in epoll_wait () from /lib64/libc.so.6 <br/> #1 0x00007f28f054cba6 in epoll_dispatch (base=0x4d58500, <br/> &nbsp;&nbsp;&nbsp;&nbsp;tv=&lt;value optimized out&gt;) at epoll.c:404 <br/> #2 0x00007f28f053bc11 in event_base_loop (base=0x4d58500, flags=0) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at event.c:1558 <br/> #3 0x00000000004139f4 in worker_libevent (arg=0x8f8500) at daemon/thread.c:301 <br/> #4 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #5 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 36 (Thread 0x7f28edd79700 (LWP 10567)): <br/> #0 0x00007f28ef770177 in sched_yield () from /lib64/libc.so.6 <br/> ---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit--- <br/> #1 0x00007f28eb1c2997 in SpinLock::acquire (this=0x602b5c0) at atomic.cc:35 <br/> #2 0x00007f28eb238d75 in lock (this=&lt;value optimized out&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;id=&lt;value optimized out&gt;) at atomic.hh:259 <br/> #3 SpinLockHolder (this=&lt;value optimized out&gt;, id=&lt;value optimized out&gt;) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at atomic.hh:251 <br/> #4 gimme (this=&lt;value optimized out&gt;, id=&lt;value optimized out&gt;) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at atomic.hh:371 <br/> #5 RCPtr (this=&lt;value optimized out&gt;, id=&lt;value optimized out&gt;) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at atomic.hh:309 <br/> #6 VBucketMap::getBucket (this=&lt;value optimized out&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;id=&lt;value optimized out&gt;) at vbucketmap.cc:32 <br/> <br/> <br/> <br/> #7 0x00007f28eb1d8438 in EventuallyPersistentStore::visit (this=0x6cf7b80, <br/> &nbsp;&nbsp;&nbsp;&nbsp;visitor=...) at ep.cc:2666 <br/> #8 0x00007f28eb1fd832 in EventuallyPersistentEngine::doEngineStats ( <br/> &nbsp;&nbsp;&nbsp;&nbsp;this=0xd3d3e00, cookie=0x4cb5340, add_stat=0x406080 &lt;append_stats&gt;) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at ep_engine.cc:2328 <br/> #9 0x00007f28eb1ff4b1 in EventuallyPersistentEngine::getStats ( <br/> &nbsp;&nbsp;&nbsp;&nbsp;this=0xd3d3e00, cookie=0x4cb5340, stat_key=0x0, nkey=0, <br/> &nbsp;&nbsp;&nbsp;&nbsp;add_stat=0x406080 &lt;append_stats&gt;) at ep_engine.cc:3315 <br/> #10 0x00007f28eb1ffd06 in EvpGetStats (handle=&lt;value optimized out&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;cookie=0x4cb5340, stat_key=0x0, nkey=0, add_stat=&lt;value optimized out&gt;) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at ep_engine.cc:185 <br/> #11 0x00007f28ee57e8b4 in bucket_get_stats (handle=&lt;value optimized out&gt;, <br/> ---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit--- <br/> &nbsp;&nbsp;&nbsp;&nbsp;cookie=0x4cb5340, stat_key=&lt;value optimized out&gt;, nkey=0, <br/> &nbsp;&nbsp;&nbsp;&nbsp;add_stat=0x406080 &lt;append_stats&gt;) at bucket_engine.c:1719 <br/> #12 0x000000000040c4da in process_bin_stat (c=0x4cb5340) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at daemon/memcached.c:2157 <br/> #13 0x0000000000410685 in complete_nread_binary (c=0x4cb5340) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at daemon/memcached.c:3707 <br/> #14 complete_nread (c=0x4cb5340) at daemon/memcached.c:3819 <br/> #15 conn_nread (c=0x4cb5340) at daemon/memcached.c:5667 <br/> #16 0x0000000000406a85 in event_handler (fd=&lt;value optimized out&gt;, which=2, <br/> &nbsp;&nbsp;&nbsp;&nbsp;arg=0x4cb5340) at daemon/memcached.c:5930 <br/> #17 0x00007f28f053bd0c in event_process_active_single_queue (base=0x4d58280, <br/> &nbsp;&nbsp;&nbsp;&nbsp;flags=0) at event.c:1308 <br/> #18 event_process_active (base=0x4d58280, flags=0) at event.c:1375 <br/> #19 event_base_loop (base=0x4d58280, flags=0) at event.c:1572 <br/> #20 0x00000000004139f4 in worker_libevent (arg=0x8f85f8) at daemon/thread.c:301 <br/> #21 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #22 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 35 (Thread 0x7f28ed578700 (LWP 10568)): <br/> #0 0x00007f28ef7892c3 in epoll_wait () from /lib64/libc.so.6 <br/> #1 0x00007f28f054cba6 in epoll_dispatch (base=0x4d58c80, <br/> &nbsp;&nbsp;&nbsp;&nbsp;tv=&lt;value optimized out&gt;) at epoll.c:404 <br/> #2 0x00007f28f053bc11 in event_base_loop (base=0x4d58c80, flags=0) <br/> ---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit--- <br/> &nbsp;&nbsp;&nbsp;&nbsp;at event.c:1558 <br/> #3 0x00000000004139f4 in worker_libevent (arg=0x8f86f0) at daemon/thread.c:301 <br/> #4 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #5 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 34 (Thread 0x7f28ecd77700 (LWP 10569)): <br/> #0 0x00007f28ef7892c3 in epoll_wait () from /lib64/libc.so.6 <br/> #1 0x00007f28f054cba6 in epoll_dispatch (base=0x4d58a00, <br/> &nbsp;&nbsp;&nbsp;&nbsp;tv=&lt;value optimized out&gt;) at epoll.c:404 <br/> #2 0x00007f28f053bc11 in event_base_loop (base=0x4d58a00, flags=0) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at event.c:1558 <br/> #3 0x00000000004139f4 in worker_libevent (arg=0x8f87e8) at daemon/thread.c:301 <br/> #4 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #5 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 33 (Thread 0x7f28ec576700 (LWP 10570)): <br/> #0 0x00007f28ef7892c3 in epoll_wait () from /lib64/libc.so.6 <br/> #1 0x00007f28f054cba6 in epoll_dispatch (base=0x4d58780, <br/> &nbsp;&nbsp;&nbsp;&nbsp;tv=&lt;value optimized out&gt;) at epoll.c:404 <br/> #2 0x00007f28f053bc11 in event_base_loop (base=0x4d58780, flags=0) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at event.c:1558 <br/> #3 0x00000000004139f4 in worker_libevent (arg=0x8f88e0) at daemon/thread.c:301 <br/> #4 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> ---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit--- <br/> #5 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 32 (Thread 0x7f28ead46700 (LWP 10572)): <br/> #0 0x00007f28ef74e15d in nanosleep () from /lib64/libc.so.6 <br/> #1 0x00007f28ef782124 in usleep () from /lib64/libc.so.6 <br/> #2 0x00007f28eb213b22 in updateStatsThread (arg=0x8f6480) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at memory_tracker.cc:31 <br/> #3 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #4 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 31 (Thread 0x7f28dbb1a700 (LWP 12563)): <br/> #0 0x00007fffb85ff901 in clock_gettime () <br/> #1 0x00007f28efe9ee46 in clock_gettime () from /lib64/librt.so.1 <br/> #2 0x00007f28eb23e411 in gethrtime () at gethrtime.c:77 <br/> #3 0x00007f28eb1d3f87 in Dispatcher::run (this=0xba46000) at dispatcher.cc:137 <br/> #4 0x00007f28eb1d47cb in launch_dispatcher_thread (arg=0xba46000) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at dispatcher.cc:28 <br/> #5 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #6 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 30 (Thread 0x7f28de31f700 (LWP 12564)): <br/> #0 0x00007f28efa3e3dc in pthread_cond_wait@@GLIBC_2.3.2 () <br/> &nbsp;&nbsp;&nbsp;from /lib64/libpthread.so.0 <br/> ---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit--- <br/> #1 0x00007f28eb1d4197 in wait (this=0x602f340) at syncobject.hh:36 <br/> #2 Dispatcher::run (this=0x602f340) at dispatcher.cc:89 <br/> #3 0x00007f28eb1d47cb in launch_dispatcher_thread (arg=0x602f340) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at dispatcher.cc:28 <br/> #4 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #5 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 29 (Thread 0x7f28deb20700 (LWP 12565)): <br/> #0 0x00007f28efa3e3dc in pthread_cond_wait@@GLIBC_2.3.2 () <br/> &nbsp;&nbsp;&nbsp;from /lib64/libpthread.so.0 <br/> #1 0x00007f28eb1d4197 in wait (this=0x602f500) at syncobject.hh:36 <br/> #2 Dispatcher::run (this=0x602f500) at dispatcher.cc:89 <br/> #3 0x00007f28eb1d47cb in launch_dispatcher_thread (arg=0x602f500) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at dispatcher.cc:28 <br/> #4 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #5 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 28 (Thread 0x7f28df321700 (LWP 12566)): <br/> #0 0x00007f28efa3e75b in pthread_cond_timedwait@@GLIBC_2.3.2 () <br/> &nbsp;&nbsp;&nbsp;from /lib64/libpthread.so.0 <br/> #1 0x00007f28eb1d1950 in wait (this=0x5253400, d=...) at syncobject.hh:47 <br/> #2 IdleTask::run (this=0x5253400, d=...) at dispatcher.cc:294 <br/> #3 0x00007f28eb1d3f40 in Dispatcher::run (this=0x602fa40) at dispatcher.cc:123 <br/> ---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit--- <br/> #4 0x00007f28eb1d47cb in launch_dispatcher_thread (arg=0x602fa40) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at dispatcher.cc:28 <br/> #5 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #6 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 27 (Thread 0x7f28ebd75700 (LWP 12567)): <br/> #0 0x00007f28eb1bd3a8 in EventuallyPersistentStore::getVBucket () <br/> &nbsp;&nbsp;&nbsp;from /opt/couchbase/lib/memcached/ep.so <br/> #1 0x00007f28eb1dabf3 in EventuallyPersistentStore::firePendingVBucketOps ( <br/> &nbsp;&nbsp;&nbsp;&nbsp;this=0x6cf9700) at ep.cc:749 <br/> #2 0x00007f28eb1f2808 in EventuallyPersistentEngine::notifyPendingConnections <br/> &nbsp;&nbsp;&nbsp;&nbsp;(this=0x4d71400) at ep_engine.cc:3424 <br/> #3 0x00007f28eb1f29b3 in EvpNotifyPendingConns (arg=0x4d71400) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at ep_engine.cc:1110 <br/> #4 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #5 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 26 (Thread 0x7f28ea337700 (LWP 12568)): <br/> #0 0x00007f28efa3e75b in pthread_cond_timedwait@@GLIBC_2.3.2 () <br/> &nbsp;&nbsp;&nbsp;from /lib64/libpthread.so.0 <br/> #1 0x00007f28eb1d1950 in wait (this=0x5252880, d=...) at syncobject.hh:47 <br/> #2 IdleTask::run (this=0x5252880, d=...) at dispatcher.cc:294 <br/> #3 0x00007f28eb1d3f40 in Dispatcher::run (this=0x602e700) at dispatcher.cc:123 <br/> ---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit--- <br/> #4 0x00007f28eb1d47cb in launch_dispatcher_thread (arg=0x602e700) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at dispatcher.cc:28 <br/> #5 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #6 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 25 (Thread 0x7f28e9b36700 (LWP 12569)): <br/> #0 0x00007f28efa3e3dc in pthread_cond_wait@@GLIBC_2.3.2 () <br/> &nbsp;&nbsp;&nbsp;from /lib64/libpthread.so.0 <br/> #1 0x00007f28eb1d4197 in wait (this=0x602fdc0) at syncobject.hh:36 <br/> #2 Dispatcher::run (this=0x602fdc0) at dispatcher.cc:89 <br/> #3 0x00007f28eb1d47cb in launch_dispatcher_thread (arg=0x602fdc0) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at dispatcher.cc:28 <br/> #4 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #5 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 24 (Thread 0x7f28e9335700 (LWP 12570)): <br/> #0 0x00007f28efa3e3dc in pthread_cond_wait@@GLIBC_2.3.2 () <br/> &nbsp;&nbsp;&nbsp;from /lib64/libpthread.so.0 <br/> #1 0x00007f28eb1d4197 in wait (this=0x602e000) at syncobject.hh:36 <br/> #2 Dispatcher::run (this=0x602e000) at dispatcher.cc:89 <br/> #3 0x00007f28eb1d47cb in launch_dispatcher_thread (arg=0x602e000) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at dispatcher.cc:28 <br/> #4 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> ---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit--- <br/> #5 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 23 (Thread 0x7f28e8b34700 (LWP 12571)): <br/> #0 0x00007f28efa3e75b in pthread_cond_timedwait@@GLIBC_2.3.2 () <br/> &nbsp;&nbsp;&nbsp;from /lib64/libpthread.so.0 <br/> #1 0x00007f28eb1d1950 in wait (this=0x5253300, d=...) at syncobject.hh:47 <br/> #2 IdleTask::run (this=0x5253300, d=...) at dispatcher.cc:294 <br/> #3 0x00007f28eb1d3f40 in Dispatcher::run (this=0x602efc0) at dispatcher.cc:123 <br/> #4 0x00007f28eb1d47cb in launch_dispatcher_thread (arg=0x602efc0) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at dispatcher.cc:28 <br/> #5 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #6 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 22 (Thread 0x7f28e8333700 (LWP 12572)): <br/> #0 0x00007f28efa3e75b in pthread_cond_timedwait@@GLIBC_2.3.2 () <br/> &nbsp;&nbsp;&nbsp;from /lib64/libpthread.so.0 <br/> #1 0x00007f28eb1f28cf in wait (this=0xd3d5200) at syncobject.hh:47 <br/> #2 wait (this=0xd3d5200) at syncobject.hh:63 <br/> #3 wait (this=0xd3d5200) at tapconnmap.hh:183 <br/> #4 EventuallyPersistentEngine::notifyPendingConnections (this=0xd3d5200) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at ep_engine.cc:3430 <br/> #5 0x00007f28eb1f29b3 in EvpNotifyPendingConns (arg=0xd3d5200) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at ep_engine.cc:1110 <br/> ---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit--- <br/> #6 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #7 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 21 (Thread 0x7f28e7b32700 (LWP 12574)): <br/> #0 0x00007fffb85ff901 in clock_gettime () <br/> #1 0x00007f28efe9ee46 in clock_gettime () from /lib64/librt.so.1 <br/> #2 0x00007f28eb23e411 in gethrtime () at gethrtime.c:77 <br/> #3 0x00007f28eb1d3ebd in Dispatcher::run (this=0x5d11c00) at dispatcher.cc:118 <br/> #4 0x00007f28eb1d47cb in launch_dispatcher_thread (arg=0x5d11c00) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at dispatcher.cc:28 <br/> #5 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #6 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 20 (Thread 0x7f28e7331700 (LWP 12575)): <br/> #0 0x00007f28efa3e3dc in pthread_cond_wait@@GLIBC_2.3.2 () <br/> &nbsp;&nbsp;&nbsp;from /lib64/libpthread.so.0 <br/> #1 0x00007f28eb1d4197 in wait (this=0x5d11dc0) at syncobject.hh:36 <br/> #2 Dispatcher::run (this=0x5d11dc0) at dispatcher.cc:89 <br/> #3 0x00007f28eb1d47cb in launch_dispatcher_thread (arg=0x5d11dc0) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at dispatcher.cc:28 <br/> #4 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #5 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> ---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit--- <br/> Thread 19 (Thread 0x7f28e6b30700 (LWP 12576)): <br/> #0 0x00007f28efa3e3dc in pthread_cond_wait@@GLIBC_2.3.2 () <br/> &nbsp;&nbsp;&nbsp;from /lib64/libpthread.so.0 <br/> #1 0x00007f28eb1d4197 in wait (this=0x602e1c0) at syncobject.hh:36 <br/> #2 Dispatcher::run (this=0x602e1c0) at dispatcher.cc:89 <br/> #3 0x00007f28eb1d47cb in launch_dispatcher_thread (arg=0x602e1c0) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at dispatcher.cc:28 <br/> #4 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #5 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 18 (Thread 0x7f28e632f700 (LWP 12577)): <br/> #0 0x00007f28efa3e75b in pthread_cond_timedwait@@GLIBC_2.3.2 () <br/> &nbsp;&nbsp;&nbsp;from /lib64/libpthread.so.0 <br/> #1 0x00007f28eb1d1950 in wait (this=0x6dc7980, d=...) at syncobject.hh:47 <br/> #2 IdleTask::run (this=0x6dc7980, d=...) at dispatcher.cc:294 <br/> #3 0x00007f28eb1d3f40 in Dispatcher::run (this=0x602fc00) at dispatcher.cc:123 <br/> #4 0x00007f28eb1d47cb in launch_dispatcher_thread (arg=0x602fc00) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at dispatcher.cc:28 <br/> #5 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #6 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 17 (Thread 0x7f28e5b2e700 (LWP 12578)): <br/> #0 0x00007f28efa40ff4 in __lll_lock_wait () from /lib64/libpthread.so.0 <br/> ---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit--- <br/> #1 0x00007f28efa3c328 in _L_lock_854 () from /lib64/libpthread.so.0 <br/> #2 0x00007f28efa3c1f7 in pthread_mutex_lock () from /lib64/libpthread.so.0 <br/> #3 0x00000000004133e4 in notify_io_complete (cookie=0x4cd8000, <br/> &nbsp;&nbsp;&nbsp;&nbsp;status=ENGINE_SUCCESS) at daemon/thread.c:485 <br/> <br/> #4 0x00007f28eb231539 in operator() (this=0x4dc2900) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at /usr/lib/gcc/x86_64-redhat-linux6E/4.4.6/../../../../include/c++/4.4.6/bits/stl_function.h:457 <br/> #5 operator() (this=0x4dc2900) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at /usr/lib/gcc/x86_64-redhat-linux6E/4.4.6/../../../../include/c++/4.4.6/backward/binders.h:153 <br/> #6 for_each&lt;std::_List_iterator&lt;void const*&gt;, std::binder2nd&lt;std::pointer_to_binary_function&lt;void const*, ENGINE_ERROR_CODE, void&gt; &gt; &gt; (this=0x4dc2900) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at /usr/lib/gcc/x86_64-redhat-linux6E/4.4.6/../../../../include/c++/4.4.6/bits/stl_algo.h:4200 <br/> #7 notifyIOComplete&lt;std::list&lt;void const*, std::allocator&lt;void const*&gt; &gt; &gt; ( <br/> &nbsp;&nbsp;&nbsp;&nbsp;this=0x4dc2900) at ep_engine.h:425 <br/> #8 TapConnMap::notifyIOThreadMain (this=0x4dc2900) at tapconnmap.cc:518 <br/> #9 0x00007f28eb1f27fc in EventuallyPersistentEngine::notifyPendingConnections <br/> &nbsp;&nbsp;&nbsp;&nbsp;(this=0xd3d3400) at ep_engine.cc:3423 <br/> #10 0x00007f28eb1f29b3 in EvpNotifyPendingConns (arg=0xd3d3400) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at ep_engine.cc:1110 <br/> #11 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #12 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> ---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit--- <br/> <br/> Thread 16 (Thread 0x7f28e532d700 (LWP 12581)): <br/> #0 0x00007fffb85ff901 in clock_gettime () <br/> #1 0x00007f28efe9ee46 in clock_gettime () from /lib64/librt.so.1 <br/> #2 0x00007f28eb23e411 in gethrtime () at gethrtime.c:77 <br/> #3 0x00007f28eb1d3f87 in Dispatcher::run (this=0x70adc00) at dispatcher.cc:137 <br/> #4 0x00007f28eb1d47cb in launch_dispatcher_thread (arg=0x70adc00) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at dispatcher.cc:28 <br/> #5 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #6 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 15 (Thread 0x7f28e4b2c700 (LWP 12582)): <br/> #0 0x00007f28efa3e3dc in pthread_cond_wait@@GLIBC_2.3.2 () <br/> &nbsp;&nbsp;&nbsp;from /lib64/libpthread.so.0 <br/> #1 0x00007f28eb1d4197 in wait (this=0x70ada40) at syncobject.hh:36 <br/> #2 Dispatcher::run (this=0x70ada40) at dispatcher.cc:89 <br/> #3 0x00007f28eb1d47cb in launch_dispatcher_thread (arg=0x70ada40) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at dispatcher.cc:28 <br/> #4 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #5 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 14 (Thread 0x7f28e432b700 (LWP 12583)): <br/> #0 0x00007f28efa3e3dc in pthread_cond_wait@@GLIBC_2.3.2 () <br/> ---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit--- <br/> &nbsp;&nbsp;&nbsp;from /lib64/libpthread.so.0 <br/> #1 0x00007f28eb1d4197 in wait (this=0x70ad880) at syncobject.hh:36 <br/> #2 Dispatcher::run (this=0x70ad880) at dispatcher.cc:89 <br/> #3 0x00007f28eb1d47cb in launch_dispatcher_thread (arg=0x70ad880) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at dispatcher.cc:28 <br/> #4 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #5 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 13 (Thread 0x7f28e3b2a700 (LWP 12584)): <br/> #0 0x00007f28efa3e75b in pthread_cond_timedwait@@GLIBC_2.3.2 () <br/> &nbsp;&nbsp;&nbsp;from /lib64/libpthread.so.0 <br/> #1 0x00007f28eb1d1950 in wait (this=0x51a8000, d=...) at syncobject.hh:47 <br/> #2 IdleTask::run (this=0x51a8000, d=...) at dispatcher.cc:294 <br/> #3 0x00007f28eb1d3f40 in Dispatcher::run (this=0x70ad6c0) at dispatcher.cc:123 <br/> #4 0x00007f28eb1d47cb in launch_dispatcher_thread (arg=0x70ad6c0) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at dispatcher.cc:28 <br/> #5 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #6 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 12 (Thread 0x7f28e3329700 (LWP 12585)): <br/> #0 0x00007f28efa3b2c0 in pthread_self () from /lib64/libpthread.so.0 <br/> #1 0x00007f28eb214a69 in setHolder (this=0xd467d78) at mutex.hh:44 <br/> #2 Mutex::acquire (this=0xd467d78) at mutex.cc:75 <br/> ---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit--- <br/> #3 0x00007f28eb235c08 in lock (this=0xd467b00, engine=...) at locks.hh:48 <br/> #4 LockHolder (this=0xd467b00, engine=...) at locks.hh:26 <br/> #5 VBucket::fireAllOps (this=0xd467b00, engine=...) at vbucket.cc:114 <br/> #6 0x00007f28eb1dac09 in EventuallyPersistentStore::firePendingVBucketOps ( <br/> &nbsp;&nbsp;&nbsp;&nbsp;this=0x6cf8100) at ep.cc:751 <br/> #7 0x00007f28eb1f2808 in EventuallyPersistentEngine::notifyPendingConnections <br/> &nbsp;&nbsp;&nbsp;&nbsp;(this=0xd3d4800) at ep_engine.cc:3424 <br/> #8 0x00007f28eb1f29b3 in EvpNotifyPendingConns (arg=0xd3d4800) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at ep_engine.cc:1110 <br/> #9 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #10 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 11 (Thread 0x7f28e2b28700 (LWP 12587)): <br/> #0 0x00007f28efa3e75b in pthread_cond_timedwait@@GLIBC_2.3.2 () <br/> &nbsp;&nbsp;&nbsp;from /lib64/libpthread.so.0 <br/> #1 0x00007f28eb1d1950 in wait (this=0x51b4f00, d=...) at syncobject.hh:47 <br/> #2 IdleTask::run (this=0x51b4f00, d=...) at dispatcher.cc:294 <br/> #3 0x00007f28eb1d3f40 in Dispatcher::run (this=0x7584e00) at dispatcher.cc:123 <br/> #4 0x00007f28eb1d47cb in launch_dispatcher_thread (arg=0x7584e00) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at dispatcher.cc:28 <br/> #5 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #6 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> ---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit--- <br/> Thread 10 (Thread 0x7f28e2327700 (LWP 12588)): <br/> #0 0x00007f28efa3e3dc in pthread_cond_wait@@GLIBC_2.3.2 () <br/> &nbsp;&nbsp;&nbsp;from /lib64/libpthread.so.0 <br/> #1 0x00007f28eb1d4197 in wait (this=0x7584c40) at syncobject.hh:36 <br/> #2 Dispatcher::run (this=0x7584c40) at dispatcher.cc:89 <br/> #3 0x00007f28eb1d47cb in launch_dispatcher_thread (arg=0x7584c40) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at dispatcher.cc:28 <br/> #4 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #5 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 9 (Thread 0x7f28e1b26700 (LWP 12589)): <br/> #0 0x00007f28efa3e3dc in pthread_cond_wait@@GLIBC_2.3.2 () <br/> &nbsp;&nbsp;&nbsp;from /lib64/libpthread.so.0 <br/> #1 0x00007f28eb1d4197 in wait (this=0x7584a80) at syncobject.hh:36 <br/> #2 Dispatcher::run (this=0x7584a80) at dispatcher.cc:89 <br/> #3 0x00007f28eb1d47cb in launch_dispatcher_thread (arg=0x7584a80) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at dispatcher.cc:28 <br/> #4 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #5 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 8 (Thread 0x7f28e1325700 (LWP 12590)): <br/> #0 0x00007f28efa3e75b in pthread_cond_timedwait@@GLIBC_2.3.2 () <br/> &nbsp;&nbsp;&nbsp;from /lib64/libpthread.so.0 <br/> ---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit--- <br/> #1 0x00007f28eb1d1950 in wait (this=0x51a7600, d=...) at syncobject.hh:47 <br/> #2 IdleTask::run (this=0x51a7600, d=...) at dispatcher.cc:294 <br/> #3 0x00007f28eb1d3f40 in Dispatcher::run (this=0x75848c0) at dispatcher.cc:123 <br/> #4 0x00007f28eb1d47cb in launch_dispatcher_thread (arg=0x75848c0) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at dispatcher.cc:28 <br/> #5 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #6 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 7 (Thread 0x7f28e0b24700 (LWP 12591)): <br/> #0 0x00007f28efa40ff4 in __lll_lock_wait () from /lib64/libpthread.so.0 <br/> #1 0x00007f28efa3c328 in _L_lock_854 () from /lib64/libpthread.so.0 <br/> #2 0x00007f28efa3c1f7 in pthread_mutex_lock () from /lib64/libpthread.so.0 <br/> #3 0x00000000004133e4 in notify_io_complete (cookie=0x4c68000, <br/> &nbsp;&nbsp;&nbsp;&nbsp;status=ENGINE_SUCCESS) at daemon/thread.c:485 <br/> #4 0x00007f28eb231539 in operator() (this=0x4dc2240) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at /usr/lib/gcc/x86_64-redhat-linux6E/4.4.6/../../../../include/c++/4.4.6/bits/stl_function.h:457 <br/> #5 operator() (this=0x4dc2240) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at /usr/lib/gcc/x86_64-redhat-linux6E/4.4.6/../../../../include/c++/4.4.6/backward/binders.h:153 <br/> #6 for_each&lt;std::_List_iterator&lt;void const*&gt;, std::binder2nd&lt;std::pointer_to_binary_function&lt;void const*, ENGINE_ERROR_CODE, void&gt; &gt; &gt; (this=0x4dc2240) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at /usr/lib/gcc/x86_64-redhat-linux6E/4.4.6/../../../../include/c++/4.4.6/bi---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit--- <br/> ts/stl_algo.h:4200 <br/> #7 notifyIOComplete&lt;std::list&lt;void const*, std::allocator&lt;void const*&gt; &gt; &gt; ( <br/> &nbsp;&nbsp;&nbsp;&nbsp;this=0x4dc2240) at ep_engine.h:425 <br/> #8 TapConnMap::notifyIOThreadMain (this=0x4dc2240) at tapconnmap.cc:518 <br/> #9 0x00007f28eb1f27fc in EventuallyPersistentEngine::notifyPendingConnections <br/> &nbsp;&nbsp;&nbsp;&nbsp;(this=0xd3d2a00) at ep_engine.cc:3423 <br/> #10 0x00007f28eb1f29b3 in EvpNotifyPendingConns (arg=0xd3d2a00) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at ep_engine.cc:1110 <br/> #11 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #12 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 6 (Thread 0x7f28e0323700 (LWP 12593)): <br/> #0 0x00007f28efa3e75b in pthread_cond_timedwait@@GLIBC_2.3.2 () <br/> &nbsp;&nbsp;&nbsp;from /lib64/libpthread.so.0 <br/> #1 0x00007f28eb1d1950 in wait (this=0x6d45980, d=...) at syncobject.hh:47 <br/> #2 IdleTask::run (this=0x6d45980, d=...) at dispatcher.cc:294 <br/> #3 0x00007f28eb1d3f40 in Dispatcher::run (this=0x70ac700) at dispatcher.cc:123 <br/> #4 0x00007f28eb1d47cb in launch_dispatcher_thread (arg=0x70ac700) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at dispatcher.cc:28 <br/> #5 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #6 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 5 (Thread 0x7f28dfb22700 (LWP 12594)): <br/> ---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit--- <br/> #0 0x00007f28efa3e3dc in pthread_cond_wait@@GLIBC_2.3.2 () <br/> &nbsp;&nbsp;&nbsp;from /lib64/libpthread.so.0 <br/> #1 0x00007f28eb1d4197 in wait (this=0x70ac540) at syncobject.hh:36 <br/> #2 Dispatcher::run (this=0x70ac540) at dispatcher.cc:89 <br/> #3 0x00007f28eb1d47cb in launch_dispatcher_thread (arg=0x70ac540) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at dispatcher.cc:28 <br/> #4 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #5 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 4 (Thread 0x7f28ddb1e700 (LWP 12595)): <br/> #0 0x00007f28efa3e3dc in pthread_cond_wait@@GLIBC_2.3.2 () <br/> &nbsp;&nbsp;&nbsp;from /lib64/libpthread.so.0 <br/> #1 0x00007f28eb1d4197 in wait (this=0x70ac380) at syncobject.hh:36 <br/> #2 Dispatcher::run (this=0x70ac380) at dispatcher.cc:89 <br/> #3 0x00007f28eb1d47cb in launch_dispatcher_thread (arg=0x70ac380) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at dispatcher.cc:28 <br/> #4 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #5 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 3 (Thread 0x7f28dd31d700 (LWP 12596)): <br/> #0 get (this=&lt;value optimized out&gt;, id=&lt;value optimized out&gt;) at atomic.hh:86 <br/> #1 operator VBucket* (this=&lt;value optimized out&gt;, id=&lt;value optimized out&gt;) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at atomic.hh:95 <br/> ---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit--- <br/> #2 gimme (this=&lt;value optimized out&gt;, id=&lt;value optimized out&gt;) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at atomic.hh:375 <br/> #3 RCPtr (this=&lt;value optimized out&gt;, id=&lt;value optimized out&gt;) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at atomic.hh:309 <br/> #4 VBucketMap::getBucket (this=&lt;value optimized out&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;id=&lt;value optimized out&gt;) at vbucketmap.cc:32 <br/> #5 0x00007f28eb1d877a in VBCBAdaptor::VBCBAdaptor (this=0x4de0e10, <br/> &nbsp;&nbsp;&nbsp;&nbsp;s=&lt;value optimized out&gt;, v=&lt;value optimized out&gt;, l=&lt;value optimized out&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;sleep=1.0948012503771682e-315) at ep.cc:2737 <br/> <br/> #6 0x00007f28eb1cfdd2 in visit (this=0x5315cb0, d=..., t=...) at ep.hh:762 <br/> #7 ClosedUnrefCheckpointRemover::callback (this=0x5315cb0, d=..., t=...) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at checkpoint_remover.cc:63 <br/> <br/> #8 0x00007f28eb1d508f in Task::run(Dispatcher&amp;, std::tr1::shared_ptr&lt;Task&gt;) () <br/> &nbsp;&nbsp;&nbsp;from /opt/couchbase/lib/memcached/ep.so <br/> #9 0x00007f28eb1d3f40 in Dispatcher::run (this=0x70ac1c0) at dispatcher.cc:123 <br/> #10 0x00007f28eb1d47cb in launch_dispatcher_thread (arg=0x70ac1c0) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at dispatcher.cc:28 <br/> #11 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #12 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 2 (Thread 0x7f28dcb1c700 (LWP 12597)): <br/> #0 0x00007f28efa3e75b in pthread_cond_timedwait@@GLIBC_2.3.2 () <br/> &nbsp;&nbsp;&nbsp;from /lib64/libpthread.so.0 <br/> ---Type &lt;return&gt; to continue, or q &lt;return&gt; to quit--- <br/> #1 0x00007f28eb1f28cf in wait (this=0xd3d3e00) at syncobject.hh:47 <br/> #2 wait (this=0xd3d3e00) at syncobject.hh:63 <br/> #3 wait (this=0xd3d3e00) at tapconnmap.hh:183 <br/> #4 EventuallyPersistentEngine::notifyPendingConnections (this=0xd3d3e00) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at ep_engine.cc:3430 <br/> #5 0x00007f28eb1f29b3 in EvpNotifyPendingConns (arg=0xd3d3e00) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at ep_engine.cc:1110 <br/> #6 0x00007f28efa3a7f1 in start_thread () from /lib64/libpthread.so.0 <br/> #7 0x00007f28ef788ccd in clone () from /lib64/libc.so.6 <br/> <br/> Thread 1 (Thread 0x7f28f0b82720 (LWP 10556)): <br/> #0 0x00007f28ef7892c3 in epoll_wait () from /lib64/libc.so.6 <br/> #1 0x00007f28f054cba6 in epoll_dispatch (base=0x4d58000, <br/> &nbsp;&nbsp;&nbsp;&nbsp;tv=&lt;value optimized out&gt;) at epoll.c:404 <br/> #2 0x00007f28f053bc11 in event_base_loop (base=0x4d58000, flags=0) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at event.c:1558 <br/> #3 0x0000000000409157 in main (argc=&lt;value optimized out&gt;, <br/> &nbsp;&nbsp;&nbsp;&nbsp;argv=&lt;value optimized out&gt;) at daemon/memcached.c:7912 In addition to the # of buckets errors since moving to build-1416, I&#39;ve also encountered a 2 other issues which have rendered the Ruby client (dp5) unusable. One is a Couchbase::Error::Connect and the other Couchbase::Error::View from the client library itself, previously on dp4 none of these errors were happening. Here are each of the stack traces from the Ruby app: <br/> <br/> 2012-07-15T22:07:45+00:00 app[web.1]: REFERER_URL: <br/> 2012-07-15T22:07:45+00:00 app[web.1]: PARAMS: {&quot;controller&quot;:&quot;articles&quot;,&quot;action&quot;:&quot;popular&quot;} <br/> 2012-07-15T22:07:45+00:00 app[web.1]: USER_AGENT: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1201.0 Safari/537.1 <br/> 2012-07-15T22:07:45+00:00 app[web.1]: HTTP_METHOD: GET <br/> 2012-07-15T22:07:45+00:00 app[web.1]: CLASS_NAME: Couchbase::Error::Connect <br/> 2012-07-15T22:07:45+00:00 app[web.1]: MESSAGE: Could not connect to server within allotted time (error=0x17) <br/> 2012-07-15T22:07:45+00:00 app[web.1]: TRACE: /app/vendor/bundle/ruby/1.9.1/gems/couchbase-1.2.0.dp5/lib/couchbase.rb:51:in `initialize&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/couchbase-1.2.0.dp5/lib/couchbase.rb:51:in `new&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/couchbase-1.2.0.dp5/lib/couchbase.rb:51:in `connect&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/app/models/couch.rb:20:in `client&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/app/models/author.rb:9:in `popular&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/app/controllers/application_controller.rb:26:in `fetch_authors_and_categories&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.6/lib/active_support/callbacks.rb:484:in `_run__1380134393223357825__process_action__2968793487945551802__callbacks&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.6/lib/active_support/callbacks.rb:405:in `__run_callback&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.6/lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.6/lib/active_support/callbacks.rb:81:in `run_callbacks&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/abstract_controller/callbacks.rb:17:in `process_action&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_controller/metal/rescue.rb:29:in `process_action&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_controller/metal/instrumentation.rb:30:in `block in process_action&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.6/lib/active_support/notifications.rb:123:in `block in instrument&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.6/lib/active_support/notifications/instrumenter.rb:20:in `instrument&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.6/lib/active_support/notifications.rb:123:in `instrument&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_controller/metal/instrumentation.rb:29:in `process_action&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_controller/metal/params_wrapper.rb:206:in `process_action&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.6/lib/active_record/railties/controller_runtime.rb:18:in `process_action&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.4.0/lib/new_relic/agent/instrumentation/rails3/action_controller.rb:34:in `block in process_action&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.4.0/lib/new_relic/agent/instrumentation/controller_instrumentation.rb:257:in `block in perform_action_with_newrelic_trace&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.4.0/lib/new_relic/agent/method_tracer.rb:242:in `trace_execution_scoped&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.4.0/lib/new_relic/agent/instrumentation/controller_instrumentation.rb:252:in `perform_action_with_newrelic_trace&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.4.0/lib/new_relic/agent/instrumentation/rails3/action_controller.rb:33:in `process_action&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/abstract_controller/base.rb:121:in `process&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/abstract_controller/rendering.rb:45:in `process&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_controller/metal.rb:203:in `dispatch&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_controller/metal/rack_delegation.rb:14:in `dispatch&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_controller/metal.rb:246:in `block in action&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/routing/route_set.rb:73:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/routing/route_set.rb:73:in `dispatch&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/routing/route_set.rb:36:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/journey-1.0.4/lib/journey/router.rb:68:in `block in call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/journey-1.0.4/lib/journey/router.rb:56:in `each&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/journey-1.0.4/lib/journey/router.rb:56:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/routing/route_set.rb:600:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/bundler/gems/rails_exception_handler-647626cd7401/lib/rails_exception_handler.rb:8:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.4.0/lib/new_relic/rack/browser_monitoring.rb:12:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/warden-1.2.1/lib/warden/manager.rb:35:in `block in call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/warden-1.2.1/lib/warden/manager.rb:34:in `catch&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/warden-1.2.1/lib/warden/manager.rb:34:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/etag.rb:23:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/conditionalget.rb:25:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/middleware/head.rb:14:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/middleware/params_parser.rb:21:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/middleware/flash.rb:242:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:205:in `context&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:200:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/middleware/cookies.rb:338:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.6/lib/active_record/query_cache.rb:64:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:473:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.6/lib/active_support/callbacks.rb:405:in `_run__3169872691135403837__call__3136679361565063233__callbacks&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.6/lib/active_support/callbacks.rb:405:in `__run_callback&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.6/lib/active_support/callbacks.rb:385:in `_run_call_callbacks&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.6/lib/active_support/callbacks.rb:81:in `run_callbacks&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/middleware/callbacks.rb:27:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/middleware/remote_ip.rb:31:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.6/lib/rails/rack/logger.rb:26:in `call_app&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.6/lib/rails/rack/logger.rb:16:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/middleware/request_id.rb:22:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/methodoverride.rb:21:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/runtime.rb:17:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.6/lib/active_support/cache/strategy/local_cache.rb:72:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/lock.rb:15:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/middleware/static.rb:62:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:245:in `fetch&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:185:in `lookup&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:66:in `call!&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.6/lib/rails/engine.rb:479:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.6/lib/rails/application.rb:220:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/content_length.rb:14:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.6/lib/rails/rack/log_tailer.rb:17:in `call&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/thin-1.3.1/lib/thin/connection.rb:80:in `block in pre_process&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/thin-1.3.1/lib/thin/connection.rb:78:in `catch&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/thin-1.3.1/lib/thin/connection.rb:78:in `pre_process&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/thin-1.3.1/lib/thin/connection.rb:53:in `process&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/thin-1.3.1/lib/thin/connection.rb:38:in `receive_data&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run_machine&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/thin-1.3.1/lib/thin/backends/base.rb:61:in `start&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/thin-1.3.1/lib/thin/server.rb:159:in `start&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/handler/thin.rb:13:in `run&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/server.rb:265:in `start&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.6/lib/rails/commands/server.rb:70:in `start&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.6/lib/rails/commands.rb:55:in `block in &lt;top (required)&gt;&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.6/lib/rails/commands.rb:50:in `tap&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.6/lib/rails/commands.rb:50:in `&lt;top (required)&gt;&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: script/rails:6:in `require&#39; <br/> 2012-07-15T22:07:45+00:00 app[web.1]: script/rails:6:in `&lt;main&gt;&#39; <br/> <br/> <br/> <br/> <br/> 2012-07-15T22:09:13+00:00 app[web.1]: REFERER_URL: <br/> 2012-07-15T22:09:13+00:00 app[web.1]: PARAMS: {&quot;controller&quot;:&quot;articles&quot;,&quot;action&quot;:&quot;popular&quot;} <br/> 2012-07-15T22:09:13+00:00 app[web.1]: USER_AGENT: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_0) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/22.0.1201.0 Safari/537.1 <br/> 2012-07-15T22:09:13+00:00 app[web.1]: HTTP_METHOD: GET <br/> 2012-07-15T22:09:13+00:00 app[web.1]: CLASS_NAME: Couchbase::Error::View <br/> 2012-07-15T22:09:13+00:00 app[web.1]: MESSAGE: <a href="http://10.114.165.253:8092/_view_merge/?descending=true&reduce=false&include_docs=true&stale=false&limit=10:">http://10.114.165.253:8092/_view_merge/?descending=true&amp;reduce=false&amp;include_docs=true&amp;stale=false&amp;limit=10:</a> row &#39;partition&#39; value is not an integer <br/> 2012-07-15T22:09:13+00:00 app[web.1]: TRACE: /app/vendor/bundle/ruby/1.9.1/gems/couchbase-1.2.0.dp5/lib/couchbase/view.rb:180:in `block in fetch&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/couchbase-1.2.0.dp5/lib/couchbase/view.rb:197:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/couchbase-1.2.0.dp5/lib/couchbase/view.rb:197:in `write&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/couchbase-1.2.0.dp5/lib/couchbase/view.rb:197:in `block in fetch&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/couchbase-1.2.0.dp5/lib/couchbase/view.rb:193:in `loop&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/couchbase-1.2.0.dp5/lib/couchbase/view.rb:193:in `fetch&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/couchbase-1.2.0.dp5/lib/couchbase/view.rb:88:in `each&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/app/models/article.rb:237:in `entries&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/app/models/article.rb:237:in `popular&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/app/controllers/articles_controller.rb:14:in `popular&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_controller/metal/implicit_render.rb:4:in `send_action&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/abstract_controller/base.rb:167:in `process_action&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_controller/metal/rendering.rb:10:in `process_action&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/abstract_controller/callbacks.rb:18:in `block in process_action&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.6/lib/active_support/callbacks.rb:491:in `_run__1380134393223357825__process_action__2968793487945551802__callbacks&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.6/lib/active_support/callbacks.rb:405:in `__run_callback&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.6/lib/active_support/callbacks.rb:385:in `_run_process_action_callbacks&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.6/lib/active_support/callbacks.rb:81:in `run_callbacks&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/abstract_controller/callbacks.rb:17:in `process_action&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_controller/metal/rescue.rb:29:in `process_action&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_controller/metal/instrumentation.rb:30:in `block in process_action&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.6/lib/active_support/notifications.rb:123:in `block in instrument&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.6/lib/active_support/notifications/instrumenter.rb:20:in `instrument&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.6/lib/active_support/notifications.rb:123:in `instrument&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_controller/metal/instrumentation.rb:29:in `process_action&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_controller/metal/params_wrapper.rb:206:in `process_action&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.6/lib/active_record/railties/controller_runtime.rb:18:in `process_action&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.4.0/lib/new_relic/agent/instrumentation/rails3/action_controller.rb:34:in `block in process_action&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.4.0/lib/new_relic/agent/instrumentation/controller_instrumentation.rb:257:in `block in perform_action_with_newrelic_trace&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.4.0/lib/new_relic/agent/method_tracer.rb:242:in `trace_execution_scoped&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.4.0/lib/new_relic/agent/instrumentation/controller_instrumentation.rb:252:in `perform_action_with_newrelic_trace&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.4.0/lib/new_relic/agent/instrumentation/rails3/action_controller.rb:33:in `process_action&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/abstract_controller/base.rb:121:in `process&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/abstract_controller/rendering.rb:45:in `process&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_controller/metal.rb:203:in `dispatch&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_controller/metal/rack_delegation.rb:14:in `dispatch&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_controller/metal.rb:246:in `block in action&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/routing/route_set.rb:73:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/routing/route_set.rb:73:in `dispatch&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/routing/route_set.rb:36:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/journey-1.0.4/lib/journey/router.rb:68:in `block in call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/journey-1.0.4/lib/journey/router.rb:56:in `each&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/journey-1.0.4/lib/journey/router.rb:56:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/routing/route_set.rb:600:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/bundler/gems/rails_exception_handler-647626cd7401/lib/rails_exception_handler.rb:8:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/newrelic_rpm-3.4.0/lib/new_relic/rack/browser_monitoring.rb:12:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/warden-1.2.1/lib/warden/manager.rb:35:in `block in call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/warden-1.2.1/lib/warden/manager.rb:34:in `catch&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/warden-1.2.1/lib/warden/manager.rb:34:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/middleware/best_standards_support.rb:17:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/etag.rb:23:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/conditionalget.rb:25:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/middleware/head.rb:14:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/middleware/params_parser.rb:21:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/middleware/flash.rb:242:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:205:in `context&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/session/abstract/id.rb:200:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/middleware/cookies.rb:338:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.6/lib/active_record/query_cache.rb:64:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activerecord-3.2.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:473:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.6/lib/active_support/callbacks.rb:405:in `_run__3169872691135403837__call__3136679361565063233__callbacks&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.6/lib/active_support/callbacks.rb:405:in `__run_callback&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.6/lib/active_support/callbacks.rb:385:in `_run_call_callbacks&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.6/lib/active_support/callbacks.rb:81:in `run_callbacks&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/middleware/callbacks.rb:27:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/middleware/remote_ip.rb:31:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/middleware/debug_exceptions.rb:16:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/middleware/show_exceptions.rb:56:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.6/lib/rails/rack/logger.rb:26:in `call_app&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.6/lib/rails/rack/logger.rb:16:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/middleware/request_id.rb:22:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/methodoverride.rb:21:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/runtime.rb:17:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/activesupport-3.2.6/lib/active_support/cache/strategy/local_cache.rb:72:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/lock.rb:15:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/actionpack-3.2.6/lib/action_dispatch/middleware/static.rb:62:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:136:in `forward&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:245:in `fetch&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:185:in `lookup&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:66:in `call!&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.6/lib/rails/engine.rb:479:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.6/lib/rails/application.rb:220:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/content_length.rb:14:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.6/lib/rails/rack/log_tailer.rb:17:in `call&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/thin-1.3.1/lib/thin/connection.rb:80:in `block in pre_process&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/thin-1.3.1/lib/thin/connection.rb:78:in `catch&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/thin-1.3.1/lib/thin/connection.rb:78:in `pre_process&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/thin-1.3.1/lib/thin/connection.rb:53:in `process&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/thin-1.3.1/lib/thin/connection.rb:38:in `receive_data&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run_machine&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/thin-1.3.1/lib/thin/backends/base.rb:61:in `start&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/thin-1.3.1/lib/thin/server.rb:159:in `start&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/handler/thin.rb:13:in `run&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/rack-1.4.1/lib/rack/server.rb:265:in `start&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.6/lib/rails/commands/server.rb:70:in `start&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.6/lib/rails/commands.rb:55:in `block in &lt;top (required)&gt;&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.6/lib/rails/commands.rb:50:in `tap&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: /app/vendor/bundle/ruby/1.9.1/gems/railties-3.2.6/lib/rails/commands.rb:50:in `&lt;top (required)&gt;&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: script/rails:6:in `require&#39; <br/> 2012-07-15T22:09:13+00:00 app[web.1]: script/rails:6:in `&lt;main&gt;&#39; Chiyoung, <br/> there are few bugs fixed last week that solves bucket deletion/creation issues but it would be important to analyze the backtrace from memcached process that i posted. <br/> is this vbucket state isssue caused by some thread hung during bucket deletion/creation ? As a workaround if you have everything automated i suggest to reinstall build 1416 and create ( without deletion ) of few buckets and create design docs after some data has been loaded. <br/> Jin,<br/> <br/> I spent some time on debugging this issue and it seems to me that the new batch read daemon task caused very high cpu usage even in the idle cluster with multiple buckets.<br/> <br/> Please take a look at it.<br/> Thanks, I&#39;m working on this issue now. The quick temp fix was merged: <br/> <br/> <a href="http://review.couchbase.org/#change,18397">http://review.couchbase.org/#change,18397</a> <br/> <br/> We&#39;re working on enhancing this temp fix now. Integrated in github-ep-engine-2-0 #355 (See [<a href="http://qa.hq.northscale.net/job/github-ep-engine-2-0/355/">http://qa.hq.northscale.net/job/github-ep-engine-2-0/355/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.couchbase.com/issues/browse/MB-5893" title="Installed build-1416 on 3 node cluster, vbucket stats is showing unusual values for # of buckets, breaking client libraries"><strike>MB-5893</strike></a> Increase the default idle sleep time for batch bg fetcher (Revision c970cb2960d3d9b6281d3fe63ba706fc2906d2d3)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> Chiyoung Seo : <br/> Files : <br/> * bgfetcher.cc<br/> * ep_testsuite.cc<br/> <a href="http://review.couchbase.org/#change,18396">http://review.couchbase.org/#change,18396</a> Integrated in github-ep-engine-2-0 #357 (See [<a href="http://qa.hq.northscale.net/job/github-ep-engine-2-0/357/">http://qa.hq.northscale.net/job/github-ep-engine-2-0/357/</a>])<br/> &nbsp;&nbsp;&nbsp;&nbsp;<a href="http://www.couchbase.com/issues/browse/MB-5893" title="Installed build-1416 on 3 node cluster, vbucket stats is showing unusual values for # of buckets, breaking client libraries"><strike>MB-5893</strike></a> Wake up the idle bg fetcher upon getting bg fetch event (Revision 46ebbf1f6eb4b9ceaed0336949c6fb828a99fff7)<br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Result = SUCCESS<br/> Chiyoung Seo : <br/> Files : <br/> * bgfetcher.cc<br/> * Makefile.am<br/> * dispatcher.cc<br/> * vbucket.hh<br/> * ep.cc<br/> * bgfetcher.hh<br/> * vbucket.cc<br/> User is reporting the same issue with build 1464: <br/> <br/> &quot;Just tried out setting the config to point to this new cluster and have immediately gotten one of the same errors which was happening previously on the previous 1416 build: <br/> Number of buckets must be a power of two &gt; 0 and &lt;= MAX_BUCKETS (error=0x15) <br/> I&#39;m rolling back to the previously working dp4 build again.&quot; <br/> <br/> Addresses of nodes in this cluster: <br/> <br/> Node 1: ec2-204-236-211-26.compute-1.amazonaws.com <br/> Node 2: ec2-67-202-55-114.compute-1.amazonaws.com <br/> Node 3: ec2-50-16-126-61.compute-1.amazonaws.com <br/> <br/> Couchbase Username: Administrator <br/> Couchbase Password: RumbleCouch <br/> SSH Username: ec2-user <br/> <br/> I&#39;ve installed ssh keys for matt, and the QAKey provided by Farshid the last time. Added the cluster map as of Tue Jul 24 03:16:19 UTC 2012 <br/> I just looked, and the map looks good for the default bucket. Which bucket is in use? Im seeig 1024 vbuckets on all buckets in the cluster ( default,views,global,..) <br/> <br/> cluster_map that was just attached also points to a map with 1024 vbuckets Note to all, here&#39;s a good way to use libvbucket to validate the cluster map for a given bucket: <br/> <br/> [<a href='mailto:ec2-user@ip-10-10-205-159'>ec2-user@ip-10-10-205-159</a> data]$ curl <a href="http://localhost:8091/pools/default/buckets/default">http://localhost:8091/pools/default/buckets/default</a> | python -mjson.tool &gt; /var/tmp/clustermap.out <br/> &nbsp;&nbsp;% Total % Received % Xferd Average Speed Time Time Time Current <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Dload Upload Total Spent Left Speed <br/> 100 9147 100 9147 0 0 350k 0 --:--:-- --:--:-- --:--:-- 372k <br/> [<a href='mailto:ec2-user@ip-10-10-205-159'>ec2-user@ip-10-10-205-159</a> data]$ /opt/couchbase/bin/tools/vbuckettool /var/tmp/clustermap.out foo bar <br/> key: foo master: 10.10.163.246:11210 vBucketId: 115 couchApiBase: <a href="http://10.10.163.246:8092/default">http://10.10.163.246:8092/default</a> replicas: 10.10.205.159:11210 <br/> key: bar master: 10.226.114.27:11210 vBucketId: 767 couchApiBase: <a href="http://10.226.114.27:8092/default">http://10.226.114.27:8092/default</a> replicas: 10.10.163.246:11210 <br/> <br/> <br/> This should do the exact same thing the Ruby client library is doing, though things like auth and/or partial responses could make a difference from the client library perspective. note, it could also be a bug in libvbucket. need to get a bit more data I think. Farshid,<br/> <br/> Please assign this to the team or person who can help better than me. It seems to me that it&#39;s not caused by the engine side anymore. All, <br/> <br/> I am going to mark this as resolved because the issue <br/> 1- is not related to 1416 or 2.0 build <br/> 2- the high cpu usage is fixed now <br/> 3- this is not reproduce anymore by simply creating 5 buckets and waiting for sometime before connecting to the cluster. <br/> <br/> please open a new bug ( heard CBD-200 is created for this ) to address the issue with bucket creation Date of First Response Fri, 13 Jul 2012 11:19:49 -0500 Rank 4661 Sprint Priority 0.0 Time In Status [MB-5855] cbbackup, cbrestore, cbtransfer and cbclusterstats are missing on 2.0 build on windows. http://www.couchbase.com/issues/browse/MB-5855 Couchbase Server windows MB-5855 cbbackup, cbrestore, cbtransfer and cbclusterstats are missing on 2.0 build on windows. Bug Blocker Resolved Fixed Public Bin Cui Bin Cui Tue, 10 Jul 2012 13:21:49 -0500 Mon, 23 Jul 2012 14:50:51 -0500 Tue, 10 Jul 2012 14:22:22 -0500 recent-builds-2.0 2.0-beta 2.0-beta tools 0 0 Rank 4699 Time In Status [MB-6667] couchbase server crashed in Sparkle code when it was running idle in the background on Mac OSX 10.6.1 http://www.couchbase.com/issues/browse/MB-6667 Couchbase Server I received a Crash Report from Couchbase Server even though it was really just running idle in the background. <br/> <br/> <br/> Process: Couchbase Server [63624] <br/> Path: /Applications/Couchbase Server 2.0 b1723/Couchbase Server.app/Contents/MacOS/Couchbase Server <br/> Identifier: com.couchbase.couchbase-server <br/> Version: ??? (2.0.0-1614-rel) <br/> Code Type: X86-64 (Native) <br/> Parent Process: launchd [212] <br/> <br/> Date/Time: 2012-09-14 12:50:21.743 -0700 <br/> OS Version: Mac OS X 10.6.8 (10K549) <br/> Report Version: 6 <br/> <br/> Interval Since Last Report: 306760 sec <br/> Crashes Since Last Report: 2 <br/> Per-App Interval Since Last Report: 252223 sec <br/> Per-App Crashes Since Last Report: 1 <br/> Anonymous UUID: 6AE9B8CD-CB33-4A60-A110-3559F289ED41 <br/> <br/> Exception Type: EXC_BAD_ACCESS (SIGSEGV) <br/> Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000028 <br/> Crashed Thread: 0 Dispatch queue: com.apple.main-thread <br/> <br/> Thread 0 Crashed: Dispatch queue: com.apple.main-thread <br/> 0 libobjc.A.dylib 0x00007fff8a60344f _objc_fixupMessageRef + 33 <br/> 1 libobjc.A.dylib 0x00007fff8a5f50bc objc_msgSend_fixup + 136 <br/> 2 org.andymatuschak.Sparkle 0x000000010001bd1b 0x100019000 + 11547 <br/> 3 org.andymatuschak.Sparkle 0x000000010001b531 0x100019000 + 9521 <br/> 4 com.apple.Foundation 0x00007fff8956cbc5 __NSFireTimer + 114 <br/> 5 com.apple.CoreFoundation 0x00007fff84323bb8 __CFRunLoopRun + 6488 <br/> 6 com.apple.CoreFoundation 0x00007fff84321d8f CFRunLoopRunSpecific + 575 <br/> 7 com.apple.HIToolbox 0x00007fff82b787ee RunCurrentEventLoopInMode + 333 <br/> 8 com.apple.HIToolbox 0x00007fff82b785f3 ReceiveNextEventCommon + 310 <br/> 9 com.apple.HIToolbox 0x00007fff82b784ac BlockUntilNextEventMatchingListInMode + 59 <br/> 10 com.apple.AppKit 0x00007fff806eceb2 _DPSNextEvent + 708 <br/> 11 com.apple.AppKit 0x00007fff806ec801 -[NSApplication nextEventMatchingMask:untilDate:inMode:dequeue:] + 155 <br/> 12 com.apple.AppKit 0x00007fff806b268f -[NSApplication run] + 395 <br/> 13 com.apple.AppKit 0x00007fff806ab3b0 NSApplicationMain + 364 <br/> 14 com.couchbase.couchbase-server 0x0000000100001e88 start + 52 <br/> <br/> <br/> (will put the full output in an attachment) Mac OS X 10.6.1, Couchbase Server 2.0 build 1723 MB-6667 couchbase server crashed in Sparkle code when it was running idle in the background on Mac OSX 10.6.1 Bug Blocker Resolved Fixed Public Jens Alfke Matt Ingenthron Fri, 14 Sep 2012 15:34:30 -0500 Mon, 17 Sep 2012 10:22:39 -0500 Mon, 17 Sep 2012 09:29:52 -0500 2.0-beta 2.0-beta 0 2 When restarting Couchbase, I get an immediate crash. The processes running under my user. Note that after I ran this, I killed epmd, still get a crash right away. I then rebooted the OS, still get a crash right away.<br/> <br/> No more couchbase for me, for now. The crash is coming from Sparkle, the 3rd party software update library we use. Looks like it&#39;s trying to notify the app of something, probably a new app version available, but the object it&#39;s trying to call is dealloced. Oops, this is a regression from some ref-counting fixes I made a few days ago. <br/> Fix: <a href="http://review.couchbase.org/#/c/20874/">http://review.couchbase.org/#/c/20874/</a> i found a 10.6 vm and have been testing the build for sometime ans able to do view queries and all. <br/> <br/> didn&#39;t think of sparkle in fact. <br/> thanks so much Matt for capturing this issue and reporting this. <br/> <br/> Jens, <br/> I will review the change and push the code after building it or if you have already built i can test it making this a beta blocker since it immediately happens if user clicks on check for updates any update on this? Verified the fix on 10.7 and 10.8 will have to find a 10.6 box today before releasing the mac build I&#39;d be glad to verify-- just point me to details. im creating a new build to resolve <a href="http://www.couchbase.com/issues/browse/MB-6668" title="build number displayed in mac builds about dialog is incorrect"><strike>MB-6668</strike></a> as well. will pass you the new binaries for verification in an hour or so. Date of First Response Fri, 14 Sep 2012 17:11:00 -0500 Rank 3956 Time In Status [MB-6599] tm_zone doesn't exist on win32 http://www.couchbase.com/issues/browse/MB-6599 Couchbase Server MB-6599 tm_zone doesn't exist on win32 Bug Blocker Resolved Fixed Public Trond Norbye Trond Norbye Tue, 11 Sep 2012 06:33:56 -0500 Tue, 11 Sep 2012 06:36:06 -0500 Tue, 11 Sep 2012 06:36:06 -0500 2.0-beta 2.0-beta bucket-engine 0 0 <a href="http://review.couchbase.org/#/c/20725/">http://review.couchbase.org/#/c/20725/</a> Rank 4023 Time In Status [MB-6580] cannot write couch file greater than 2g (even on 64-bit GNU/Linux) http://www.couchbase.com/issues/browse/MB-6580 Couchbase Server Create single node cluster with single vbucket. Write stuff. Observe how it&#39;ll continuously fail to commit starting from 2gb of file size. MB-6580 cannot write couch file greater than 2g (even on 64-bit GNU/Linux) Bug Blocker Resolved Fixed Public Aaron Miller Aleksey Kondratenko Sun, 9 Sep 2012 08:02:14 -0500 Sun, 9 Sep 2012 18:17:31 -0500 Sun, 9 Sep 2012 18:17:31 -0500 2.0-beta 2.0-beta storage-engine 0 0 Does anything fail when you run couchstore&#39;s &quot;make test&quot; on the system in question? no. make test passes. <br/> <br/> BTW things fail with something like &#39;cannot find header in non-zero length file&#39; <br/> upon opening a file, Couchstore searches backwards in the file looking at the 4k boundaries for the signal byte indicating a file header (trailer). If it doesn&#39;t find any, and the file had data in it when it opened it, it refuses to continue, as this is more than likely because it was told to open something that wasn&#39;t a couchstore file and it should avoid clobbering it. <br/> <br/> It is very odd that -this- would be happening in this case, as it should find previous headers if the file&#39;s been written to correctly before. This makes me think it may be an integer wraparound issue.. Alright, we do have a test for this (lua test tests/largefile.lua), will need to figure out when it broke, as it is broken on my mac as well. Interestingly, on my mac, things seem to break at 1GB instead of 2GB. Qe will add a test to reproduce this by tomorrow It fails Couchstore&#39;s make slow_test (separate test target for tests that take a while to run.), as it includes a test that makes a 5GB file. It is one of the lua tests that currently don&#39;t run on the automated builders though. Got it: <a href="http://review.couchbase.org/#/c/20679/">http://review.couchbase.org/#/c/20679/</a> merged Date of First Response Sun, 9 Sep 2012 08:16:59 -0500 Rank 4039