<!-- 
RSS generated by JIRA (5.2.4#845-sha1:c9f4cc41abe72fb236945343a1f485c2c844dac9) at Tue May 21 11:22:58 CDT 2013

It is possible to restrict the fields that are returned in this document by specifying the 'field' parameter in your request.
For example, to request only the issue key and summary add field=key&field=summary to the URL of your request.
For example:
http://www.couchbase.com/issues/si/jira.issueviews:issue-xml/MB-7272/MB-7272.xml?field=key&field=summary
-->
<rss version="0.92" >
<channel>
    <title>Couchbase</title>
    <link>http://www.couchbase.com/issues</link>
    <description>This file is an XML representation of an issue</description>
    <language>en-us</language>    <build-info>
        <version>5.2.4</version>
        <build-number>845</build-number>
        <build-date>26-12-2012</build-date>
    </build-info>

<item>
            <title>[MB-7272] memcached/ep-engine crashes in flusher or other paths when it receives a shutdown message from ns-server</title>
                <link>http://www.couchbase.com/issues/browse/MB-7272</link>
                <project id="10010" key="MB">Couchbase Server</project>
                        <description>this case could occur in many places&lt;br/&gt;
&lt;br/&gt;
1- when node is waming up and ns-server sneds a shutdown command to delete the bucket during warmup&lt;br/&gt;
2- when node is warming up ( a failed over node ) ns-server sends a shutdown command to delete the bucket &lt;br/&gt;
3- when a node which was rebalanced out but for some reason memcached is still doing sth , ns-server sends a shut down command&lt;br/&gt;
&lt;br/&gt;
scenario #2 is very very common and in large environments where warm up takes 8 hours or so user will keep retrying the rebalance button and it wont succeed unless user manually kills the memcached process manually by running kill command.&lt;br/&gt;
&lt;br/&gt;
in general ep-engine needs to abort instead of crashing&lt;br/&gt;
on the other hand during normal shutdown , when ns-server sends a command to ep-engine to shut down . ep-engine should wait until all items are flushed and then shutdown.&lt;br/&gt;
&lt;br/&gt;
seems like we need to differentiate a command that says shut down gracefully or shut down with force.&lt;br/&gt;
&lt;br/&gt;
some of the bugs  :&lt;br/&gt;
&lt;br/&gt;
&lt;a href=&quot;http://www.couchbase.com/issues/browse/MB-7110&quot;&gt;http://www.couchbase.com/issues/browse/MB-7110&lt;/a&gt;&lt;br/&gt;
&lt;a href=&quot;http://www.couchbase.com/issues/browse/MB-7263&quot;&gt;http://www.couchbase.com/issues/browse/MB-7263&lt;/a&gt;</description>
                <environment></environment>
            <key id="20963">MB-7272</key>
            <summary>memcached/ep-engine crashes in flusher or other paths when it receives a shutdown message from ns-server</summary>
                <type id="1" iconUrl="http://www.couchbase.com/issues/images/icons/issuetypes/bug.png">Bug</type>
                                <priority id="1" iconUrl="http://www.couchbase.com/issues/images/icons/priorities/blocker.png">Blocker</priority>
                    <status id="5" iconUrl="http://www.couchbase.com/issues/images/icons/statuses/resolved.png">Resolved</status>
                    <resolution id="1">Fixed</resolution>
                    <security id="10011">Public</security>
                        <assignee username="jin">Jin Lim</assignee>
                                <reporter username="farshid">Farshid Ghods</reporter>
                        <labels>
                    </labels>
                <created>Tue, 27 Nov 2012 15:27:30 -0600</created>
                <updated>Wed, 19 Dec 2012 03:21:34 -0600</updated>
                    <resolved>Wed, 5 Dec 2012 14:27:49 -0600</resolved>
                            <version>1.8.1</version>
                <version>2.0</version>
                                <fixVersion>2.0</fixVersion>
                                <component>couchbase-bucket</component>
                                <votes>0</votes>
                        <watches>1</watches>
                                                                                  <comments>
                    <comment id="45125" author="jin" created="Fri, 30 Nov 2012 19:27:02 -0600"  >The toy build for a fix candidate has been uploaded for testing. QE and the development team will be verifying the fix for next few days. Thanks!&lt;br/&gt;
&lt;br/&gt;
&lt;a href=&quot;http://builds.hq.northscale.net/latestbuilds/couchbase-server-community_toy-couchstore-x86_64_2.0.0-11302012A-toy.rpm&quot;&gt;http://builds.hq.northscale.net/latestbuilds/couchbase-server-community_toy-couchstore-x86_64_2.0.0-11302012A-toy.rpm&lt;/a&gt;</comment>
                    <comment id="45210" author="andreibaranouski" created="Mon, 3 Dec 2012 05:14:52 -0600"  >tried to test toy build for cases in &lt;a href=&quot;http://www.couchbase.com/issues/browse/MB-7110&quot; title=&quot;[system test] rebalance failed due to &amp;quot;Failed to wait deletion of some buckets on some nodes&amp;quot;&quot;&gt;&lt;strike&gt;MB-7110&lt;/strike&gt;&lt;/a&gt; [system test] rebalance failed due to &amp;quot;Failed to wait deletion of some buckets on some nodes&amp;quot;&lt;br/&gt;
with steps:&lt;br/&gt;
1. cluster 4 nodes, 1 default and 1 sasl bucket with 1500MB of RAM allocated &lt;br/&gt;
10.3.121.112, 10.3.121.113, 10.3.121.114, 10.3.121.115 &lt;br/&gt;
2. load ~1.6M items in each bucket &lt;br/&gt;
3. add node in cluster 10.3.121.116&lt;br/&gt;
&lt;br/&gt;
result: received exactly the same erros as in the &lt;br/&gt;
&lt;a href=&quot;http://www.couchbase.com/issues/browse/MB-7263&quot; title=&quot;Service memcached constantly exited on dest master node after certain steps in XDCR + rebalance scenarious: Port server memcached on node &amp;#39;ns_1@10.3.121.63&amp;#39; exited with status 71.  failed to listen on TCP port 11210: Address already in use&quot;&gt;&lt;strike&gt;MB-7263&lt;/strike&gt;&lt;/a&gt; Service memcached constantly exited on dest master node after certain steps in XDCR + rebalance scenarious: Port server memcached on node &amp;#39;&lt;a href=&apos;mailto:ns_1@10.3.121.63&apos;&gt;ns_1@10.3.121.63&lt;/a&gt;&amp;#39; exited with status 71. failed to listen on TCP port 11210: Address already in use&lt;br/&gt;
&lt;br/&gt;
Port server memcached on node &amp;#39;&lt;a href=&apos;mailto:ns_1@10.3.121.112&apos;&gt;ns_1@10.3.121.112&lt;/a&gt;&amp;#39; exited with status 71. Restarting. Messages: Mon Dec 3 03:11:51.120720 PST 3: failed to listen on TCP port 11210: Address already in use&lt;br/&gt;
&lt;br/&gt;
leave the cluster alive for investigation&lt;br/&gt;
&amp;nbsp;&lt;br/&gt;
&lt;br/&gt;
</comment>
                    <comment id="45254" author="jin" created="Mon, 3 Dec 2012 13:23:39 -0600"  >Thanks Andrei. Please leave the cluster while the development team is investigating the issue.&lt;br/&gt;
&lt;br/&gt;
In the mean time please note that:&lt;br/&gt;
&lt;br/&gt;
1) this bug is to track ep-engine crash when it receives the shutdown (delete) while warminging up. The toy build must have addressed the issue and your last test didn&amp;#39;t see the crash from ep-engine threads.&lt;br/&gt;
2) as you stated, the latest error (OSERR = 71, port already being in use) you encountered sounds much like the original issue of &lt;a href=&quot;http://www.couchbase.com/issues/browse/MB-7263&quot; title=&quot;Service memcached constantly exited on dest master node after certain steps in XDCR + rebalance scenarious: Port server memcached on node &amp;#39;ns_1@10.3.121.63&amp;#39; exited with status 71.  failed to listen on TCP port 11210: Address already in use&quot;&gt;&lt;strike&gt;MB-7263&lt;/strike&gt;&lt;/a&gt;. Which I will continue to investigate from this point on.&lt;br/&gt;
&lt;br/&gt;
Thanks,&lt;br/&gt;
Jin&lt;br/&gt;
&amp;nbsp;&amp;nbsp;</comment>
                    <comment id="45396" author="steve" created="Tue, 4 Dec 2012 14:14:16 -0600"  >from bug-scrub mtg,&lt;br/&gt;
&lt;br/&gt;
looks like there&amp;#39;s fix from Jin and from ns-server team (the infinity fix), and they both need to go in.</comment>
                    <comment id="45503" author="farshid" created="Wed, 5 Dec 2012 14:27:59 -0600"  >build 1974 has this fix</comment>
                    <comment id="45619" author="kzeller" created="Thu, 6 Dec 2012 18:28:47 -0600"  >Added to RN as: &lt;br/&gt;
&lt;br/&gt;
	During Couchbase Server warmup or rebalance, if you delete a data bucket, &lt;br/&gt;
				it will cause the node to crash.</comment>
                    <comment id="46295" author="thuan" created="Wed, 19 Dec 2012 03:21:34 -0600"  >Integrated in github-ep-engine-2-0 #461 (See [&lt;a href=&quot;http://qa.hq.northscale.net/job/github-ep-engine-2-0/461/&quot;&gt;http://qa.hq.northscale.net/job/github-ep-engine-2-0/461/&lt;/a&gt;])&lt;br/&gt;
&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&lt;a href=&quot;http://www.couchbase.com/issues/browse/MB-7272&quot; title=&quot;memcached/ep-engine crashes in flusher or other paths when it receives a shutdown message from ns-server&quot;&gt;&lt;strike&gt;MB-7272&lt;/strike&gt;&lt;/a&gt; stop warmup task immediately if shutdown is being requested (Revision 6b89027ba3b2b461d978d593b14918040c819e2c)&lt;br/&gt;
&lt;br/&gt;
&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;&amp;nbsp;Result = SUCCESS&lt;br/&gt;
Jin : &lt;br/&gt;
Files : &lt;br/&gt;
* src/warmup.cc&lt;br/&gt;
* src/warmup.hh&lt;br/&gt;
* src/ep.cc&lt;br/&gt;
* src/ep.hh&lt;br/&gt;
</comment>
                </comments>
                    <attachments>
                </attachments>
            <subtasks>
            <subtask id="20584">MB-7110</subtask>
            <subtask id="20944">MB-7263</subtask>
        </subtasks>
                <customfields>
                                                                        <customfield id="customfield_10180" key="com.atlassian.jira.ext.charting:firstresponsedate">
                <customfieldname>Date of First Response</customfieldname>
                <customfieldvalues>
                    <customfieldvalue>Fri, 30 Nov 2012 19:27:02 -0600</customfieldvalue>

                </customfieldvalues>
            </customfield>
                                                                                                                                                                                                            <customfield id="customfield_10081" key="com.pyxis.greenhopper.jira:gh-global-rank">
                <customfieldname>Rank</customfieldname>
                <customfieldvalues>
                    <customfieldvalue>3488</customfieldvalue>
                </customfieldvalues>
            </customfield>
                                                                                                                                                                                        <customfield id="customfield_10181" key="com.atlassian.jira.ext.charting:timeinstatus">
                <customfieldname>Time In Status</customfieldname>
                <customfieldvalues>
                    
                </customfieldvalues>
            </customfield>
                                                </customfields>
    </item>
</channel>
</rss>