http://www.couchbase.com/issues This file is an XML representation of an issue en-us 5.2.4 845 26-12-2012 http://www.couchbase.com/issues/browse/MB-6592 Couchbase Server Cluster information: <br/> - 11 centos 6.2 64bit server with 4 cores CPU <br/> - Each server has 10 GB RAM and 150 GB disk. <br/> - 8 GB RAM for couchbase server at each node (80% total system memmories) <br/> - Disk format ext3 on both data and root <br/> - Each server has its own drive, no disk sharing with other server. <br/> - Load 9 million items to both buckets <br/> - Initial indexing, so cpu a little heavy load <br/> - Cluster has 2 buckets, default (3GB) and saslbucket (3GB) <br/> - Each bucket has one doc and 2 views for each doc (default d1 and saslbucket d11) <br/> <br/> * Create cluster with 10 nodes installed couchbase server 2.0.0-1697 <br/> <br/> 10.3.121.13 <br/> 10.3.121.14 <br/> 10.3.121.15 <br/> 10.3.121.16 <br/> 10.3.121.17 <br/> 10.3.121.20 <br/> 10.3.121.22 <br/> 10.3.121.24 <br/> 10.3.121.25 <br/> 10.3.121.23 <br/> * Data path /data <br/> * View path /data <br/> <br/> * Do swap rebalance. Add node 26 and remove node 25 <br/> * Rebalance failed and saw a lot of error message memcached exited with status 71 in log page. <br/> <br/> Link to diags of all nodes <a href="https://s3.amazonaws.com/packages.couchbase/diag-logs/orange/201209/11nodes-1697-memcached-exit-71-20120910.tgz">https://s3.amazonaws.com/packages.couchbase/diag-logs/orange/201209/11nodes-1697-memcached-exit-71-20120910.tgz</a> <br/> <br/> Link to atop node 13 <a href="https://s3.amazonaws.com/packages.couchbase/atop-files/orange/201209/atop-node13">https://s3.amazonaws.com/packages.couchbase/atop-files/orange/201209/atop-node13</a> <br/> Due to large size of atop file, all other atop files are in /tmp directory of each node <br/> centos 6.2 64 bit build 2.0.0-1697 MB-6592

[longevity] memcached hangs when aborting during swap rebalance operation and fails to restart ( exit 71 )

Bug Blocker Closed Cannot Reproduce Public Thuan Nguyen Thuan Nguyen 2.0-beta-release-notes system-test Mon, 10 Sep 2012 16:17:46 -0500 Thu, 10 Jan 2013 01:21:38 -0600 Wed, 26 Sep 2012 20:34:12 -0500 2.0-beta 2.0-beta-2 couchbase-bucket 0 0 From the memcached logs <br/> <br/> <br/> <br/> Sat Sep 8 17:56:06.159645 3: Connection closed by mccouch <br/> Sat Sep 8 17:56:06.159864 3: Trying to connect to mccouch: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.160432 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.160462 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.165925 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.165961 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.171397 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.171430 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.176813 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.176839 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.182241 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.182267 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.187653 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.187679 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.193033 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.193058 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.198462 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.198518 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.204013 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.204038 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.209451 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.209477 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.214788 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.214813 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.220685 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.220712 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.226387 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.226415 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.231879 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.231909 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.237313 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.237339 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.242731 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.242757 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.248138 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.248164 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.253603 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.253630 3: Failed to connect to: &quot;localhost:11213&quot; <br/> Sat Sep 8 17:56:06.261017 3: Failed to connect to: &quot;localhost:11213&quot; <br/> GDB output of the memcached process:- <br/> <a href="https://friendpaste.com/5gFQw9wPBFOgNjue64HfM2">https://friendpaste.com/5gFQw9wPBFOgNjue64HfM2</a> <br/> <br/> <br/> Not sure if related to <br/> <a href="http://www.couchbase.com/issues/browse/MB-5653">http://www.couchbase.com/issues/browse/MB-5653</a> <br/> We have the core file of the memcached process available if needed. 10.3.121.13 (root/couchbase) <br/> <br/> /root/karan Karan, <br/> <br/> shouldn&#39;t this be assigned to couchbase bucket team first ? Assigning to Chiyoung to see if this is ep-engine related Mike, please take a look at this issue. Just guessing here, but from the callstack it looks like the crash is in the assign() method for the basic_string in flushOneDelOrSet.. (the other stacks seems somewhat sane).. Given we&#39;re spinning, looks similar to a case I hit some time ago when some list in flusher was &#39;infinite&#39;. I even showed this to Chiyoung but we were unable to make any sense of what we saw Trond,<br/> <br/> I think this might be in the logging thread. I found this in the backtrace with gdb.<br/> <br/> Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler.&#39;.<br/> #0 0x0000003d7160b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0<br/> <br/> There are a few ep-engine threads at this location also, but since the logging stuff is relatively new I think it might be the cause. Please take a look at this and assign it back to me if you think the problem is in ep-engine. I find it relatively hard to believe that it may dump core on that line given that the code for that looks like: <br/> <br/> At file scope we have: <br/> <br/> static pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER; <br/> static pthread_cond_t cond = PTHREAD_COND_INITIALIZER; <br/> <br/> The call we&#39;re currently stuck in looks like: <br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;pthread_mutex_lock(&amp;mutex); <br/> &nbsp;&nbsp;&nbsp;&nbsp;while (run) { <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;struct timeval tp; <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;gettimeofday(&amp;tp, NULL); <br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;[ ... cut ...] <br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;gettimeofday(&amp;tp, NULL); <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;next = tp.tv_sec + (unsigned int)sleeptime; <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;struct timespec ts = { .tv_sec = next }; <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;pthread_cond_timedwait(&amp;cond, &amp;mutex, &amp;ts); &lt;- This is where we&#39;re stuck <br/> &nbsp;&nbsp;&nbsp;&nbsp;} <br/> <br/> I can&#39;t see how we can pass stuff to pthread_cond_timedwait here that may cause it to _crash_ (it _could_ return with EINVAL for invalid input arguments)... <br/> <br/> If only I figured out how to ask gdb to show me the thread that caused the crash (and why.. which signal etc) <br/> <br/> <br/> Let me take a look at this issue to see if there are anything suspicious in ep-engine. Tony,<br/> <br/> I was not able to reproduce this issue with 4 node cluster and still don&#39;t know why it happened.<br/> <br/> Did you see the same issue recently in your manual and longevity test? I have not seen this issue since then in my system test. Let&#39;s close this bug at this time, and create a new bug if we see this issue again. There have been lots of fixes including bucket destroy in ep-engine. Date of First Response Mon, 10 Sep 2012 16:36:12 -0500 Rank 4045 Time In Status