http://www.couchbase.com/issues This file is an XML representation of an issue en-us 5.2.4 845 26-12-2012 http://www.couchbase.com/issues/browse/CCBC-62 Couchbase C client library libcouchbase When one node is down and dropping packets, a store op trying to write to that node times out (OK) but libcouchbase_wait doesn&#39;t return (not OK). <br/> <br/> [<a href='mailto:root@localhost'>root@localhost</a> cb-crank]# gcc -g -Wall -W -I$I/include -L$I/lib t.c -lcouchbase -o t <br/> [<a href='mailto:root@localhost'>root@localhost</a> cb-crank]# export LD_LIBRARY_PATH=$I/lib <br/> [<a href='mailto:root@localhost'>root@localhost</a> cb-crank]# ./t <br/> Stored: hello (6) <br/> Got: hellow (7) = world! (7) <br/> [<a href='mailto:root@localhost'>root@localhost</a> cb-crank]# ./t hellow foo <br/> Stored: hellow (7) <br/> Got: hellow (7) = foo (4) <br/> [<a href='mailto:root@localhost'>root@localhost</a> cb-crank]# ./t xyz <br/> Got: hellow (7) = foo (4) <br/> Failed during store &#39;xyz&#39;: Operation timed out <br/> <br/> <br/> <br/> Setup: Start two nodes of Couchbase Server. In my test, they&#39;re on 10.4.2.13 and 10.4.2.14. Client (see attached code) is being run on host 10.4.2.14. <br/> <br/> On one node (10.4.2.13), simulate a network problem with iptables: <br/> <br/> iptables -I INPUT 1 -p tcp --sport 1000:60000 -j DROP <br/> iptables -I INPUT 2 -p tcp --dport 1000:60000 -j DROP <br/> <br/> Run the sample app a few times with different keys until you hit one that needs to access the down node. As seen in the output above, the storage callback is called with a timeout error, but the libcouchbase_wait() call never returns. Attach debugger to the client and view thread stacks: <br/> <br/> # gdb -p 21746 <br/> GNU gdb (GDB) CentOS (7.0.1-42.el5.centos) <br/> Copyright (C) 2009 Free Software Foundation, Inc. <br/> License GPLv3+: GNU GPL version 3 or later &lt;<a href="http://gnu.org/licenses/gpl.html">http://gnu.org/licenses/gpl.html</a>&gt; <br/> This is free software: you are free to change and redistribute it. <br/> There is NO WARRANTY, to the extent permitted by law. Type &quot;show copying&quot; <br/> and &quot;show warranty&quot; for details. <br/> This GDB was configured as &quot;x86_64-redhat-linux-gnu&quot;. <br/> For bug reporting instructions, please see: <br/> &lt;<a href="http://www.gnu.org/software/gdb/bugs/">http://www.gnu.org/software/gdb/bugs/</a>&gt;. <br/> Attaching to process 21746 <br/> Reading symbols from /root/code/cb-crank/t...done. <br/> Reading symbols from /root/code/install/lib/libcouchbase.so.1...done. <br/> Loaded symbols for /root/code/install/lib/libcouchbase.so.1 <br/> Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done. <br/> Loaded symbols for /lib64/libc.so.6 <br/> Reading symbols from /root/code/install/lib/libvbucket.so.1...done. <br/> Loaded symbols for /root/code/install/lib/libvbucket.so.1 <br/> Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done. <br/> Loaded symbols for /lib64/libdl.so.2 <br/> Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done. <br/> Loaded symbols for /lib64/ld-linux-x86-64.so.2 <br/> Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done. <br/> Loaded symbols for /lib64/libm.so.6 <br/> Reading symbols from /root/code/install/lib/libcouchbase_libevent.so.1...done. <br/> Loaded symbols for /root/code/install/lib/libcouchbase_libevent.so.1 <br/> Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...(no debugging symbols found)...done. <br/> Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5 <br/> Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done. <br/> Loaded symbols for /lib64/librt.so.1 <br/> Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done. <br/> [Thread debugging using libthread_db enabled] <br/> Loaded symbols for /lib64/libpthread.so.0 <br/> <br/> warning: no loadable sections found in added symbol-file system-supplied DSO at 0x7fff822c0000 <br/> 0x0000003407cd3603 in __epoll_wait_nocancel () from /lib64/libc.so.6 <br/> (gdb) where <br/> #0 0x0000003407cd3603 in __epoll_wait_nocancel () from /lib64/libc.so.6 <br/> #1 0x00002b0211713c28 in ?? () from /opt/couchbase/lib/libevent-2.0.so.5 <br/> #2 0x00002b0211702a4c in event_base_loop () from /opt/couchbase/lib/libevent-2.0.so.5 <br/> #3 0x0000000000400d60 in main (argc=1, argv=0x7fff822027f0) at t.c:113 <br/> (gdb) thread apply all bt <br/> <br/> Thread 1 (Thread 0x2b02114eeaf0 (LWP 21746)): <br/> #0 0x0000003407cd3603 in __epoll_wait_nocancel () from /lib64/libc.so.6 <br/> #1 0x00002b0211713c28 in ?? () from /opt/couchbase/lib/libevent-2.0.so.5 <br/> #2 0x00002b0211702a4c in event_base_loop () from /opt/couchbase/lib/libevent-2.0.so.5 <br/> #3 0x0000000000400d60 in main (argc=1, argv=0x7fff822027f0) at t.c:113 <br/> (gdb) quit <br/> <br/> <br/> Line t.c:113 is libcouchbase_wait(). <br/> <br/> <br/> Client compiled as: <br/> <br/> gcc -g -Wall -W -I$I/include -L$I/lib t.c -lcouchbase -o t <br/> <br/> libcouchbase built from <a href="git://github.com/couchbase/libcouchbase.git">git://github.com/couchbase/libcouchbase.git</a> master branch at commit 9cfda9d40a270fc3dd05018eb16a2089c83bf24a <br/> <br/> CentOS 5.6, couchbase server 1.8.0, 2-node cluster CCBC-62

hang in libcouchbase_wait() after get timeout

Bug Major Closed Fixed Public Sergey Avseyev Tim Smith Tue, 1 May 2012 17:31:42 -0500 Tue, 13 Nov 2012 04:21:33 -0600 Tue, 15 May 2012 17:48:07 -0500 1.0.4 library 0 1 I forgot to re-enable packets, so I was unable to log in the machine. So I made this little script to hopefully avoid that mistake. Might be handy for you. Just save it, and run: <br/> <br/> ./drop-packets 90 <br/> ./drop-packets 2m <br/> ./drop-packets 1h30m <br/> <br/> <br/> #! /usr/bin/env perl <br/> <br/> use strict; <br/> use warnings; <br/> <br/> my $time = shift || &#39;10m&#39;; <br/> my $seconds = fromNiceTime($time); <br/> <br/> $| = 1; # Auto-flush STDOUT <br/> <br/> die &quot;Usage: $0 TIME\n\n TIME looks like: 30m or 1m30s, etc.\n&quot; unless $time; <br/> <br/> print &quot;Dropping for $time ($seconds seconds), in 5 seconds (Hit ^C to abort) &quot;; <br/> sleep(5); <br/> print &quot;\n&quot;; <br/> <br/> startDroppingPackets(); <br/> <br/> sleep($seconds); <br/> <br/> stopDroppingPackets(); <br/> <br/> exit 0; <br/> <br/> <br/> sub fromNiceTime { <br/> &nbsp;&nbsp;&nbsp;&nbsp;my ($str) = @_; <br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;my $seconds = 0; <br/> &nbsp;&nbsp;&nbsp;&nbsp;if ($str =~ s/^(\d+)d//) { <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$seconds += $1 * 24 * 60 * 60; <br/> &nbsp;&nbsp;&nbsp;&nbsp;} <br/> &nbsp;&nbsp;&nbsp;&nbsp;if ($str =~ s/^(\d+)h//) { <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$seconds += $1 * 60 * 60; <br/> &nbsp;&nbsp;&nbsp;&nbsp;} <br/> &nbsp;&nbsp;&nbsp;&nbsp;if ($str =~ s/^(\d+)m//) { <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$seconds += $1 * 60; <br/> &nbsp;&nbsp;&nbsp;&nbsp;} <br/> &nbsp;&nbsp;&nbsp;&nbsp;if ($str =~ s/^(\d+)s?$//) { <br/> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;$seconds += $1; <br/> &nbsp;&nbsp;&nbsp;&nbsp;} <br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;return $seconds if $str eq &#39;&#39;; <br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;return undef; <br/> } <br/> <br/> <br/> sub startDroppingPackets { <br/> &nbsp;&nbsp;&nbsp;&nbsp;print &quot;Dropping...\n&quot;; <br/> &nbsp;&nbsp;&nbsp;&nbsp;execute(qw(iptables -I INPUT 1 -p tcp --sport 1000:60000 -j DROP)); <br/> &nbsp;&nbsp;&nbsp;&nbsp;execute(qw(iptables -I INPUT 2 -p tcp --dport 1000:60000 -j DROP)); <br/> } <br/> <br/> <br/> sub stopDroppingPackets { <br/> &nbsp;&nbsp;&nbsp;&nbsp;execute(qw(iptables -D INPUT 1)); <br/> &nbsp;&nbsp;&nbsp;&nbsp;execute(qw(iptables -D INPUT 1)); <br/> &nbsp;&nbsp;&nbsp;&nbsp;print &quot;Stopped.\n&quot;; <br/> } <br/> <br/> sub execute { <br/> &nbsp;&nbsp;&nbsp;&nbsp;my @args = @_; <br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;my $rc = system(@args); <br/> &nbsp;&nbsp;&nbsp;&nbsp;return 1 if $rc == 0; <br/> <br/> &nbsp;&nbsp;&nbsp;&nbsp;warn &quot;Failed running [@args]: Code &quot;, ($rc &gt;&gt; 8), &quot;: $!\n&quot;; <br/> &nbsp;&nbsp;&nbsp;&nbsp;return undef; <br/> } <br/> A lesser problem happens when the host is not dropping packets, but just the couchbase-server service is stopped. <br/> <br/> In this case, there&#39;s a 2.5-sec wait after the get callback hits the error, until libcouchbase_wait() call returns. I&#39;d expect it to return right away, because there&#39;s no more work to be done. <br/> <br/> To reproduce, start a cluster with 2 or 3 nodes. Ensure everything&#39;s working OK. Then shut down one of the nodes: <br/> <br/> service couchbase-server stop <br/> <br/> Now run the client, try with a few different keys until you find one for the down server: <br/> <br/> <a href='mailto:root@localhost'>root@localhost</a> cb-crank]# time ./cbget 10.4.2.14:8091 Administrator password default hello <br/> Got: hello: 776f726c642100 <br/> <br/> real 0m0.017s <br/> user 0m0.005s <br/> sys 0m0.002s <br/> [<a href='mailto:root@localhost'>root@localhost</a> cb-crank]# time ./cbget 10.4.2.14:8091 Administrator password default xyz <br/> Failed during get &#39;xyz&#39;: 23 (Connection failure) <br/> <br/> real 0m2.517s <br/> user 0m0.005s <br/> sys 0m0.001s <br/> [<a href='mailto:root@localhost'>root@localhost</a> cb-crank]# <br/> <br/> Note tha the &quot;Failed during get&quot; message appears immediately, and then there is a 2.5-second wait before the client exits. <br/> <br/> Complete client code (a bit simpler than the first one) is attached as cbget.c, compiled as: <br/> <br/> gcc -g -Wall -W -I/root/code/install/include -L/root/code/install/lib -Wl,-rpath,/root/code/install/lib cbget.c -lcouchbase -o cbget <br/> <br/> Simple cbget command to just get a single key. This could be related to the issue found and fixed today. 1. it is better not to use Administrator account to connect to cluster (default bucket should be accessible) <br/> 2. looks like your cbget.c has typo, but it depends how you are storing the key (I&#39;ve used &#39;cbc cp&#39; command which doesn&#39;t send zero byte for key, and therefore &#39;nkey[0]&#39; should be &#39;strlen(key)&#39;) <br/> 3. interesting, but my ruby gem (based on libcouchbase) fails fast if some node (which should receive the key) is down This is because your program ignores errors in the callback, the libcouchbase call your get callback with the error immediately, I&#39;m looking for why libcouchbase continue waiting then Looks like it isn&#39;t possible to get notification from libevent regarding socket closed (if address is valid), this is why we are using timeouts to check when to breakout the event loop. <br/> <br/> You are free to use libcouchbase_wait(), the libcouchbase allows event loops, executed by user (this is how I handle it in ruby client). In this case you should build IO object and breakout event loop immediately when you received the error. Take a look at couchview.c example <a href="https://github.com/couchbase/libcouchbase/blob/master/example/couchview.c#L289">https://github.com/couchbase/libcouchbase/blob/master/example/couchview.c#L289</a> I think I found the issue, and it is indeed bug in the libcouchbase. It has enough information to breakout event loop and release wait() function. The problem in releasing one of the internal buffers and I&#39;m working on fix now Fixed here <a href="http://review.couchbase.org/15966">http://review.couchbase.org/15966</a> and awaiting for review <a href="https://github.com/couchbase/libcouchbase/commit/3d101adff79725dcad0f6e374733f4f593e6c980">https://github.com/couchbase/libcouchbase/commit/3d101adff79725dcad0f6e374733f4f593e6c980</a> Rank 10743 Time In Status