http://www.couchbase.com/issues This file is an XML representation of an issue en-us 5.2.4 845 26-12-2012 http://www.couchbase.com/issues/browse/MB-3592 Couchbase Server rebalance is failing on the latest v0.0.0. 132 builds with these error messages : <br/> <br/> error messages: <br/> <br/> Port server memcached on node &#39;<a href='mailto:ns_1@10.1.5.228'>ns_1@10.1.5.228</a>&#39; exited with status 134. Restarting. Messages: memcached: bucket_engine.c:1876: bucket_engine_release_cookie: Assertion `peh&#39; failed <br/> <br/> Rebalance exited with reason {{{badmatch,{error,closed}}, <br/> [{mc_client_binary,cmd_binary_vocal_recv,5}, <br/> {mc_client_binary,get_vbucket,2}, <br/> {ns_memcached,handle_call,3}, <br/> {gen_server,handle_msg,5}, <br/> {proc_lib,init_p_do_apply,3}]}, <br/> {gen_server,call, <br/> [{&#39;ns_memcached-default&#39;,&#39;<a href='mailto:ns_1@10.1.5.228'>ns_1@10.1.5.228</a>&#39;}, <br/> {get_vbucket,512}, <br/> 30000]}} <br/> ns_orchestrator002 <a href='mailto:ns_1@10.1.5.227'>ns_1@10.1.5.227</a> 11:23:55 - Tue Apr 12, 2011 <br/> Control connection to memcached on &#39;<a href='mailto:ns_1@10.1.5.228'>ns_1@10.1.5.228</a>&#39; disconnected: {{badmatch, <br/> {error, <br/> closed}}, <br/> [{mc_client_binary, <br/> cmd_binary_vocal_recv, <br/> 5}, <br/> {mc_client_binary, <br/> get_vbucket, <br/> 2}, <br/> {ns_memcached, <br/> handle_call, <br/> 3}, <br/> {gen_server, <br/> handle_msg, <br/> 5}, <br/> {proc_lib, <br/> init_p_do_apply, <br/> <br/> <br/> <br/> <br/> steps to reproduce : <br/> <br/> 1- install <br/> 2- create bucket &#39;default&#39; <br/> 3- add node X <br/> 4- rebalance <br/> 5- remove X <br/> 6- rebalance <br/> <br/> rebalance fails in step 4 and step 6 <br/> MB-3592

rebalance fails with bucket_engine error : bucket_engine.c:1876: bucket_engine_release_cookie: Assertion `peh' failed

Bug Major Resolved Duplicate Public Trond Norbye Farshid Ghods Tue, 12 Apr 2011 14:00:50 -0500 Wed, 25 May 2011 09:00:48 -0500 Wed, 25 May 2011 09:00:35 -0500 1.7 alpha 1 1.7 GA bucket-engine 0 0 Fixed by the recent bucket_engine and ep_engine fixes this crash happened on the latest changes again <br/> <br/> Thread 1 (Thread 22470):<br/> #0 0x00007f0347746a75 in raise () from /lib/libc.so.6<br/> #1 0x00007f034774a5c0 in abort () from /lib/libc.so.6<br/> #2 0x00007f034773f941 in __assert_fail () from /lib/libc.so.6<br/> #3 0x00007f0346307bb8 in bucket_engine_release_cookie (cookie=0x3ea0da8)<br/> &nbsp;&nbsp;&nbsp;&nbsp;at bucket_engine.c:2004<br/> #4 0x00007f034380a04f in TapConnection::releaseReference (this=0x3f24e20, <br/> &nbsp;&nbsp;&nbsp;&nbsp;force=198) at tapconnection.cc:35<br/> #5 0x00007f034381b6d1 in TapConnectionReaperCallback::TapConnectionReaperCallback(EventuallyPersistentEngine&amp;, TapConnection*)<br/> <br/> <br/> will post the core logs and diags installed version:<br/> <br/> basestar-260-g2330d10<br/> <br/> ubuntu 64-bit This is a variant of <a href="http://www.couchbase.com/issues/browse/MB-3764" title="memcached core dump in bucket_engine.c:1977 "><strike>MB-3764</strike></a> saw this on last night run against basestar-262 <br/> <br/> I think this crash happens when we delete a huge bucket few seconds after calling flush on the bucket <br/> <br/> Thread 4 (Thread 9793): <br/> #0 0x00000037fc60aee9 in pthread_cond_wait@@GLIBC_2.3.2 () <br/> &nbsp;&nbsp;&nbsp;from /lib64/libpthread.so.0 <br/> #1 0x00002aaaaacfdd5a in wait (this=0x4b33630) at syncobject.hh:31 <br/> #2 Dispatcher::run (this=0x4b33630) at dispatcher.cc:85 <br/> #3 0x00002aaaaacfe953 in launch_dispatcher_thread (arg=0x4b3367c) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at dispatcher.cc:28 <br/> #4 0x00000037fc60673d in start_thread () from /lib64/libpthread.so.0 <br/> #5 0x00000037fbad3f6d in clone () from /lib64/libc.so.6 <br/> <br/> Thread 3 (Thread 9794): <br/> #0 0x00000037fbacd587 in fdatasync () from /lib64/libc.so.6 <br/> #1 0x00002aaaaad7aca9 in full_fsync (id=&lt;value optimized out&gt;, flags=2) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at embedded/sqlite3.c:25510 <br/> #2 unixSync (id=&lt;value optimized out&gt;, flags=2) at embedded/sqlite3.c:25558 <br/> #3 0x00002aaaaadd0777 in vdbeCommit (db=0x60a3ad8, p=&lt;value optimized out&gt;) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at embedded/sqlite3.c:13413 <br/> #4 0x00002aaaaadd1fbd in sqlite3VdbeHalt (p=0x604bcd8) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at embedded/sqlite3.c:56514 <br/> #5 0x00002aaaaae21be9 in sqlite3VdbeExec (p=0x604bcd8) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at embedded/sqlite3.c:62196 <br/> #6 0x00002aaaaae0055a in sqlite3Step (pStmt=0x604bcd8) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at embedded/sqlite3.c:57947 <br/> #7 sqlite3_step (pStmt=0x604bcd8) at embedded/sqlite3.c:58011 <br/> #8 0x00002aaaaad6f199 in PreparedStatement::execute (this=0x464d8aa0) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at sqlite-pst.cc:73 <br/> #9 0x00002aaaaad70648 in SqliteStrategy::execute ( <br/> &nbsp;&nbsp;&nbsp;&nbsp;this=&lt;value optimized out&gt;, query=&lt;value optimized out&gt;) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at sqlite-strategies.cc:151 <br/> #10 0x00002aaaaad72da4 in SqliteStrategy::open (this=0x3926c50) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at sqlite-strategies.cc:122 <br/> #11 0x00002aaaaad6d23d in open (this=0x3926d30) at sqlite-kvstore.hh:175 <br/> #12 StrategicSqlite3::reset (this=0x3926d30) at sqlite-kvstore.cc:144 <br/> #13 0x00002aaaaad00e6e in EventuallyPersistentStore::flushOneDeleteAll ( <br/> &nbsp;&nbsp;&nbsp;&nbsp;this=&lt;value optimized out&gt;) at ep.cc:1731 <br/> #14 0x00002aaaaad0b13d in EventuallyPersistentStore::flushOne ( <br/> &nbsp;&nbsp;&nbsp;&nbsp;this=0x3bf4800, q=0x3bf4920, rejectQueue=0x67130c0) at ep.cc:1869 <br/> #15 0x00002aaaaad0b32e in EventuallyPersistentStore::flushSome ( <br/> &nbsp;&nbsp;&nbsp;&nbsp;this=0x3bf4800, q=0x3bf4920, rejectQueue=0x67130c0) at ep.cc:1487 <br/> #16 0x00002aaaaad40e1b in Flusher::doFlush (this=0x58db9b0) at flusher.cc:240 <br/> #17 0x00002aaaaad416e5 in Flusher::step (this=0x51, d=..., <br/> &nbsp;&nbsp;&nbsp;&nbsp;tid=std::tr1::shared_ptr (count 0) 0x464d8f20) at flusher.cc:154 <br/> #18 0x00002aaaaad41eae in FlusherStepper::callback (this=0x4b338c0, d=..., <br/> &nbsp;&nbsp;&nbsp;&nbsp;t=&lt;value optimized out&gt;) at flusher.cc:23 <br/> #19 0x00002aaaaacff34f in Task::run (this=&lt;value optimized out&gt;, d=..., <br/> &nbsp;&nbsp;&nbsp;&nbsp;t=&lt;value optimized out&gt;) at dispatcher.hh:139 <br/> #20 0x00002aaaaacfdf2b in Dispatcher::run (this=0x49b5b90) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at dispatcher.cc:119 <br/> #21 0x00002aaaaacfe953 in launch_dispatcher_thread (arg=0x51) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at dispatcher.cc:28 <br/> #22 0x00000037fc60673d in start_thread () from /lib64/libpthread.so.0 <br/> #23 0x00000037fbad3f6d in clone () from /lib64/libc.so.6 <br/> <br/> Thread 2 (Thread 9795): <br/> #0 0x00000037fc60b150 in pthread_cond_timedwait@@GLIBC_2.3.2 () <br/> &nbsp;&nbsp;&nbsp;from /lib64/libpthread.so.0 <br/> #1 0x00002aaaaacfbc19 in wait (this=0x58db900, d=...) at syncobject.hh:42 <br/> #2 IdleTask::run (this=0x58db900, d=...) at dispatcher.cc:244 <br/> #3 0x00002aaaaacfdf2b in Dispatcher::run (this=0x58da870) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at dispatcher.cc:119 <br/> #4 0x00002aaaaacfe953 in launch_dispatcher_thread (arg=0x58da8bc) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at dispatcher.cc:28 <br/> #5 0x00000037fc60673d in start_thread () from /lib64/libpthread.so.0 <br/> #6 0x00000037fbad3f6d in clone () from /lib64/libc.so.6 <br/> <br/> Thread 1 (Thread 9806): <br/> #0 0x00000037fba30265 in raise () from /lib64/libc.so.6 <br/> #1 0x00000037fba31d10 in abort () from /lib64/libc.so.6 <br/> #2 0x00000037fba296e6 in __assert_fail () from /lib64/libc.so.6 <br/> #3 0x00002aaaaaaad968 in bucket_engine_release_cookie (cookie=0x621ebe8) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at bucket_engine.c:2004 <br/> #4 0x00002aaaaad54a59 in TapConnection::releaseReference (this=0x6025620, <br/> &nbsp;&nbsp;&nbsp;&nbsp;force=78) at tapconnection.cc:35 <br/> #5 0x00002aaaaad66867 in TapConnectionReaperCallback (this=0x391e440) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at tapconnmap.cc:21 <br/> #6 TapConnMap::shutdownAllTapConnections (this=0x391e440) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at tapconnmap.cc:341 <br/> #7 0x00002aaaaad1df93 in EventuallyPersistentEngine::destroy ( <br/> &nbsp;&nbsp;&nbsp;&nbsp;this=0x391dea0, force=false) at ep_engine.cc:1734 <br/> #8 0x00002aaaaad29ef4 in EvpDestroy (handle=0x391dea0, force=true) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at ep_engine.cc:96 <br/> #9 0x00002aaaaaaae711 in engine_shutdown_thread (arg=0x38e9c60) <br/> &nbsp;&nbsp;&nbsp;&nbsp;at bucket_engine.c:1099 <br/> #10 0x00000037fc60673d in start_thread () from /lib64/libpthread.so.0 <br/> #11 0x00000037fbad3f6d in clone () from /lib64/libc.so.6 <a href="http://www.couchbase.com/issues/browse/MB-3777" title="bucket_engine.c:1413 core dump while possibly retrieving TAP stats during rebalance tests"><strike>MB-3777</strike></a> It manifests itself in different ways<br/> <br/> -- Posted from Bugbox for Android <a href="http://www.couchbase.com/issues/browse/MB-3777" title="bucket_engine.c:1413 core dump while possibly retrieving TAP stats during rebalance tests"><strike>MB-3777</strike></a> It manifests itself in different ways<br/> <br/> -- Posted from Bugbox for Android <a href="http://www.couchbase.com/issues/browse/MB-3777" title="bucket_engine.c:1413 core dump while possibly retrieving TAP stats during rebalance tests"><strike>MB-3777</strike></a> It manifests itself in different ways<br/> <br/> -- Posted from Bugbox for Android Date of First Response Wed, 27 Apr 2011 02:15:03 -0500 Rank 6788 Time In Status