[MB-5166] Bucket Engine __assert_fail () in bucket_engine_release_cookie (cookie=0x10678848)

Wed, 9 Jan 2013 22:59:20 -0600

CRASH REPORT
Basic crash dump analysis of /tmp//core.memcached.31641.

Please send the file to support@couchbase.com

--------------------------------------------------------------------------------
File information:
-rwxr-xr-x 1 bin bin 406668 Apr 25 17:10 /opt/couchbase/./bin/memcached
-rw------- 1 couchbase couchbase 316112896 Apr 24 13:01 /tmp//core.memcached.31641
7c5e5f50c8a383a9906a3ebbd651007f /opt/couchbase/./bin/memcached
16538cc7e6b24f546e50b7cadae172b3 /tmp//core.memcached.31641
memcached 1.4.4_482_g798ed1d
memcached 1.4.4_482_g798ed1d
VERSION 1.4.4_482_g798ed1d
--------------------------------------------------------------------------------
Core file callstacks:
GNU gdb (GDB) CentOS (7.0.1-42.el5.centos)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/bin/memcached...done.

warning: exec file is newer than core file.
[New Thread 31671]
[New Thread 31670]
[New Thread 31669]
[New Thread 31668]
[New Thread 31661]
[New Thread 31660]
[New Thread 31659]
[New Thread 31658]
[New Thread 31657]
[New Thread 31655]
[New Thread 31654]
[New Thread 31653]
[New Thread 31652]
[New Thread 31651]
[New Thread 31650]
[New Thread 31649]
[New Thread 31641]

warning: .dynamic section for "/opt/couchbase/lib/memcached/ep.so" is not at the expected address (wrong library or version mismatch?)
Reading symbols from /opt/couchbase/lib/memcached/libmemcached_utilities.so.0...done.
Loaded symbols for /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
Reading symbols from /opt/couchbase/lib/libevent-2.0.so.5...done.
Loaded symbols for /opt/couchbase/lib/libevent-2.0.so.5
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /opt/couchbase/lib/libtcmalloc_minimal.so.4...done.
Loaded symbols for /opt/couchbase/lib/libtcmalloc_minimal.so.4
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /opt/couchbase/lib/memcached/stdin_term_handler.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/stdin_term_handler.so
Reading symbols from /opt/couchbase/lib/memcached/bucket_engine.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/bucket_engine.so
Reading symbols from /opt/couchbase/lib/memcached/ep.so...done.
Loaded symbols for /opt/couchbase/lib/memcached/ep.so
Core was generated by `/opt/couchbase/bin/memcached -X /opt/couchbase/lib/memcached/stdin_term_handler'.
Program terminated with signal 6, Aborted.
#0 0x0000003171230265 in raise () from /lib64/libc.so.6

Thread 17 (Thread 0x2ad64db92ca0 (LWP 31641)):
#0 0x00000031712d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002ad64d711c28 in epoll_dispatch (base=0x10384000,
    tv=<value optimized out>) at epoll.c:404
#2 0x00002ad64d700a4c in event_base_loop (base=0x10384000, flags=0)
    at event.c:1558
#3 0x0000000000409df3 in main (argc=<value optimized out>,
    argv=<value optimized out>) at daemon/memcached.c:7583

Thread 16 (Thread 31649):
#0 0x00000031712c678b in read () from /lib64/libc.so.6
#1 0x000000317126cd57 in _IO_new_file_underflow () from /lib64/libc.so.6
#2 0x000000317126d71e in _IO_default_uflow_internal () from /lib64/libc.so.6
#3 0x0000003171268fdb in getc () from /lib64/libc.so.6
#4 0x00002ad64db94875 in check_stdin_thread (arg=0x403500)
    at extensions/daemon/stdin_check.c:19
#5 0x0000003171e0673d in start_thread () from /lib64/libpthread.so.0
#6 0x00000031712d44bd in clone () from /lib64/libc.so.6

Thread 15 (Thread 31650):
#0 0x000000317129a541 in nanosleep () from /lib64/libc.so.6
#1 0x000000317129a364 in sleep () from /lib64/libc.so.6
#2 0x0000000000415928 in check_isasl_db_thread (arg=<value optimized out>)
    at daemon/isasl.c:233
#3 0x0000003171e0673d in start_thread () from /lib64/libpthread.so.0
#4 0x00000031712d44bd in clone () from /lib64/libc.so.6

Thread 14 (Thread 31651):
#0 0x00000031712d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002ad64d711c28 in epoll_dispatch (base=0x10384500,
    tv=<value optimized out>) at epoll.c:404
#2 0x00002ad64d700a4c in event_base_loop (base=0x10384500, flags=0)
    at event.c:1558
#3 0x00000000004139a4 in worker_libevent (arg=0x1035c500)
    at daemon/thread.c:305
#4 0x0000003171e0673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000031712d44bd in clone () from /lib64/libc.so.6

Thread 13 (Thread 31652):
#0 0x00000031712bb187 in sched_yield () from /lib64/libc.so.6
#1 0x00002aaaaad20625 in VBucketDeletionCallback::~VBucketDeletionCallback (
    this=0x15a482f0, __in_chrg=<value optimized out>) at ep.cc:204
#2 0x0000000015a48280 in ?? ()
#3 0x000000001699dad0 in ?? ()
#4 0x00002aaaaad0d45b in release (this=0x15a482f0,
    vbid=<value optimized out>, vb_version=0, row_range=..., isLastChunk=false)
    at atomic.hh:264
#5 unlock (this=0x15a482f0, vbid=<value optimized out>, vb_version=0,
    row_range=..., isLastChunk=false) at atomic.hh:292
#6 ~SpinLockHolder (this=0x15a482f0, vbid=<value optimized out>,
    vb_version=0, row_range=..., isLastChunk=false) at atomic.hh:282
#7 gimme (this=0x15a482f0, vbid=<value optimized out>, vb_version=0,
    row_range=..., isLastChunk=false) at atomic.hh:402
#8 RCPtr (this=0x15a482f0, vbid=<value optimized out>, vb_version=0,
    row_range=..., isLastChunk=false) at atomic.hh:336
#9 setBucketDeletion (this=0x15a482f0, vbid=<value optimized out>,
    vb_version=0, row_range=..., isLastChunk=false) at vbucket.hh:507
#10 EventuallyPersistentStore::completeVBucketDeletion (this=0x15a482f0,
    vbid=<value optimized out>, vb_version=0, row_range=..., isLastChunk=false)
    at ep.cc:779
#11 0x0000003100000000 in ?? ()
#12 0x0000000013205000 in ?? ()
#13 0xffffffffffffffff in ?? ()
#14 0x00000000442c4030 in ?? ()
#15 0x0000000015b74dc8 in ?? ()
#16 0x0000000013205000 in ?? ()
#17 0x0000000000000035 in ?? ()
#18 0x0000000016716780 in ?? ()
#19 0x00002aaaaad3694e in EventuallyPersistentEngine::doCheckpointStats::StatCheckpointVisitor::visitBucket (this=0x1699dac0, vb=...) at ep_engine.cc:3248
#20 0x0000000000000035 in ?? ()
#21 0x0000000000000000 in ?? ()

Thread 12 (Thread 31653):
#0 0x00000031712d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002ad64d711c28 in epoll_dispatch (base=0x10384c80,
    tv=<value optimized out>) at epoll.c:404
#2 0x00002ad64d700a4c in event_base_loop (base=0x10384c80, flags=0)
    at event.c:1558
#3 0x00000000004139a4 in worker_libevent (arg=0x1035c700)
    at daemon/thread.c:305
#4 0x0000003171e0673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000031712d44bd in clone () from /lib64/libc.so.6

Thread 11 (Thread 31654):
#0 0x00000031712d48a8 in epoll_wait () from /lib64/libc.so.6
#1 0x00002ad64d711c28 in epoll_dispatch (base=0x10384a00,
    tv=<value optimized out>) at epoll.c:404
#2 0x00002ad64d700a4c in event_base_loop (base=0x10384a00, flags=0)
    at event.c:1558
#3 0x00000000004139a4 in worker_libevent (arg=0x1035c800)
    at daemon/thread.c:305
#4 0x0000003171e0673d in start_thread () from /lib64/libpthread.so.0
#5 0x00000031712d44bd in clone () from /lib64/libc.so.6

Thread 10 (Thread 31655):
#0 0x00000031712bb187 in sched_yield () from /lib64/libc.so.6
#1 0x00002aaaaad206e7 in ~HashTableVisitor (this=0x460c73a0,
    __in_chrg=<value optimized out>) at stored-value.hh:609
#2 ~VBucketDeletionVisitor (this=0x460c73a0, __in_chrg=<value optimized out>)
    at ep.hh:252
#3 VBucketDeletionCallback::~VBucketDeletionCallback (this=0x460c73a0,
    __in_chrg=<value optimized out>) at ep.cc:204
#4 0x0000000015978148 in ?? ()
#5 0x0000000015a482f0 in ?? ()
#6 0x00002aaaaad66ee2 in TapProducer::rollback (this=0x0)
    at tapconnection.cc:490
#7 0x00000000172c1b00 in ?? ()
#8 0x00002aaa00000000 in ?? ()
#9 0x0000000015978000 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 9 (Thread 31657):
#0 0x000000317129a541 in nanosleep () from /lib64/libc.so.6
#1 0x00000031712cded4 in usleep () from /lib64/libc.so.6
#2 0x00002aaaaad50fb8 in _M_dispose (this=<value optimized out>,
    v=<value optimized out>)
    at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/basic_string.h:233
#3 ~basic_string (this=<value optimized out>, v=<value optimized out>)
    at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/basic_string.h:478
#4 ~pair (this=<value optimized out>, v=<value optimized out>)
    at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/stl_pair.h:69
#5 PagingVisitor::visit (this=<value optimized out>, v=<value optimized out>)
    at item_pager.cc:41
#6 0x0000000000000000 in ?? ()

Thread 8 (Thread 31658):
#0 0x0000003171e0aee9 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1 0x00002aaaaad07cba in Dispatcher::completeNonDaemonTasks (this=0x41957000)
    at dispatcher.cc:249
#2 0x0000000000000000 in ?? ()

Thread 7 (Thread 31659):
#0 0x0000003171e0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1 0x00002aaaaad05b90 in ~RCPtr (this=0x1039e594,
    __in_chrg=<value optimized out>) at atomic.hh:340
#2 ~VBucketVisitor (this=0x1039e594, __in_chrg=<value optimized out>)
    at ep.hh:61
#3 CheckpointVisitor::~CheckpointVisitor (this=0x1039e594,
    __in_chrg=<value optimized out>) at checkpoint_remover.cc:11
#4 0x000000001039e540 in ?? ()
#5 0x0000000010406300 in ?? ()
#6 0x0000000010406300 in ?? ()
#7 0x0000000000000047 in ?? ()
#8 0x00002aaaaad07e86 in acquire (this=0x2aaaaad05a90) at mutex.hh:73
#9 lock (this=0x2aaaaad05a90) at locks.hh:48
#10 LockHolder (this=0x2aaaaad05a90) at locks.hh:26
#11 Dispatcher::completeNonDaemonTasks (this=0x2aaaaad05a90)
    at dispatcher.cc:229
#12 0x0000000000000000 in ?? ()

Thread 6 (Thread 31660):
#0 0x0000003171e0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1 0x00002aaaaad05b90 in ~RCPtr (this=0x1039ead4,
    __in_chrg=<value optimized out>) at atomic.hh:340
#2 ~VBucketVisitor (this=0x1039ead4, __in_chrg=<value optimized out>)
    at ep.hh:61
#3 CheckpointVisitor::~CheckpointVisitor (this=0x1039ead4,
    __in_chrg=<value optimized out>) at checkpoint_remover.cc:11
#4 0x000000001039ea80 in ?? ()
#5 0x0000000010406980 in ?? ()
#6 0x0000000010406980 in ?? ()
#7 0x0000000000000043 in ?? ()
#8 0x00002aaaaad07e86 in acquire (this=0x2aaaaad05a90) at mutex.hh:73
#9 lock (this=0x2aaaaad05a90) at locks.hh:48
#10 LockHolder (this=0x2aaaaad05a90) at locks.hh:26
#11 Dispatcher::completeNonDaemonTasks (this=0x2aaaaad05a90)
    at dispatcher.cc:229
#12 0x0000000000000000 in ?? ()

Thread 5 (Thread 31661):
#0 0x0000003171e0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1 0x00002aaaaad314f9 in EventuallyPersistentEngine::doEngineStats (
    this=0x103ee5f0, cookie=0x103ee5e8, add_stat=0) at ep_engine.cc:2859
#2 0x0000000000000000 in ?? ()

Thread 4 (Thread 31668):
#0 0x0000003171e0aee9 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1 0x00002aaaaad07cba in Dispatcher::completeNonDaemonTasks (this=0x492ce000)
    at dispatcher.cc:249
#2 0x0000000000000000 in ?? ()

Thread 3 (Thread 31669):
#0 0x00000031712cda97 in fdatasync () from /lib64/libc.so.6
#1 0x00002aaaaad8b089 in sqlite3BitvecClear (p=0x103893a8,
    i=<value optimized out>, pBuf=0x2) at embedded/sqlite3.c:32077
#2 0x0000000000000000 in ?? ()

Thread 2 (Thread 31670):
#0 0x0000003171e0b150 in pthread_cond_timedwait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1 0x00002aaaaad05b90 in ~RCPtr (this=0x1039f554,
    __in_chrg=<value optimized out>) at atomic.hh:340
#2 ~VBucketVisitor (this=0x1039f554, __in_chrg=<value optimized out>)
    at ep.hh:61
#3 CheckpointVisitor::~CheckpointVisitor (this=0x1039f554,
    __in_chrg=<value optimized out>) at checkpoint_remover.cc:11
#4 0x000000001039f500 in ?? ()
#5 0x0000000015974600 in ?? ()
#6 0x0000000015974600 in ?? ()
#7 0x0000000000000048 in ?? ()
#8 0x00002aaaaad07e86 in acquire (this=0x2aaaaad05a90) at mutex.hh:73
#9 lock (this=0x2aaaaad05a90) at locks.hh:48
#10 LockHolder (this=0x2aaaaad05a90) at locks.hh:26
#11 Dispatcher::completeNonDaemonTasks (this=0x2aaaaad05a90)
    at dispatcher.cc:229
#12 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x4b0d0940 (LWP 31671)):
#0 0x0000003171230265 in raise () from /lib64/libc.so.6
#1 0x0000003171231d10 in abort () from /lib64/libc.so.6
#2 0x00000031712296e6 in __assert_fail () from /lib64/libc.so.6
#3 0x00002aaaaaaae280 in bucket_engine_release_cookie (cookie=0x10678848)
    at bucket_engine.c:2388
#4 0x00002aaaaad61099 in _M_erase (this=0x0, __x=0x18a75180)
    at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/ext/new_allocator.h:94
#5 std::_Rb_tree<key_spec_t, std::pair<key_spec_t const, unsigned char>, std::_Select1st<std::pair<key_spec_t const, unsigned char> >, std::less<key_spec_t>, std::allocator<std::pair<key_spec_t const, unsigned char> > >::_M_erase (
    this=0x0, __x=0x18a75180)
    at /usr/lib/gcc/x86_64-redhat-linux/4.1.2/../../../../include/c++/4.1.2/bits/stl_tree.h:1264
#6 0x00000000159778d8 in ?? ()
#7 0x00000000159778d8 in ?? ()
#8 0x00000000159778d8 in ?? ()
#9 0x0000000015977900 in ?? ()
#10 0x0000000015977ad8 in ?? ()
#11 0x00000031842ed720 in ?? () from /usr/lib64/libstdc++.so.6
#12 0x0000000000000018 in ?? ()
#13 0x00000000159778d8 in ?? ()
#14 0x00000031842e6630 in vtable for std::ios_base ()
   from /usr/lib64/libstdc++.so.6
#15 0x0000000000000006 in ?? ()
#16 0x0000000000000000 in ?? ()
--------------------------------------------------------------------------------
Module information:
/opt/couchbase/lib/memcached/libmemcached_utilities.so.0:
lrwxrwxrwx 1 bin bin 31 Apr 25 22:20 /opt/couchbase/lib/memcached/libmemcached_utilities.so.0 -> libmemcached_utilities.so.0.0.0
fbfba9cc688a46a330a7bf0c0c8aac8c /opt/couchbase/lib/memcached/libmemcached_utilities.so.0
/opt/couchbase/lib/libevent-2.0.so.5:
lrwxrwxrwx 1 bin bin 21 Apr 25 22:20 /opt/couchbase/lib/libevent-2.0.so.5 -> libevent-2.0.so.5.1.0
36d9d9ac73e679d4f3ca9f5f35b0f9c5 /opt/couchbase/lib/libevent-2.0.so.5
/lib64/libdl.so.2:
lrwxrwxrwx 1 root root 12 Feb 9 12:05 /lib64/libdl.so.2 -> libdl-2.5.so
930ce0559663be96be1712a367568a8c /lib64/libdl.so.2
/lib64/libm.so.6:
lrwxrwxrwx 1 root root 11 Feb 9 12:05 /lib64/libm.so.6 -> libm-2.5.so
29c798ebcc7a479a7c16dba3c9f3ce2c /lib64/libm.so.6
/lib64/librt.so.1:
lrwxrwxrwx 1 root root 12 Feb 9 12:05 /lib64/librt.so.1 -> librt-2.5.so
a0fd97e8bfbfda6fdbe3fe1eb7cb8ddb /lib64/librt.so.1
/opt/couchbase/lib/libtcmalloc_minimal.so.4:
lrwxrwxrwx 1 bin bin 28 Apr 25 22:20 /opt/couchbase/lib/libtcmalloc_minimal.so.4 -> libtcmalloc_minimal.so.4.1.0
3fb775713c138c3675722a6c6b392dcc /opt/couchbase/lib/libtcmalloc_minimal.so.4
/lib64/libpthread.so.0:
lrwxrwxrwx 1 root root 17 Feb 9 12:05 /lib64/libpthread.so.0 -> libpthread-2.5.so
35a380efbd8c36d290bd14ef40a2389a /lib64/libpthread.so.0
/lib64/libc.so.6:
lrwxrwxrwx 1 root root 11 Feb 9 12:05 /lib64/libc.so.6 -> libc-2.5.so
7e1f8318e471bf27827477b392cd5b0f /lib64/libc.so.6
/lib64/ld-linux-x86-64.so.2:
lrwxrwxrwx 1 root root 9 Feb 9 12:05 /lib64/ld-linux-x86-64.so.2 -> ld-2.5.so
e4c19bd6566fc08034a8b7460abc7491 /lib64/ld-linux-x86-64.so.2
/usr/lib64/libstdc++.so.6:
lrwxrwxrwx 1 root root 18 Feb 9 12:05 /usr/lib64/libstdc++.so.6 -> libstdc++.so.6.0.8
119c22e4195d512f01a6695036f62e96 /usr/lib64/libstdc++.so.6
/lib64/libgcc_s.so.1:
lrwxrwxrwx 1 root root 28 Feb 9 12:05 /lib64/libgcc_s.so.1 -> libgcc_s-4.1.2-20080825.so.1
3c4c387a43a50be2d60406bd76bb8b77 /lib64/libgcc_s.so.1
/opt/couchbase/lib/memcached/stdin_term_handler.so:
-rwxr-xr-x 1 bin bin 16122 Apr 25 17:10 /opt/couchbase/lib/memcached/stdin_term_handler.so
ee2008dc2bcdef868becc54e220c2ab7 /opt/couchbase/lib/memcached/stdin_term_handler.so
/opt/couchbase/lib/memcached/bucket_engine.so:
lrwxrwxrwx 1 bin bin 22 Apr 25 22:20 /opt/couchbase/lib/memcached/bucket_engine.so -> bucket_engine.so.0.0.0
8bdad57dcb75d241e3135182d4bbb7d4 /opt/couchbase/lib/memcached/bucket_engine.so
/opt/couchbase/lib/memcached/ep.so:
lrwxrwxrwx 1 bin bin 11 Apr 25 22:20 /opt/couchbase/lib/memcached/ep.so -> ep.so.0.0.0
EP engine v1.8.0_54_ge0dfd52
cdf9c74a832e28be82ade15c51be4c99 /opt/couchbase/lib/memcached/ep.so

Couchbase

[MB-5166] Bucket Engine __assert_fail () in bucket_engine_release_cookie (cookie=0x10678848)