Couchbase Server

Failing to lock mutex errors : bucket_engine cored, when loading data and querying views in parallel.

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Major Major
  • Resolution: Duplicate
  • Affects Version/s: 2.0
  • Fix Version/s: 2.0-beta
  • Component/s: bucket-engine
  • Security Level: Public
  • Labels:
    None
  • Environment:
    Centos 64 bit.

    2.0.0r-637-g3cf42ae

Description

1) Create view on default
2) Load data
3) Query this view in parallel to loading data.

Attaching all the logs.


Port server memcached on node 'ns_1@10.1.2.31' exited with status 134. Restarting. Messages: Failed to set id on vb95 ``dev_test_view_on_10000_docs-3a32e25-083b531-767''
Failed to set id on vb95 ``dev_test_view_on_10000_docs-3a32e25-083b531-80''
Failed to set id on vb95 ``dev_test_view_on_10000_docs-3a32e25-083b531-842''
Failed to set id on vb96 ``dev_test_view_on_10000_docs-3a32e25-083b531-347''
Failed to set id on vb96 ``dev_test_view_on_10000_docs-3a32e25-083b531-435''
Failed to set id on vb96 ``dev_test_view_on_10000_docs-3a32e25-083b531-744''
Failed to set id on vb96 ``dev_test_view_on_10000_docs-3a32e25-083b531-861''
Failed to set id on vb97 ``dev_test_view_on_10000_docs-3a32e25-083b531-293''
Failed to set id on vb97 ``dev_test_view_on_10000_docs-3a32e25-083b531-303''
Failed to set id on vb97 ``dev_test_view_on_10000_docs-3a32e25-083b531-471''
Failed to set id on vb97 ``dev_test_view_on_10000_docs-3a32e25-083b531-690''
Failed to set id on vb97 ``dev_test_view_on_10000_docs-3a32e25-083b531-700''
Failed to set id on vb97 ``dev_test_view_on_10000_docs-3a32e25-083b531-825''
Failed to set id on vb99 ``dev_test_view_on_10000_docs-3a32e25-083b531-569''
Failed to set id on vb99 ``dev_test_view_on_10000_docs-3a32e25-083b531-618''
Failed to set id on vb99 ``dev_test_view_on_10000_docs-3a32e25-083b531-788''
Failed to set id on vb100 ``dev_test_view_on_10000_docs-3a32e25-083b531-568''
Failed to set id on vb100 ``dev_test_view_on_10000_docs-3a32e25-083b531-619''
Failed to set id on vb100 ``dev_test_view_on_10000_docs-3a32e25-083b531-789''
Failed to set id on vb102 ``dev_test_view_on_10000_docs-3a32e25-083b531-292''
Failed to set id on vb102 ``dev_test_view_on_10000_docs-3a32e25-083b531-302''
Failed to set id on vb102 ``dev_test_view_on_10000_docs-3a32e25-083b531-470''
Failed to set id on vb102 ``dev_test_view_on_10000_docs-3a32e25-083b531-691''
Failed to set id on vb102 ``dev_test_view_on_10000_docs-3a32e25-083b531-701''
Failed to set id on vb102 ``dev_test_view_on_10000_docs-3a32e25-083b531-824''
Failed to set id on vb137 ``dev_test_view_on_10000_docs-3a32e25-083b531-143''
Failed to set id on vb137 ``dev_test_view_on_10000_docs-3a32e25-083b531-232''
Failed to set id on vb137 ``dev_test_view_on_10000_docs-3a32e25-083b531-46''
Failed to set id on vb137 ``dev_test_view_on_10000_docs-3a32e25-083b531-540''
Failed to set id on vb137 ``dev_test_view_on_10000_docs-3a32e25-083b531-631''
Failed to set id on vb139 ``dev_test_view_on_10000_docs-3a32e25-083b531-458''
Failed to set id on vb139 ``dev_test_view_on_10000_docs-3a32e25-083b531-729''
Failed to set id on vb140 ``dev_test_view_on_10000_docs-3a32e25-083b531-459''
Failed to set id on vb140 ``dev_test_view_on_10000_docs-3a32e25-083b531-728''
Failed to set id on vb142 ``dev_test_view_on_10000_docs-3a32e25-083b531-142''
Failed to set id on vb142 ``dev_test_view_on_10000_docs-3a32e25-083b531-233''
Failed to set id on vb142 ``dev_test_view_on_10000_docs-3a32e25-083b531-47''
Failed to set id on vb142 ``dev_test_view_on_10000_docs-3a32e25-083b531-541''
Failed to set id on vb142 ``dev_test_view_on_10000_docs-3a32e25-083b531-630''
Failed to set id on vb143 ``dev_test_view_on_10000_docs-3a32e25-083b531-106''
Failed to set id on vb143 ``dev_test_view_on_10000_docs-3a32e25-083b531-277''
Failed to set id on vb143 ``dev_test_view_on_10000_docs-3a32e25-083b531-495''
Failed to set id on vb143 ``dev_test_view_on_10000_docs-3a32e25-083b531-505''
Failed to set id on vb143 ``dev_test_view_on_10000_docs-3a32e25-083b531-674''
Failed to set id on vb144 ``dev_test_view_on_10000_docs-3a32e25-083b531-448''
Failed to set id on vb144 ``dev_test_view_on_10000_docs-3a32e25-083b531-739''
Failed to set id on vb182 ``dev_test_view_on_10000_docs-3a32e25-083b531-160''
Failed to set id on vb182 ``dev_test_view_on_10000_docs-3a32e25-083b531-211''
Failed to set id on vb182 ``dev_test_view_on_10000_docs-3a32e25-083b531-381''
Failed to set id on vb182 ``dev_test_view_on_10000_docs-3a32e25-083b531-563''
Failed to set id on vb182 ``dev_test_view_on_10000_docs-3a32e25-083b531-612''
Failed to set id on vb182 ``dev_test_view_on_10000_docs-3a32e25-083b531-65''
Failed to set id on vb182 ``dev_test_view_on_10000_docs-3a32e25-083b531-782''
Failed to set id on vb183 ``dev_test_view_on_10000_docs-3a32e25-083b531-124''
Failed to set id on vb183 ``dev_test_view_on_10000_docs-3a32e25-083b531-21''
Failed to set id on vb183 ``dev_test_view_on_10000_docs-3a32e25-083b531-255''
Failed to set id on vb183 ``dev_test_view_on_10000_docs-3a32e25-083b531-527''
Failed to set id on vb183 ``dev_test_view_on_10000_docs-3a32e25-083b531-656''
Failed to set id on vb184 ``dev_test_view_on_10000_docs-3a32e25-083b531-162''
Failed to set id on vb184 ``dev_test_view_on_10000_docs-3a32e25-083b531-213''
Failed to set id on vb184 ``dev_test_view_on_10000_docs-3a32e25-083b531-383''
Failed to set id on vb184 ``dev_test_view_on_10000_docs-3a32e25-083b531-561''
Failed to set id on vb184 ``dev_test_view_on_10000_docs-3a32e25-083b531-610''
Failed to set id on vb184 ``dev_test_view_on_10000_docs-3a32e25-083b531-67''
Failed to set id on vb184 ``dev_test_view_on_10000_docs-3a32e25-083b531-780''
Failed to set id on vb185 ``dev_test_view_on_10000_docs-3a32e25-083b531-126''
Failed to set id on vb185 ``dev_test_view_on_10000_docs-3a32e25-083b531-23''
Failed to set id on vb185 ``dev_test_view_on_10000_docs-3a32e25-083b531-257''
Failed to set id on vb185 ``dev_test_view_on_10000_docs-3a32e25-083b531-525''
Failed to set id on vb185 ``dev_test_view_on_10000_docs-3a32e25-083b531-654''
Failed to set id on vb186 ``dev_test_view_on_10000_docs-3a32e25-083b531-479''
Failed to set id on vb186 ``dev_test_view_on_10000_docs-3a32e25-083b531-698''
Failed to set id on vb186 ``dev_test_view_on_10000_docs-3a32e25-083b531-708''
Failed to set id on vb223 ``dev_test_view_on_10000_docs-3a32e25-083b531-348''
Failed to set id on vb224 ``dev_test_view_on_10000_docs-3a32e25-083b531-419''
Failed to set id on vb224 ``dev_test_view_on_10000_docs-3a32e25-083b531-589''
Failed to set id on vb224 ``dev_test_view_on_10000_docs-3a32e25-083b531-768''
Failed to set id on vb225 ``dev_test_view_on_10000_docs-3a32e25-083b531-809''
Failed to set id on vb226 ``dev_test_view_on_10000_docs-3a32e25-083b531-102''
Failed to set id on vb226 ``dev_test_view_on_10000_docs-3a32e25-083b531-273''
Failed to set id on vb226 ``dev_test_view_on_10000_docs-3a32e25-083b531-491''
Failed to set id on vb226 ``dev_test_view_on_10000_docs-3a32e25-083b531-501''
Failed to set id on vb226 ``dev_test_view_on_10000_docs-3a32e25-083b531-670''
Failed to set id on vb227 ``dev_test_view_on_10000_docs-3a32e25-083b531-146''
Failed to set id on vb227 ``dev_test_view_on_10000_docs-3a32e25-083b531-237''
Failed to set id on vb227 ``dev_test_view_on_10000_docs-3a32e25-083b531-43''
Failed to set id on vb227 ``dev_test_view_on_10000_docs-3a32e25-083b531-545''
Failed to set id on vb227 ``dev_test_view_on_10000_docs-3a32e25-083b531-634''
Failed to set id on vb228 ``dev_test_view_on_10000_docs-3a32e25-083b531-147''
Failed to set id on vb228 ``dev_test_view_on_10000_docs-3a32e25-083b531-236''
Failed to set id on vb228 ``dev_test_view_on_10000_docs-3a32e25-083b531-42''
Failed to set id on vb228 ``dev_test_view_on_10000_docs-3a32e25-083b531-544''
Failed to set id on vb228 ``dev_test_view_on_10000_docs-3a32e25-083b531-635''
Failed to set id on vb229 ``dev_test_view_on_10000_docs-3a32e25-083b531-103''
Failed to set id on vb229 ``dev_test_view_on_10000_docs-3a32e25-083b531-272''
Failed to set id on vb229 ``dev_test_view_on_10000_docs-3a32e25-083b531-490''
Failed to set id on vb229 ``dev_test_view_on_10000_docs-3a32e25-083b531-500''
Failed to set id on vb229 ``dev_test_view_on_10000_docs-3a32e25-083b531-671''
Failed to set id on vb230 ``dev_test_view_on_10000_docs-3a32e25-083b531-808''
FATAL: Failed to lock mutex: 22


  7. Text File
    core.memcached.569.log
    02/Feb/12 7:28 PM
    10 kB
    Karan Kumar

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Karan Kumar added a comment -
Attaching the bt from the core file as well.
Show
Karan Kumar added a comment - Attaching the bt from the core file as well.
Hide
Permalink
Steve Yen added a comment -
No symbols for this. Looks like bucket_engine's not linked with symbols.

Still, reassigning to Trond as perhaps he can deduce some must_lock() issue.
Show
Steve Yen added a comment - No symbols for this. Looks like bucket_engine's not linked with symbols. Still, reassigning to Trond as perhaps he can deduce some must_lock() issue.
Hide
Permalink
Aleksey Kondratenko added a comment -
root@beta:~# perror 22
OS error code 22: Invalid argument
Show
Aleksey Kondratenko added a comment - root@beta :~# perror 22 OS error code 22: Invalid argument
Hide
Permalink
Aleksey Kondratenko added a comment -
just prior to that assertion we're seeing that bucket is actually being deleted. So it's not 'just' loading data and querying views in parallel. It's our 'good-old' bucket deletion bug. Assigning back to Tronds with that info.

Also looks like we indeed have issue getting debug symbols. Is that Centos 5.7 or 5.6? Or maybe 6?
Show
Aleksey Kondratenko added a comment - just prior to that assertion we're seeing that bucket is actually being deleted. So it's not 'just' loading data and querying views in parallel. It's our 'good-old' bucket deletion bug. Assigning back to Tronds with that info. Also looks like we indeed have issue getting debug symbols. Is that Centos 5.7 or 5.6? Or maybe 6?
Hide
Permalink
Farshid Ghods added a comment -
didn't happen in the latest runs.

lowering the priority to major
Show
Farshid Ghods added a comment - didn't happen in the latest runs. lowering the priority to major
Hide
Permalink
Aleksey Kondratenko added a comment -
Trond here's another issue seemingly during shutdown:


warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Core was generated by `/root/src/altoros/moxi/repo20/install/bin/memcached.orig -u root -X /root/src/a'.
Program terminated with signal 6, Aborted.
#0 0xf7719430 in __kernel_vsyscall ()
(gdb) bt
#0 0xf7719430 in __kernel_vsyscall ()
#1 0xf752a941 in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2 0xf752dd72 in *__GI_abort () at abort.c:92
#3 0xf7523b58 in *__GI___assert_fail (assertion=0xf770d393 "peh->refcount > 0", file=0xf770d361 "bucket_engine.c", line=2593, function=0xf770d831 "bucket_engine_release_cookie") at assert.c:81
#4 0xf770b918 in bucket_engine_release_cookie (cookie=0xc695078) at bucket_engine.c:2593
#5 0xf37db051 in TapConnection::releaseReference (this=0xf14bc1c0, force=false) at tapconnection.cc:50
#6 0xf37ef757 in TapConnectionReaperCallback (c=0xf14bc1c0, e=<optimized out>, this=0xc776b48) at tapconnmap.cc:21
#7 TapConnMap::shutdownAllTapConnections (this=0xc7531e8) at tapconnmap.cc:400
#8 0xf37a90c6 in destroy (force=<optimized out>, this=0xc6a24b0) at ep_engine.cc:1215
#9 EvpDestroy (handle=0xc6a24b0, force=false) at ep_engine.cc:84
#10 0xf7708981 in engine_shutdown_thread (arg=0xc6cfa68) at bucket_engine.c:1422
#11 engine_shutdown_thread (arg=0xc6cfa68) at bucket_engine.c:1396
#12 0xf7662c39 in start_thread (arg=0xf0fffb70) at pthread_create.c:304
#13 0xf75d012e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
Backtrace stopped: Not enough registers or memory available to unwind further
(gdb) thread apply all bt

Thread 12 (Thread 0xf74fe6c0 (LWP 11929)):
#0 0xf7719430 in __kernel_vsyscall ()
#1 0xf75d0b96 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#2 0xf76cd38c in ?? () from /usr/lib/libevent-2.0.so.5
#3 0xf76b7c77 in event_base_loop () from /usr/lib/libevent-2.0.so.5
#4 0x0804c221 in main (argc=17, argv=0xffa19f74) at daemon/memcached.c:7781

Thread 11 (Thread 0xf64c6b70 (LWP 11933)):
#0 0xf7719430 in __kernel_vsyscall ()
#1 0xf75d0b96 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#2 0xf76cd38c in ?? () from /usr/lib/libevent-2.0.so.5
#3 0xf76b7c77 in event_base_loop () from /usr/lib/libevent-2.0.so.5
#4 0x0805b3e5 in worker_libevent (arg=0xc6cee70) at daemon/thread.c:304
#5 0xf7662c39 in start_thread (arg=0xf64c6b70) at pthread_create.c:304
#6 0xf75d012e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
Backtrace stopped: Not enough registers or memory available to unwind further

Thread 10 (Thread 0xf44c2b70 (LWP 11937)):
#0 0xf7719430 in __kernel_vsyscall ()
#1 0xf75d0b96 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#2 0xf76cd38c in ?? () from /usr/lib/libevent-2.0.so.5
#3 0xf76b7c77 in event_base_loop () from /usr/lib/libevent-2.0.so.5
#4 0x0805b3e5 in worker_libevent (arg=0xc6cf0b0) at daemon/thread.c:304
#5 0xf7662c39 in start_thread (arg=0xf44c2b70) at pthread_create.c:304
#6 0xf75d012e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
Backtrace stopped: Not enough registers or memory available to unwind further

Thread 9 (Thread 0xf4cc3b70 (LWP 11936)):
#0 0xf7719430 in __kernel_vsyscall ()
#1 0xf75d0b96 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#2 0xf76cd38c in ?? () from /usr/lib/libevent-2.0.so.5
#3 0xf76b7c77 in event_base_loop () from /usr/lib/libevent-2.0.so.5
#4 0x0805b3e5 in worker_libevent (arg=0xc6cf020) at daemon/thread.c:304
#5 0xf7662c39 in start_thread (arg=0xf4cc3b70) at pthread_create.c:304
#6 0xf75d012e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
Backtrace stopped: Not enough registers or memory available to unwind further

Thread 8 (Thread 0xf5cc5b70 (LWP 11934)):
#0 0xf7719430 in __kernel_vsyscall ()
#1 0xf75d0b96 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#2 0xf76cd38c in ?? () from /usr/lib/libevent-2.0.so.5
#3 0xf76b7c77 in event_base_loop () from /usr/lib/libevent-2.0.so.5
#4 0x0805b3e5 in worker_libevent (arg=0xc6cef00) at daemon/thread.c:304
#5 0xf7662c39 in start_thread (arg=0xf5cc5b70) at pthread_create.c:304
#6 0xf75d012e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
Backtrace stopped: Not enough registers or memory available to unwind further

Thread 7 (Thread 0xf54c4b70 (LWP 11935)):
#0 0xf7719430 in __kernel_vsyscall ()
#1 0xf75d0b96 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82
#2 0xf76cd38c in ?? () from /usr/lib/libevent-2.0.so.5
#3 0xf76b7c77 in event_base_loop () from /usr/lib/libevent-2.0.so.5
#4 0x0805b3e5 in worker_libevent (arg=0xc6cef90) at daemon/thread.c:304
#5 0xf7662c39 in start_thread (arg=0xf54c4b70) at pthread_create.c:304
#6 0xf75d012e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
Backtrace stopped: Not enough registers or memory available to unwind further

---Type <return> to continue, or q <return> to quit---
Thread 6 (Thread 0xf6cc7b70 (LWP 11932)):
#0 0xf7719430 in __kernel_vsyscall ()
#1 0xf759abc6 in nanosleep () at ../sysdeps/unix/syscall-template.S:82
#2 0xf759a9f0 in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:138
#3 0x0805d870 in check_isasl_db_thread (arg=0x806b3c0) at daemon/isasl.c:233
#4 0xf7662c39 in start_thread (arg=0xf6cc7b70) at pthread_create.c:304
#5 0xf75d012e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
Backtrace stopped: Not enough registers or memory available to unwind further

Thread 5 (Thread 0xf2f43b70 (LWP 11939)):
#0 0xf7719430 in __kernel_vsyscall ()
#1 0xf759abc6 in nanosleep () at ../sysdeps/unix/syscall-template.S:82
#2 0xf759a9f0 in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:138
#3 0xf3780ddc in TransactionContext::commit (this=0xc7530ac) at ep.cc:2859
#4 0xf378f362 in EventuallyPersistentStore::flushSome (this=0xc752f50, q=0xc75305c, rejectQueue=0xf131ce60) at ep.cc:2031
#5 0xf37ba425 in Flusher::doFlush (this=0xc762778) at flusher.cc:326
#6 0xf37bc058 in Flusher::step (this=0xc762778, tid=..., d=<optimized out>) at flusher.cc:240
#7 0xf37bc8b2 in FlusherStepper::callback (this=0xf16b4418, d=..., t=...) at flusher.cc:23
#8 0xf377fa23 in Task::run (this=0xf16b4650, d=..., t=...) at dispatcher.hh:139
#9 0xf377d2d0 in Dispatcher::run (this=0xc75afc0) at dispatcher.cc:119
#10 0xf377f6b6 in launch_dispatcher_thread (arg=0xc75afc0) at dispatcher.cc:28
#11 0xf7662c39 in start_thread (arg=0xf2f43b70) at pthread_create.c:304
#12 0xf75d012e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
Backtrace stopped: Not enough registers or memory available to unwind further

Thread 4 (Thread 0xf3744b70 (LWP 11938)):
#0 0xf7719430 in __kernel_vsyscall ()
#1 0xf766720a in __pthread_cond_wait (cond=0xc760f2c, mutex=0xc760f0c) at pthread_cond_wait.c:153
#2 0xf377d8be in wait (this=<optimized out>) at syncobject.hh:31
#3 Dispatcher::run (this=0xc760f00) at dispatcher.cc:85
#4 0xf377f6b6 in launch_dispatcher_thread (arg=0xc760f00) at dispatcher.cc:28
#5 0xf7662c39 in start_thread (arg=0xf3744b70) at pthread_create.c:304
#6 0xf75d012e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
Backtrace stopped: Not enough registers or memory available to unwind further

Thread 3 (Thread 0xf2742b70 (LWP 11940)):
#0 _M_erase (__x=<optimized out>, this=<optimized out>) at /usr/include/c++/4.6/bits/stl_tree.h:1072
#1 _M_erase (__x=0xf14b9b00, this=0xf14bab54) at /usr/include/c++/4.6/bits/stl_tree.h:1074
#2 std::_Rb_tree<unsigned short, std::pair<unsigned short const, TapCheckpointState>, std::_Select1st<std::pair<unsigned short const, TapCheckpointState> >, std::less<unsigned short>, std::allocator<std::pair<unsigned short const, TapCheckpointState> > >::_M_erase (this=0xf14bab54, __x=0xf14b9be0) at /usr/include/c++/4.6/bits/stl_tree.h:1074
#3 0xf37e83ef in ~_Rb_tree (this=0xf14bab54, __in_chrg=<optimized out>) at /usr/include/c++/4.6/bits/stl_tree.h:639
#4 ~map (this=0xf14bab54, __in_chrg=<optimized out>) at /usr/include/c++/4.6/bits/stl_map.h:88
#5 TapProducer::~TapProducer (this=0xf14baaa0, __in_chrg=<optimized out>) at tapconnection.hh:893
#6 0xf37e85a2 in TapProducer::~TapProducer (this=0xf14baaa0, __in_chrg=<optimized out>) at tapconnection.hh:897
#7 0xf37f0044 in callback (this=<optimized out>) at tapconnmap.cc:30
#8 TapConnectionReaperCallback::callback (this=0xc787b00) at tapconnmap.cc:28
#9 0xf377fa23 in Task::run (this=0xc79a3a8, d=..., t=...) at dispatcher.hh:139
#10 0xf377d2d0 in Dispatcher::run (this=0xc761b50) at dispatcher.cc:119
#11 0xf377f6b6 in launch_dispatcher_thread (arg=0xc761b50) at dispatcher.cc:28
#12 0xf7662c39 in start_thread (arg=0xf2742b70) at pthread_create.c:304
#13 0xf75d012e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
Backtrace stopped: Not enough registers or memory available to unwind further

Thread 2 (Thread 0xf74fdb70 (LWP 11931)):
#0 0xf7719430 in __kernel_vsyscall ()
#1 0xf75c0e3b in read () at ../sysdeps/unix/syscall-template.S:82
#2 0xf756a0cb in _IO_new_file_underflow (fp=0xf7659440) at fileops.c:606
#3 0xf756b96b in _IO_default_uflow (fp=0xf7659440) at genops.c:440
---Type <return> to continue, or q <return> to quit---
#4 0xf756cd98 in *__GI___uflow (fp=0xf7659440) at genops.c:394
#5 0xf756267c in _IO_getc (fp=0xf7659440) at getc.c:41
#6 0xf771171c in check_stdin_thread (arg=0x804ccb0) at extensions/daemon/stdin_check.c:19
#7 0xf7662c39 in start_thread (arg=0xf74fdb70) at pthread_create.c:304
#8 0xf75d012e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
Backtrace stopped: Not enough registers or memory available to unwind further

Thread 1 (Thread 0xf0fffb70 (LWP 12394)):
#0 0xf7719430 in __kernel_vsyscall ()
#1 0xf752a941 in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#2 0xf752dd72 in *__GI_abort () at abort.c:92
#3 0xf7523b58 in *__GI___assert_fail (assertion=0xf770d393 "peh->refcount > 0", file=0xf770d361 "bucket_engine.c", line=2593, function=0xf770d831 "bucket_engine_release_cookie") at assert.c:81
#4 0xf770b918 in bucket_engine_release_cookie (cookie=0xc695078) at bucket_engine.c:2593
#5 0xf37db051 in TapConnection::releaseReference (this=0xf14bc1c0, force=false) at tapconnection.cc:50
#6 0xf37ef757 in TapConnectionReaperCallback (c=0xf14bc1c0, e=<optimized out>, this=0xc776b48) at tapconnmap.cc:21
#7 TapConnMap::shutdownAllTapConnections (this=0xc7531e8) at tapconnmap.cc:400
#8 0xf37a90c6 in destroy (force=<optimized out>, this=0xc6a24b0) at ep_engine.cc:1215
#9 EvpDestroy (handle=0xc6a24b0, force=false) at ep_engine.cc:84
#10 0xf7708981 in engine_shutdown_thread (arg=0xc6cfa68) at bucket_engine.c:1422
#11 engine_shutdown_thread (arg=0xc6cfa68) at bucket_engine.c:1396
#12 0xf7662c39 in start_thread (arg=0xf0fffb70) at pthread_create.c:304
#13 0xf75d012e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130
Backtrace stopped: Not enough registers or memory available to unwind further
(gdb)
Show
Aleksey Kondratenko added a comment - Trond here's another issue seemingly during shutdown: warning: Can't read pathname for load map: Input/output error. [Thread debugging using libthread_db enabled] Core was generated by `/root/src/altoros/moxi/repo20/install/bin/memcached.orig -u root -X /root/src/a'. Program terminated with signal 6, Aborted. #0 0xf7719430 in __kernel_vsyscall () (gdb) bt #0 0xf7719430 in __kernel_vsyscall () #1 0xf752a941 in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 #2 0xf752dd72 in *__GI_abort () at abort.c:92 #3 0xf7523b58 in *__GI___assert_fail (assertion=0xf770d393 "peh->refcount > 0", file=0xf770d361 "bucket_engine.c", line=2593, function=0xf770d831 "bucket_engine_release_cookie") at assert.c:81 #4 0xf770b918 in bucket_engine_release_cookie (cookie=0xc695078) at bucket_engine.c:2593 #5 0xf37db051 in TapConnection::releaseReference (this=0xf14bc1c0, force=false) at tapconnection.cc:50 #6 0xf37ef757 in TapConnectionReaperCallback (c=0xf14bc1c0, e=<optimized out>, this=0xc776b48) at tapconnmap.cc:21 #7 TapConnMap::shutdownAllTapConnections (this=0xc7531e8) at tapconnmap.cc:400 #8 0xf37a90c6 in destroy (force=<optimized out>, this=0xc6a24b0) at ep_engine.cc:1215 #9 EvpDestroy (handle=0xc6a24b0, force=false) at ep_engine.cc:84 #10 0xf7708981 in engine_shutdown_thread (arg=0xc6cfa68) at bucket_engine.c:1422 #11 engine_shutdown_thread (arg=0xc6cfa68) at bucket_engine.c:1396 #12 0xf7662c39 in start_thread (arg=0xf0fffb70) at pthread_create.c:304 #13 0xf75d012e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 Backtrace stopped: Not enough registers or memory available to unwind further (gdb) thread apply all bt Thread 12 (Thread 0xf74fe6c0 (LWP 11929)): #0 0xf7719430 in __kernel_vsyscall () #1 0xf75d0b96 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 #2 0xf76cd38c in ?? () from /usr/lib/libevent-2.0.so.5 #3 0xf76b7c77 in event_base_loop () from /usr/lib/libevent-2.0.so.5 #4 0x0804c221 in main (argc=17, argv=0xffa19f74) at daemon/memcached.c:7781 Thread 11 (Thread 0xf64c6b70 (LWP 11933)): #0 0xf7719430 in __kernel_vsyscall () #1 0xf75d0b96 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 #2 0xf76cd38c in ?? () from /usr/lib/libevent-2.0.so.5 #3 0xf76b7c77 in event_base_loop () from /usr/lib/libevent-2.0.so.5 #4 0x0805b3e5 in worker_libevent (arg=0xc6cee70) at daemon/thread.c:304 #5 0xf7662c39 in start_thread (arg=0xf64c6b70) at pthread_create.c:304 #6 0xf75d012e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 Backtrace stopped: Not enough registers or memory available to unwind further Thread 10 (Thread 0xf44c2b70 (LWP 11937)): #0 0xf7719430 in __kernel_vsyscall () #1 0xf75d0b96 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 #2 0xf76cd38c in ?? () from /usr/lib/libevent-2.0.so.5 #3 0xf76b7c77 in event_base_loop () from /usr/lib/libevent-2.0.so.5 #4 0x0805b3e5 in worker_libevent (arg=0xc6cf0b0) at daemon/thread.c:304 #5 0xf7662c39 in start_thread (arg=0xf44c2b70) at pthread_create.c:304 #6 0xf75d012e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 Backtrace stopped: Not enough registers or memory available to unwind further Thread 9 (Thread 0xf4cc3b70 (LWP 11936)): #0 0xf7719430 in __kernel_vsyscall () #1 0xf75d0b96 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 #2 0xf76cd38c in ?? () from /usr/lib/libevent-2.0.so.5 #3 0xf76b7c77 in event_base_loop () from /usr/lib/libevent-2.0.so.5 #4 0x0805b3e5 in worker_libevent (arg=0xc6cf020) at daemon/thread.c:304 #5 0xf7662c39 in start_thread (arg=0xf4cc3b70) at pthread_create.c:304 #6 0xf75d012e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 Backtrace stopped: Not enough registers or memory available to unwind further Thread 8 (Thread 0xf5cc5b70 (LWP 11934)): #0 0xf7719430 in __kernel_vsyscall () #1 0xf75d0b96 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 #2 0xf76cd38c in ?? () from /usr/lib/libevent-2.0.so.5 #3 0xf76b7c77 in event_base_loop () from /usr/lib/libevent-2.0.so.5 #4 0x0805b3e5 in worker_libevent (arg=0xc6cef00) at daemon/thread.c:304 #5 0xf7662c39 in start_thread (arg=0xf5cc5b70) at pthread_create.c:304 #6 0xf75d012e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 Backtrace stopped: Not enough registers or memory available to unwind further Thread 7 (Thread 0xf54c4b70 (LWP 11935)): #0 0xf7719430 in __kernel_vsyscall () #1 0xf75d0b96 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 #2 0xf76cd38c in ?? () from /usr/lib/libevent-2.0.so.5 #3 0xf76b7c77 in event_base_loop () from /usr/lib/libevent-2.0.so.5 #4 0x0805b3e5 in worker_libevent (arg=0xc6cef90) at daemon/thread.c:304 #5 0xf7662c39 in start_thread (arg=0xf54c4b70) at pthread_create.c:304 #6 0xf75d012e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 Backtrace stopped: Not enough registers or memory available to unwind further ---Type <return> to continue, or q <return> to quit--- Thread 6 (Thread 0xf6cc7b70 (LWP 11932)): #0 0xf7719430 in __kernel_vsyscall () #1 0xf759abc6 in nanosleep () at ../sysdeps/unix/syscall-template.S:82 #2 0xf759a9f0 in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:138 #3 0x0805d870 in check_isasl_db_thread (arg=0x806b3c0) at daemon/isasl.c:233 #4 0xf7662c39 in start_thread (arg=0xf6cc7b70) at pthread_create.c:304 #5 0xf75d012e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 Backtrace stopped: Not enough registers or memory available to unwind further Thread 5 (Thread 0xf2f43b70 (LWP 11939)): #0 0xf7719430 in __kernel_vsyscall () #1 0xf759abc6 in nanosleep () at ../sysdeps/unix/syscall-template.S:82 #2 0xf759a9f0 in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:138 #3 0xf3780ddc in TransactionContext::commit (this=0xc7530ac) at ep.cc:2859 #4 0xf378f362 in EventuallyPersistentStore::flushSome (this=0xc752f50, q=0xc75305c, rejectQueue=0xf131ce60) at ep.cc:2031 #5 0xf37ba425 in Flusher::doFlush (this=0xc762778) at flusher.cc:326 #6 0xf37bc058 in Flusher::step (this=0xc762778, tid=..., d=<optimized out>) at flusher.cc:240 #7 0xf37bc8b2 in FlusherStepper::callback (this=0xf16b4418, d=..., t=...) at flusher.cc:23 #8 0xf377fa23 in Task::run (this=0xf16b4650, d=..., t=...) at dispatcher.hh:139 #9 0xf377d2d0 in Dispatcher::run (this=0xc75afc0) at dispatcher.cc:119 #10 0xf377f6b6 in launch_dispatcher_thread (arg=0xc75afc0) at dispatcher.cc:28 #11 0xf7662c39 in start_thread (arg=0xf2f43b70) at pthread_create.c:304 #12 0xf75d012e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 Backtrace stopped: Not enough registers or memory available to unwind further Thread 4 (Thread 0xf3744b70 (LWP 11938)): #0 0xf7719430 in __kernel_vsyscall () #1 0xf766720a in __pthread_cond_wait (cond=0xc760f2c, mutex=0xc760f0c) at pthread_cond_wait.c:153 #2 0xf377d8be in wait (this=<optimized out>) at syncobject.hh:31 #3 Dispatcher::run (this=0xc760f00) at dispatcher.cc:85 #4 0xf377f6b6 in launch_dispatcher_thread (arg=0xc760f00) at dispatcher.cc:28 #5 0xf7662c39 in start_thread (arg=0xf3744b70) at pthread_create.c:304 #6 0xf75d012e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 Backtrace stopped: Not enough registers or memory available to unwind further Thread 3 (Thread 0xf2742b70 (LWP 11940)): #0 _M_erase (__x=<optimized out>, this=<optimized out>) at /usr/include/c++/4.6/bits/stl_tree.h:1072 #1 _M_erase (__x=0xf14b9b00, this=0xf14bab54) at /usr/include/c++/4.6/bits/stl_tree.h:1074 #2 std::_Rb_tree<unsigned short, std::pair<unsigned short const, TapCheckpointState>, std::_Select1st<std::pair<unsigned short const, TapCheckpointState> >, std::less<unsigned short>, std::allocator<std::pair<unsigned short const, TapCheckpointState> > >::_M_erase (this=0xf14bab54, __x=0xf14b9be0) at /usr/include/c++/4.6/bits/stl_tree.h:1074 #3 0xf37e83ef in ~_Rb_tree (this=0xf14bab54, __in_chrg=<optimized out>) at /usr/include/c++/4.6/bits/stl_tree.h:639 #4 ~map (this=0xf14bab54, __in_chrg=<optimized out>) at /usr/include/c++/4.6/bits/stl_map.h:88 #5 TapProducer::~TapProducer (this=0xf14baaa0, __in_chrg=<optimized out>) at tapconnection.hh:893 #6 0xf37e85a2 in TapProducer::~TapProducer (this=0xf14baaa0, __in_chrg=<optimized out>) at tapconnection.hh:897 #7 0xf37f0044 in callback (this=<optimized out>) at tapconnmap.cc:30 #8 TapConnectionReaperCallback::callback (this=0xc787b00) at tapconnmap.cc:28 #9 0xf377fa23 in Task::run (this=0xc79a3a8, d=..., t=...) at dispatcher.hh:139 #10 0xf377d2d0 in Dispatcher::run (this=0xc761b50) at dispatcher.cc:119 #11 0xf377f6b6 in launch_dispatcher_thread (arg=0xc761b50) at dispatcher.cc:28 #12 0xf7662c39 in start_thread (arg=0xf2742b70) at pthread_create.c:304 #13 0xf75d012e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 Backtrace stopped: Not enough registers or memory available to unwind further Thread 2 (Thread 0xf74fdb70 (LWP 11931)): #0 0xf7719430 in __kernel_vsyscall () #1 0xf75c0e3b in read () at ../sysdeps/unix/syscall-template.S:82 #2 0xf756a0cb in _IO_new_file_underflow (fp=0xf7659440) at fileops.c:606 #3 0xf756b96b in _IO_default_uflow (fp=0xf7659440) at genops.c:440 ---Type <return> to continue, or q <return> to quit--- #4 0xf756cd98 in *__GI___uflow (fp=0xf7659440) at genops.c:394 #5 0xf756267c in _IO_getc (fp=0xf7659440) at getc.c:41 #6 0xf771171c in check_stdin_thread (arg=0x804ccb0) at extensions/daemon/stdin_check.c:19 #7 0xf7662c39 in start_thread (arg=0xf74fdb70) at pthread_create.c:304 #8 0xf75d012e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 Backtrace stopped: Not enough registers or memory available to unwind further Thread 1 (Thread 0xf0fffb70 (LWP 12394)): #0 0xf7719430 in __kernel_vsyscall () #1 0xf752a941 in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 #2 0xf752dd72 in *__GI_abort () at abort.c:92 #3 0xf7523b58 in *__GI___assert_fail (assertion=0xf770d393 "peh->refcount > 0", file=0xf770d361 "bucket_engine.c", line=2593, function=0xf770d831 "bucket_engine_release_cookie") at assert.c:81 #4 0xf770b918 in bucket_engine_release_cookie (cookie=0xc695078) at bucket_engine.c:2593 #5 0xf37db051 in TapConnection::releaseReference (this=0xf14bc1c0, force=false) at tapconnection.cc:50 #6 0xf37ef757 in TapConnectionReaperCallback (c=0xf14bc1c0, e=<optimized out>, this=0xc776b48) at tapconnmap.cc:21 #7 TapConnMap::shutdownAllTapConnections (this=0xc7531e8) at tapconnmap.cc:400 #8 0xf37a90c6 in destroy (force=<optimized out>, this=0xc6a24b0) at ep_engine.cc:1215 #9 EvpDestroy (handle=0xc6a24b0, force=false) at ep_engine.cc:84 #10 0xf7708981 in engine_shutdown_thread (arg=0xc6cfa68) at bucket_engine.c:1422 #11 engine_shutdown_thread (arg=0xc6cfa68) at bucket_engine.c:1396 #12 0xf7662c39 in start_thread (arg=0xf0fffb70) at pthread_create.c:304 #13 0xf75d012e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 Backtrace stopped: Not enough registers or memory available to unwind further (gdb)
Hide
Permalink
Trond Norbye added a comment -
I guess this one belongs on your table now that you've fixed up the bucket-engine thing :) You should have the privilege to close it as fixed :)
Show
Trond Norbye added a comment - I guess this one belongs on your table now that you've fixed up the bucket-engine thing :) You should have the privilege to close it as fixed :)

People

  • Assignee:
    Dustin Sallings
    Reporter:
    Karan Kumar
Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: