Deadlock in flusher thread and assertion failure

Aleksey Kondratenko added a comment - 01/Feb/12 3:13 PM Same assert seemingly on current code. Here's backtrace from core dump. GNU gdb (GDB) 7.3-debian Copyright (C) 2011 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later < http://gnu.org/licenses/gpl.html > This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i486-linux-gnu". For bug reporting instructions, please see: < http://www.gnu.org/software/gdb/bugs/ >... Reading symbols from /root/src/altoros/moxi/repo20/install/bin/memcached.orig...done. [New LWP 4087] [New LWP 4077] [New LWP 4088] [New LWP 4078] [New LWP 4080] [New LWP 4086] [New LWP 4082] [New LWP 4083] [New LWP 13267] [New LWP 4079] [New LWP 4081] [New LWP 4075] warning: Can't read pathname for load map: Input/output error. [Thread debugging using libthread_db enabled] Core was generated by `/root/src/altoros/moxi/repo20/install/bin/memcached.orig -u root -X /root/src/a'. Program terminated with signal 6, Aborted. #0 0xf76fe430 in __kernel_vsyscall () (gdb) thread apply all bt Thread 12 (Thread 0xf74e36c0 (LWP 4075)): #0 0xf76fe430 in __kernel_vsyscall () #1 0xf75b5b96 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 #2 0xf76b238c in ?? () from /usr/lib/libevent-2.0.so.5 #3 0xf769cc77 in event_base_loop () from /usr/lib/libevent-2.0.so.5 #4 0x0804c1b8 in main (argc=17, argv=0xffec82b4) at daemon/memcached.c:7544 Thread 11 (Thread 0xf54a9b70 (LWP 4081)): #0 0xf76fe430 in __kernel_vsyscall () #1 0xf75b5b96 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 #2 0xf76b238c in ?? () from /usr/lib/libevent-2.0.so.5 #3 0xf769cc77 in event_base_loop () from /usr/lib/libevent-2.0.so.5 #4 0x0805a675 in worker_libevent (arg=0x9ca6418) at daemon/thread.c:304 #5 0xf7647c39 in start_thread (arg=0xf54a9b70) at pthread_create.c:304 #6 0xf75b512e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 Backtrace stopped: Not enough registers or memory available to unwind further Thread 10 (Thread 0xf64abb70 (LWP 4079)): #0 0xf76fe430 in __kernel_vsyscall () #1 0xf75b5b96 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 #2 0xf76b238c in ?? () from /usr/lib/libevent-2.0.so.5 #3 0xf769cc77 in event_base_loop () from /usr/lib/libevent-2.0.so.5 #4 0x0805a675 in worker_libevent (arg=0x9ca62f8) at daemon/thread.c:304 #5 0xf7647c39 in start_thread (arg=0xf64abb70) at pthread_create.c:304 #6 0xf75b512e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 Backtrace stopped: Not enough registers or memory available to unwind further Thread 9 (Thread 0xe69ffb70 (LWP 13267)): #0 0xf76fe430 in __kernel_vsyscall () #1 0xf757fbc6 in nanosleep () at ../sysdeps/unix/syscall-template.S:82 #2 0xf75aee2c in usleep (useconds=1000) at ../sysdeps/unix/sysv/linux/usleep.c:33 #3 0xf39ba3d4 in Flusher::wait (this=0x9d2e190) at flusher.cc:59 #4 0xf3986a8b in stopFlusher (this=0x9d1e978) at ep.cc:748 #5 EventuallyPersistentStore::~EventuallyPersistentStore (this=0x9d1e978, __in_chrg=<optimized out>) at ep.cc:714 #6 0xf39b5144 in EventuallyPersistentEngine::~EventuallyPersistentEngine (this=0x9d15a48, __in_chrg=<optimized out>) at ep_engine.h:509 #7 0xf39aadf6 in EvpDestroy (handle=0x9d15a48, force=false) at ep_engine.cc:85 #8 0xf76ed981 in engine_shutdown_thread (arg=0x9cbfbe8) at bucket_engine.c:1422 #9 engine_shutdown_thread (arg=0x9cbfbe8) at bucket_engine.c:1396 #10 0xf7647c39 in start_thread (arg=0xe69ffb70) at pthread_create.c:304 #11 0xf75b512e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 Backtrace stopped: Not enough registers or memory available to unwind further Thread 8 (Thread 0xf44a7b70 (LWP 4083)): #0 0xf76fe430 in __kernel_vsyscall () #1 0xf75b5b96 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 #2 0xf76b238c in ?? () from /usr/lib/libevent-2.0.so.5 #3 0xf769cc77 in event_base_loop () from /usr/lib/libevent-2.0.so.5 #4 0x0805a675 in worker_libevent (arg=0x9ca6538) at daemon/thread.c:304 #5 0xf7647c39 in start_thread (arg=0xf44a7b70) at pthread_create.c:304 #6 0xf75b512e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 Backtrace stopped: Not enough registers or memory available to unwind further Thread 7 (Thread 0xf4ca8b70 (LWP 4082)): #0 0xf76fe430 in __kernel_vsyscall () #1 0xf75b5b96 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 #2 0xf76b238c in ?? () from /usr/lib/libevent-2.0.so.5 #3 0xf769cc77 in event_base_loop () from /usr/lib/libevent-2.0.so.5 ---Type <return> to continue, or q <return> to quit--- #4 0x0805a675 in worker_libevent (arg=0x9ca64a8) at daemon/thread.c:304 #5 0xf7647c39 in start_thread (arg=0xf4ca8b70) at pthread_create.c:304 #6 0xf75b512e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 Backtrace stopped: Not enough registers or memory available to unwind further Thread 6 (Thread 0xf385db70 (LWP 4086)): #0 0xf76fe430 in __kernel_vsyscall () #1 0xf764c20a in __pthread_cond_wait (cond=0x9d2c944, mutex=0x9d2c924) at pthread_cond_wait.c:153 #2 0xf397f6be in wait (this=<optimized out>) at syncobject.hh:31 #3 Dispatcher::run (this=0x9d2c918) at dispatcher.cc:85 #4 0xf39814b6 in launch_dispatcher_thread (arg=0x9d2c918) at dispatcher.cc:28 #5 0xf7647c39 in start_thread (arg=0xf385db70) at pthread_create.c:304 #6 0xf75b512e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 Backtrace stopped: Not enough registers or memory available to unwind further Thread 5 (Thread 0xf5caab70 (LWP 4080)): #0 0xf76fe430 in __kernel_vsyscall () #1 0xf75b5b96 in epoll_wait () at ../sysdeps/unix/syscall-template.S:82 #2 0xf76b238c in ?? () from /usr/lib/libevent-2.0.so.5 #3 0xf769cc77 in event_base_loop () from /usr/lib/libevent-2.0.so.5 #4 0x0805a675 in worker_libevent (arg=0x9ca6388) at daemon/thread.c:304 #5 0xf7647c39 in start_thread (arg=0xf5caab70) at pthread_create.c:304 #6 0xf75b512e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 Backtrace stopped: Not enough registers or memory available to unwind further Thread 4 (Thread 0xf6cacb70 (LWP 4078)): #0 0xf76fe430 in __kernel_vsyscall () #1 0xf757fbc6 in nanosleep () at ../sysdeps/unix/syscall-template.S:82 #2 0xf757f9f0 in __sleep (seconds=0) at ../sysdeps/unix/sysv/linux/sleep.c:138 #3 0x0805cb00 in check_isasl_db_thread (arg=0x806a380) at daemon/isasl.c:233 #4 0xf7647c39 in start_thread (arg=0xf6cacb70) at pthread_create.c:304 #5 0xf75b512e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 Backtrace stopped: Not enough registers or memory available to unwind further Thread 3 (Thread 0xf285bb70 (LWP 4088)): #0 0xf76fe430 in __kernel_vsyscall () #1 0xf764c703 in pthread_cond_timedwait@@GLIBC_2.3.2 () at ../nptl/sysdeps/unix/sysv/linux/i386/i686/../i486/pthread_cond_timedwait.S:236 #2 0xf3979644 in wait (tv=..., this=0x9d2d570) at syncobject.hh:42 #3 IdleTask::run (this=0x9d2e110, d=...) at dispatcher.cc:244 #4 0xf397f0d0 in Dispatcher::run (this=0x9d2d568) at dispatcher.cc:119 #5 0xf39814b6 in launch_dispatcher_thread (arg=0x9d2d568) at dispatcher.cc:28 #6 0xf7647c39 in start_thread (arg=0xf285bb70) at pthread_create.c:304 #7 0xf75b512e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 Backtrace stopped: Not enough registers or memory available to unwind further Thread 2 (Thread 0xf74e2b70 (LWP 4077)): #0 0xf76fe430 in __kernel_vsyscall () #1 0xf75a5e3b in read () at ../sysdeps/unix/syscall-template.S:82 #2 0xf754f0cb in _IO_new_file_underflow (fp=0xf763e440) at fileops.c:606 #3 0xf755096b in _IO_default_uflow (fp=0xf763e440) at genops.c:440 #4 0xf7551d98 in *__GI___uflow (fp=0xf763e440) at genops.c:394 #5 0xf754767c in _IO_getc (fp=0xf763e440) at getc.c:41 #6 0xf76f671c in check_stdin_thread (arg=0x804cc60) at extensions/daemon/stdin_check.c:19 #7 0xf7647c39 in start_thread (arg=0xf74e2b70) at pthread_create.c:304 #8 0xf75b512e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 Backtrace stopped: Not enough registers or memory available to unwind further Thread 1 (Thread 0xf305cb70 (LWP 4087)): ---Type <return> to continue, or q <return> to quit--- #0 0xf76fe430 in __kernel_vsyscall () #1 0xf750f941 in *__GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 #2 0xf7512d72 in *__GI_abort () at abort.c:92 #3 0xf7508b58 in *__GI___assert_fail (assertion=0xf3ab3f37 "false", file=0xf3ab9941 "flusher.cc", line=276, function=0xf3ab9c20 "bool Flusher::step(Dispatcher&, TaskId)") at assert.c:81 #4 0xf39bcf7d in Flusher::step (this=0x9d2e190, tid=..., d=<optimized out>) at flusher.cc:276 #5 0xf39bde42 in FlusherStepper::callback (this=0xa3bc890, d=..., t=...) at flusher.cc:23 #6 0xf3981823 in Task::run (this=0xc4baee0, d=..., t=...) at dispatcher.hh:139 #7 0xf397f0d0 in Dispatcher::run (this=0x9d269d8) at dispatcher.cc:119 #8 0xf39814b6 in launch_dispatcher_thread (arg=0x9d269d8) at dispatcher.cc:28 #9 0xf7647c39 in start_thread (arg=0xf305cb70) at pthread_create.c:304 #10 0xf75b512e in clone () at ../sysdeps/unix/sysv/linux/i386/clone.S:130 Backtrace stopped: Not enough registers or memory available to unwind further (gdb) ^CQuit (gdb) ^CQuit

Steve Yen added a comment - 03/Feb/12 6:35 PM Trond, any update on this?

Steve Yen added a comment - 09/Feb/12 7:49 PM Mike, another one to look at as Trond's busy on windows fires.

Steve Yen added a comment - 14/Feb/12 11:08 AM Hi Mike, any news on this? Thanks, Steve

Mike Wiederhold added a comment - 14/Feb/12 12:07 PM This is very hard to track down due to the way we handle mutex errors. I added code to help track down the issue when it happens again. I don't think it makes sense to continue until then.