Memcached segfaults on Alestic Ubuntu 10.04LTS 32 bit AMI

Sean Lynch added a comment - 08/Feb/11 12:22 AM Using the Alestic Ubuntu 10.04LTS ami (ami-7000f019) The ns_server SASL log indicates that memcached started crashing as soon as vbucketmigrator connected, but we were able to activate the vbuckets before that just fine. He'd set a few values as well prior to this. Matt had him move our tcmalloc library out of the way and use the one from google-perftools0, and the problem went away.

Sean Lynch added a comment - 08/Feb/11 12:22 AM Stack traces: ubuntu@ip-10-112-34-115 :/opt/membase$ sudo gdb bin/memcached/memcached core GNU gdb (GDB) 7.1-ubuntu Copyright (C) 2010 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later < http://gnu.org/licenses/gpl.html > This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "i486-linux-gnu". For bug reporting instructions, please see: < http://www.gnu.org/software/gdb/bugs/ >... Reading symbols from /opt/membase/1.6.5/bin/memcached/memcached...done. [New Thread 3375] [New Thread 3320] [New Thread 3329] [New Thread 3330] [New Thread 3331] [New Thread 3332] [New Thread 3333] [New Thread 3334] [New Thread 3335] [New Thread 3368] [New Thread 3369] [New Thread 3366] [New Thread 3367] warning: Can't read pathname for load map: Input/output error. Reading symbols from /opt/membase/lib/libtcmalloc.so.0...done. Loaded symbols for /opt/membase/lib/libtcmalloc.so.0 Reading symbols from /lib/tls/i686/nosegneg/libnsl.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/tls/i686/nosegneg/libnsl.so.1 Reading symbols from /lib/tls/i686/nosegneg/librt.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/tls/i686/nosegneg/librt.so.1 Reading symbols from /lib/tls/i686/nosegneg/libresolv.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/tls/i686/nosegneg/libresolv.so.2 Reading symbols from /lib/tls/i686/nosegneg/libdl.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/tls/i686/nosegneg/libdl.so.2 Reading symbols from /lib/tls/i686/nosegneg/libpthread.so.0...(no debugging symbols found)...done. Loaded symbols for /lib/tls/i686/nosegneg/libpthread.so.0 Reading symbols from /lib/tls/i686/nosegneg/libc.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/tls/i686/nosegneg/libc.so.6 Reading symbols from /lib/tls/i686/nosegneg/libm.so.6...(no debugging symbols found)...done. Loaded symbols for /lib/tls/i686/nosegneg/libm.so.6 Reading symbols from /usr/lib/libstdc++.so.6...(no debugging symbols found)...done. Loaded symbols for /usr/lib/libstdc++.so.6 Reading symbols from /lib/libgcc_s.so.1...(no debugging symbols found)...done. Loaded symbols for /lib/libgcc_s.so.1 Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done. Loaded symbols for /lib/ld-linux.so.2 Reading symbols from ./bin/memcached/stdin_term_handler.so...done. Loaded symbols for ./bin/memcached/stdin_term_handler.so Reading symbols from ./bin/bucket_engine/bucket_engine.so...done. Loaded symbols for ./bin/bucket_engine/bucket_engine.so Reading symbols from bin/ep_engine/ep.so...done. Loaded symbols for bin/ep_engine/ep.so Core was generated by `./bin/memcached/memcached -X ./bin/memcached/stdin_term_handler.so -p 11210 -E'. Program terminated with signal 11, Segmentation fault. #0 0xb777739b in __nptl_deallocate_tsd () from /lib/tls/i686/nosegneg/libpthread.so.0 (gdb) thread apply all bt Thread 13 (Thread 3367): #0 0xb7821416 in __kernel_vsyscall () #1 0xb76dd12f in fdatasync () from /lib/tls/i686/nosegneg/libc.so.6 #2 0xb3c0c0a1 in full_fsync (id=0xb83a048, flags=2) at embedded/sqlite3.c:25510 #3 unixSync (id=0xb83a048, flags=2) at embedded/sqlite3.c:25558 #4 0xb3c592ce in sqlite3OsSync (p=0xb191008) at embedded/sqlite3.c:13413 #5 vdbeCommit (p=0xb191008) at embedded/sqlite3.c:56177 #6 sqlite3VdbeHalt (p=0xb191008) at embedded/sqlite3.c:56514 #7 0xb3c90b47 in sqlite3VdbeExec (p=0xb191008) at embedded/sqlite3.c:62196 #8 0xb3c9102a in sqlite3Step (pStmt=0xb191008) at embedded/sqlite3.c:57947 #9 sqlite3_step (pStmt=0xb191008) at embedded/sqlite3.c:58011 #10 0xb3bfc8ef in PreparedStatement::execute (this=0xb33720b4) at sqlite-pst.cc:61 #11 0xb3bfc53d in StrategicSqlite3::commit() () from bin/ep_engine/ep.so #12 0xb3bfc112 in StrategicSqlite3::storeMap<std::pair<unsigned short, unsigned short> > (this=0x961a320, clearSt=0x94f73b8, insSt=0x94f73c0, m=...)     at sqlite-kvstore.cc:234 #13 0xb3bfa77e in StrategicSqlite3::snapshotVBuckets (this=0x961a320, m=...) at sqlite-kvstore.cc:183 #14 0xb3bb05f5 in EventuallyPersistentStore::snapshotVBuckets (this=0xa3ba7e0, priority=...) at ep.cc:635 #15 0xb3bbf8a6 in SnapshotVBucketsCallback::callback(Dispatcher&, std::shared_ptr<Task>) () from bin/ep_engine/ep.so #16 0xb3bac1bc in Task::run (this=0xb83a440, d=..., t=...) at dispatcher.hh:133 #17 0xb3baaa03 in Dispatcher::run (this=0xa3c73c0) at dispatcher.cc:116 #18 0xb3bab6ae in launch_dispatcher_thread (arg=0xa3c73c0) at dispatcher.cc:26 #19 0xb7777985 in start_thread () from /lib/tls/i686/nosegneg/libpthread.so.0 #20 0xb76e413e in clone () from /lib/tls/i686/nosegneg/libc.so.6 Thread 12 (Thread 3366): #0 0xb777c781 in pthread_cond_broadcast@@GLIBC_2.3.2 () from /lib/tls/i686/nosegneg/libpthread.so.0 #1 0xb3bd0591 in SyncObject::notify() () from bin/ep_engine/ep.so #2 0xb3bd64f0 in BackfillDiskLoad::callback(Dispatcher&, std::shared_ptr<Task>) () from bin/ep_engine/ep.so #3 0xb3bac1bc in Task::run (this=0xc1d31c0, d=..., t=...) at dispatcher.hh:133 ---Type <return> to continue, or q <return> to quit--- #4 0xb3baaa03 in Dispatcher::run (this=0xb187780) at dispatcher.cc:116 #5 0xb3bab6ae in launch_dispatcher_thread (arg=0xb187780) at dispatcher.cc:26 #6 0xb7777985 in start_thread () from /lib/tls/i686/nosegneg/libpthread.so.0 #7 0xb76e413e in clone () from /lib/tls/i686/nosegneg/libc.so.6 Thread 11 (Thread 3369): #0 0xb7821416 in __kernel_vsyscall () #1 0xb777ec19 in __lll_lock_wait () from /lib/tls/i686/nosegneg/libpthread.so.0 #2 0xb778205c in _L_cond_lock_826 () from /lib/tls/i686/nosegneg/libpthread.so.0 #3 0xb7781e80 in __pthread_mutex_cond_lock () from /lib/tls/i686/nosegneg/libpthread.so.0 #4 0xb777c494 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/tls/i686/nosegneg/libpthread.so.0 #5 0xb3bce40e in SyncObject::wait (this=0x95f0000) at syncobject.hh:42 #6 SyncObject::wait (this=0x95f0000) at syncobject.hh:58 #7 TapConnMap::wait (this=0x95f0000) at tapconnmap.hh:180 #8 EventuallyPersistentEngine::notifyTapIoThread (this=0x95f0000) at ep_engine.cc:2813 #9 0xb3bce5bf in EvpNotifyTapIo (arg=0x95f0000) at ep_engine.cc:731 #10 0xb7777985 in start_thread () from /lib/tls/i686/nosegneg/libpthread.so.0 #11 0xb76e413e in clone () from /lib/tls/i686/nosegneg/libc.so.6 Thread 10 (Thread 3368): #0 0xb7821416 in __kernel_vsyscall () #1 0xb777c3d2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/tls/i686/nosegneg/libpthread.so.0 #2 0xb3ba8cfa in SyncObject::wait (this=0xa4b82d0, d=..., t=...) at syncobject.hh:42 #3 IdleTask::run (this=0xa4b82d0, d=..., t=...) at dispatcher.cc:220 #4 0xb3baaa03 in Dispatcher::run (this=0xb187690) at dispatcher.cc:116 #5 0xb3bab6ae in launch_dispatcher_thread (arg=0xb187690) at dispatcher.cc:26 #6 0xb7777985 in start_thread () from /lib/tls/i686/nosegneg/libpthread.so.0 #7 0xb76e413e in clone () from /lib/tls/i686/nosegneg/libc.so.6 Thread 9 (Thread 3335): ---Type <return> to continue, or q <return> to quit--- #0 0xb7821416 in __kernel_vsyscall () #1 0xb76e4a76 in epoll_wait () from /lib/tls/i686/nosegneg/libc.so.6 #2 0x08062d7a in epoll_dispatch (base=0x9521800, arg=0x94f8180, tv=0x9512480) at epoll.c:198 #3 0x080618cb in event_base_loop (base=0x9521800, flags=0) at event.c:516 #4 0x0805c787 in worker_libevent (arg=0x9520240) at thread.c:253 #5 0xb7777985 in start_thread () from /lib/tls/i686/nosegneg/libpthread.so.0 #6 0xb76e413e in clone () from /lib/tls/i686/nosegneg/libc.so.6 Thread 8 (Thread 3334): #0 0xb7821416 in __kernel_vsyscall () #1 0xb76e4a76 in epoll_wait () from /lib/tls/i686/nosegneg/libc.so.6 #2 0x08062d7a in epoll_dispatch (base=0x9521c00, arg=0x94f81a0, tv=0x9512600) at epoll.c:198 #3 0x080618cb in event_base_loop (base=0x9521c00, flags=0) at event.c:516 #4 0x0805c787 in worker_libevent (arg=0x95201b0) at thread.c:253 #5 0xb7777985 in start_thread () from /lib/tls/i686/nosegneg/libpthread.so.0 #6 0xb76e413e in clone () from /lib/tls/i686/nosegneg/libc.so.6 Thread 7 (Thread 3333): #0 0xb7821416 in __kernel_vsyscall () #1 0xb76e4a76 in epoll_wait () from /lib/tls/i686/nosegneg/libc.so.6 #2 0x08062d7a in epoll_dispatch (base=0x9511c00, arg=0x94f81c0, tv=0x9512780) at epoll.c:198 #3 0x080618cb in event_base_loop (base=0x9511c00, flags=0) at event.c:516 #4 0x0805c787 in worker_libevent (arg=0x9520120) at thread.c:253 #5 0xb7777985 in start_thread () from /lib/tls/i686/nosegneg/libpthread.so.0 #6 0xb76e413e in clone () from /lib/tls/i686/nosegneg/libc.so.6 Thread 6 (Thread 3332): #0 0xb7821416 in __kernel_vsyscall () #1 0xb76e4a76 in epoll_wait () from /lib/tls/i686/nosegneg/libc.so.6 #2 0x08062d7a in epoll_dispatch (base=0x9511400, arg=0x94f80c0, tv=0x9512180) at epoll.c:198 ---Type <return> to continue, or q <return> to quit--- #3 0x080618cb in event_base_loop (base=0x9511400, flags=0) at event.c:516 #4 0x0805c787 in worker_libevent (arg=0x9520090) at thread.c:253 #5 0xb7777985 in start_thread () from /lib/tls/i686/nosegneg/libpthread.so.0 #6 0xb76e413e in clone () from /lib/tls/i686/nosegneg/libc.so.6 Thread 5 (Thread 3331): #0 0xb7821416 in __kernel_vsyscall () #1 0xb76e4a76 in epoll_wait () from /lib/tls/i686/nosegneg/libc.so.6 #2 0x08062d7a in epoll_dispatch (base=0x9511800, arg=0x94f80e0, tv=0x9512300) at epoll.c:198 #3 0x080618cb in event_base_loop (base=0x9511800, flags=0) at event.c:516 #4 0x0805c787 in worker_libevent (arg=0x9520000) at thread.c:253 #5 0xb7777985 in start_thread () from /lib/tls/i686/nosegneg/libpthread.so.0 #6 0xb76e413e in clone () from /lib/tls/i686/nosegneg/libc.so.6 Thread 4 (Thread 3330): #0 0xb7821416 in __kernel_vsyscall () #1 0xb76ad286 in nanosleep () from /lib/tls/i686/nosegneg/libc.so.6 #2 0xb76ad080 in sleep () from /lib/tls/i686/nosegneg/libc.so.6 #3 0x0805e0b0 in check_isasl_db_thread (arg=0x806d918) at isasl.c:215 #4 0xb7777985 in start_thread () from /lib/tls/i686/nosegneg/libpthread.so.0 #5 0xb76e413e in clone () from /lib/tls/i686/nosegneg/libc.so.6 Thread 3 (Thread 3329): #0 0xb7821416 in __kernel_vsyscall () #1 0xb76d3bbb in read () from /lib/tls/i686/nosegneg/libc.so.6 #2 0xb767c14b in _IO_file_underflow () from /lib/tls/i686/nosegneg/libc.so.6 #3 0xb767da2b in _IO_default_uflow () from /lib/tls/i686/nosegneg/libc.so.6 #4 0xb767ee58 in __uflow () from /lib/tls/i686/nosegneg/libc.so.6 #5 0xb76746bc in getc () from /lib/tls/i686/nosegneg/libc.so.6 #6 0xb77c6812 in check_stdin_thread (arg=0x804ad60) at stdin_check.c:18 ---Type <return> to continue, or q <return> to quit--- #7 0xb7777985 in start_thread () from /lib/tls/i686/nosegneg/libpthread.so.0 #8 0xb76e413e in clone () from /lib/tls/i686/nosegneg/libc.so.6 Thread 2 (Thread 3320): #0 0xb7821416 in __kernel_vsyscall () #1 0xb76e4a76 in epoll_wait () from /lib/tls/i686/nosegneg/libc.so.6 #2 0x08062d7a in epoll_dispatch (base=0x9511000, arg=0x94f8120, tv=0x9512000) at epoll.c:198 #3 0x080618cb in event_base_loop (base=0x9511000, flags=0) at event.c:516 #4 0x0805230a in main (argc=15, argv=0xbf8a1f44) at memcached.c:7203 Thread 1 (Thread 3375): #0 0xb777739b in __nptl_deallocate_tsd () from /lib/tls/i686/nosegneg/libpthread.so.0 #1 0xb7777993 in start_thread () from /lib/tls/i686/nosegneg/libpthread.so.0 #2 0xb76e413e in clone () from /lib/tls/i686/nosegneg/libc.so.6 (gdb)

Sean Lynch added a comment - 08/Feb/11 12:45 AM This is with http://builds.hq.northscale.net/latestbuilds/membase-server-community_x86_1.6.5.deb, which I didn't initially realize is *not* the build on the web site (that one says "enterprise" instead of "community"). He used that because one can't use wget to download the one on the web site. He's testing with the one off the web site now.

Sean Lynch added a comment - 08/Feb/11 12:47 AM He tested with the build off the web site, too, same problem.

Perry Krug added a comment - 08/Feb/11 6:31 PM This appears to be an issue with the specific AMI as we've tested Ubuntu 10.04 32-bit in other environments with success. Would be good to get verification that this works on a verified AMI (the 'rightscale' provided ones are known to work) while we work to diagnose and fix the present issue.

Thuan Nguyen added a comment - 09/Feb/11 11:49 AM I am able to reproduce this bug on ec2

Matt Ingenthron added a comment - 09/Feb/11 11:52 AM This is likely fixed with the change to tcmalloc-minimal, so please test with 1.6.5.1.

Garret Heaton added a comment - 09/Feb/11 9:46 PM @Perry Confirmed working using RightScale ami-f8b35e91 on EC2 m1.small instances. @Thuan That's the workaround suggested to me in IRC yesterday after first reporting the bug. Did the trick for me as well.

Aleksey Kondratenko added a comment - 10/Feb/11 9:29 PM I think this might be affected by TLS usage of libtcmalloc. I've built version that uses 'xen-friendy' TLS access method. But was unable to reproduce original bug. So I cannot verify it. Uploading my libtcmalloc built for someone else to test.

Aleksey Kondratenko added a comment - 14/Jun/12 11:34 AM Voltron builds all i386 code with -mno-tls-direct-seg-refs that helps with some (presumably Xen) bugs w.r.t thread local storage access. We do that since before 1.7.0 AFAIR