memcached event_handler crash

osx dev environment: I wrote some scripts to do 1 million rebalancings and left it on. One time, memcached crashed, according to the Console...

Process: memcached [10386]
Path: /Users/steveyen/work/membase/dev/memcached/memcached
Identifier: memcached
Version: ??? (???)
Code Type: X86-64 (Native)
Parent Process: beam.smp [2245]

Date/Time: 2010-11-06 08:53:08.200 -0700
OS Version: Mac OS X 10.6.3 (10D578)
Report Version: 6

Exception Type: EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x00000000000000e8
Crashed Thread: 5

Thread 0: Dispatch queue: com.apple.main-thread
0 libSystem.B.dylib 0x00007fff841bc4ea kevent + 10
1 libevent-1.4.2.dylib 0x000000010003b8a3 kq_dispatch + 83
2 libevent-1.4.2.dylib 0x000000010002cd08 event_base_loop + 664
3 memcached 0x0000000100005c39 main + 4409 (memcached.c:7097)
4 memcached 0x0000000100001444 start + 52

Thread 1:
0 libSystem.B.dylib 0x00007fff841a5362 read$NOCANCEL + 10
1 libSystem.B.dylib 0x00007fff841f48be __sread + 16
2 libSystem.B.dylib 0x00007fff841ec604 _sread + 19
3 libSystem.B.dylib 0x00007fff841ec49a __srefill + 41
4 libSystem.B.dylib 0x00007fff841f765a __srget + 17
5 libSystem.B.dylib 0x00007fff841f7616 getc + 63
6 stdin_term_handler.so 0x0000000100066dd8 check_stdin_thread + 40 (stdin_check.c:17)
7 libSystem.B.dylib 0x00007fff841dc8b6 _pthread_start + 331
8 libSystem.B.dylib 0x00007fff841dc769 thread_start + 13

Thread 2:
0 libSystem.B.dylib 0x00007fff841de316 __semwait_signal + 10
1 libSystem.B.dylib 0x00007fff841de1a5 nanosleep + 148
2 libSystem.B.dylib 0x00007fff8422b0a4 sleep + 61
3 memcached 0x0000000100014b48 check_isasl_db_thread + 88 (isasl.c:192)
4 libSystem.B.dylib 0x00007fff841dc8b6 _pthread_start + 331
5 libSystem.B.dylib 0x00007fff841dc769 thread_start + 13

Thread 3:
0 libSystem.B.dylib 0x00007fff841bc4ea kevent + 10
1 libevent-1.4.2.dylib 0x000000010003b8a3 kq_dispatch + 83
2 libevent-1.4.2.dylib 0x000000010002cd08 event_base_loop + 664
3 memcached 0x0000000100013091 worker_libevent + 65 (thread.c:254)
4 libSystem.B.dylib 0x00007fff841dc8b6 _pthread_start + 331
5 libSystem.B.dylib 0x00007fff841dc769 thread_start + 13

Thread 4:
0 libSystem.B.dylib 0x00007fff841bc4ea kevent + 10
1 libevent-1.4.2.dylib 0x000000010003b8a3 kq_dispatch + 83
2 libevent-1.4.2.dylib 0x000000010002cd08 event_base_loop + 664
3 memcached 0x0000000100013091 worker_libevent + 65 (thread.c:254)
4 libSystem.B.dylib 0x00007fff841dc8b6 _pthread_start + 331
5 libSystem.B.dylib 0x00007fff841dc769 thread_start + 13

Thread 5 Crashed:
0 memcached 0x0000000100006c1b event_handler + 731 (memcached.c:5450)
1 libevent-1.4.2.dylib 0x000000010002cff1 event_base_loop + 1409
2 memcached 0x0000000100013091 worker_libevent + 65 (thread.c:254)
3 libSystem.B.dylib 0x00007fff841dc8b6 _pthread_start + 331
4 libSystem.B.dylib 0x00007fff841dc769 thread_start + 13

Thread 6:
0 libSystem.B.dylib 0x00007fff841bc4ea kevent + 10
1 libevent-1.4.2.dylib 0x000000010003b8a3 kq_dispatch + 83
2 libevent-1.4.2.dylib 0x000000010002cd08 event_base_loop + 664
3 memcached 0x0000000100013091 worker_libevent + 65 (thread.c:254)
4 libSystem.B.dylib 0x00007fff841dc8b6 _pthread_start + 331
5 libSystem.B.dylib 0x00007fff841dc769 thread_start + 13

Thread 7:
0 libSystem.B.dylib 0x00007fff841bc4ea kevent + 10
1 libevent-1.4.2.dylib 0x000000010003b8a3 kq_dispatch + 83
2 libevent-1.4.2.dylib 0x000000010002cd08 event_base_loop + 664
3 memcached 0x0000000100013091 worker_libevent + 65 (thread.c:254)
4 libSystem.B.dylib 0x00007fff841dc8b6 _pthread_start + 331
5 libSystem.B.dylib 0x00007fff841dc769 thread_start + 13

Thread 8:
0 libSystem.B.dylib 0x00007fff841e516a fsync + 10
1 ep.0.so 0x0000000101865fee full_fsync + 72 (sqlite3.c:25502)
2 ep.0.so 0x000000010186603a unixSync + 68 (sqlite3.c:25558)
3 ep.0.so 0x000000010185c44e sqlite3OsSync + 35 (sqlite3.c:13414)
4 ep.0.so 0x0000000101888ad3 vdbeCommit + 1107 (sqlite3.c:56177)
5 ep.0.so 0x000000010188906c sqlite3VdbeHalt + 497 (sqlite3.c:56514)
6 ep.0.so 0x00000001018913b4 sqlite3VdbeExec + 16886 (sqlite3.c:62196)
7 ep.0.so 0x000000010188b28d sqlite3Step + 353 (sqlite3.c:57947)
8 ep.0.so 0x000000010188b44f sqlite3_step + 127 (sqlite3.c:58011)
9 ep.0.so 0x0000000101845a1c PreparedStatement::execute() + 194 (sqlite-pst.cc:56)
10 ep.0.so 0x0000000101812643 StrategicSqlite3::execute(char const*) + 47 (sqlite-kvstore.hh:138)
11 ep.0.so 0x00000001018126a4 StrategicSqlite3::commit() + 40 (sqlite-kvstore.hh:76)
12 ep.0.so 0x000000010184534f bool StrategicSqlite3::storeMap<std::pair<unsigned short, unsigned short> >(PreparedStatement*, PreparedStatement*, std::map<std::pair<unsigned short, unsigned short>, std::string, std::less<std::pair<unsigned short, unsigned short> >, std::allocator<std::pair<std::pair<unsigned short, unsigned short> const, std::string> > > const&, bool) + 205 (sqlite-kvstore.cc:231)
13 ep.0.so 0x0000000101843821 StrategicSqlite3::snapshotVBuckets(std::map<std::pair<unsigned short, unsigned short>, std::string, std::less<std::pair<unsigned short, unsigned short> >, std::allocator<std::pair<std::pair<unsigned short, unsigned short> const, std::string> > > const&) + 71 (sqlite-kvstore.cc:184)
14 ep.0.so 0x000000010180ae4c EventuallyPersistentStore::snapshotVBuckets(Priority const&) + 126 (ep.cc:603)
15 ep.0.so 0x000000010181e849 SnapshotVBucketsCallback::callback(Dispatcher&, std::tr1::shared_ptr<Task>) + 41 (ep.cc:136)
16 ep.0.so 0x000000010180399a Task::run(Dispatcher&, std::tr1::shared_ptr<Task>) + 76 (dispatcher.hh:122)
17 ep.0.so 0x0000000101802259 Dispatcher::run() + 613 (dispatcher.cc:113)
18 ep.0.so 0x00000001018026ea launch_dispatcher_thread + 30 (dispatcher.cc:26)
19 libSystem.B.dylib 0x00007fff841dc8b6 _pthread_start + 331
20 libSystem.B.dylib 0x00007fff841dc769 thread_start + 13

Thread 9:
0 libSystem.B.dylib 0x00007fff841de316 __semwait_signal + 10
1 libSystem.B.dylib 0x00007fff841e2131 _pthread_cond_wait + 1286
2 ep.0.so 0x0000000101805670 SyncObject::wait() + 34 (syncobject.hh:31)
3 ep.0.so 0x000000010180209f Dispatcher::run() + 171 (dispatcher.cc:84)
4 ep.0.so 0x00000001018026ea launch_dispatcher_thread + 30 (dispatcher.cc:26)
5 libSystem.B.dylib 0x00007fff841dc8b6 _pthread_start + 331
6 libSystem.B.dylib 0x00007fff841dc769 thread_start + 13

Thread 10:
0 libSystem.B.dylib 0x00007fff841de316 __semwait_signal + 10
1 libSystem.B.dylib 0x00007fff841e2131 _pthread_cond_wait + 1286
2 ep.0.so 0x00000001018057c8 SyncObject::wait(timeval const&) + 72 (syncobject.hh:42)
3 ep.0.so 0x000000010182ff9e SyncObject::wait(double) + 58 (syncobject.hh:58)
4 ep.0.so 0x000000010182ffd1 TapConnMap::wait(double) + 45 (tapconnmap.hh:127)
5 ep.0.so 0x0000000101820e4c EventuallyPersistentEngine::notifyTapIoThread() + 84 (ep_engine.cc:2425)
6 ep.0.so 0x0000000101820e75 EvpNotifyTapIo + 21 (ep_engine.cc:731)
7 libSystem.B.dylib 0x00007fff841dc8b6 _pthread_start + 331
8 libSystem.B.dylib 0x00007fff841dc769 thread_start + 13

Thread 5 crashed with X86 Thread State (64-bit):
  rax: 0x0000000000000000 rbx: 0x000000010082ce20 rcx: 0x00007fff841b5b56 rdx: 0x0000000000000000
  rdi: 0x0000000100020d08 rsi: 0x0000000000000000 rbp: 0x0000000100509e50 rsp: 0x00000001005095f0
   r8: 0x0000000100020d10 r9: 0x0000000100020c60 r10: 0x0000000100020d50 r11: 0x0000000000000246
  r12: 0x000000010020ddf8 r13: 0x000000010082cec8 r14: 0x0000000000000000 r15: 0x00000001005095f0
  rip: 0x0000000100006c1b rfl: 0x0000000000010246 cr2: 0x00000000000000e8

Binary Images:
       0x100000000 - 0x10001eff7 +memcached ??? (???) <4962C645-1D21-4662-5FB1-AFD9DC6EF1EE> /Users/steveyen/work/membase/dev/ns_server/bin/memcached/memcached
       0x10002b000 - 0x100041fef +libevent-1.4.2.dylib 4.3.0 (compatibility 4.0.0) <37AB7C78-F18E-867D-50BE-8EC01C1F8538> /usr/local/Cellar/libevent/1.4.13/lib/libevent-1.4.2.dylib
       0x100066000 - 0x100066ff7 +stdin_term_handler.so ??? (???) <CF045105-EB9F-8003-0006-92952B9F2961> /Users/steveyen/work/membase/dev/memcached/.libs/stdin_term_handler.so
       0x1000ec000 - 0x1000f1fff +bucket_engine.0.so ??? (???) <F248817B-983A-81D1-6CDB-E8C0FC6BE0CE> /Users/steveyen/work/membase/dev/bucket_engine/.libs/bucket_engine.0.so
       0x101800000 - 0x101933fef +ep.0.so ??? (???) <064406C5-CB90-7080-F6D3-9C436305325B> /Users/steveyen/work/membase/dev/ep-engine/.libs/ep.0.so
    0x7fff5fc00000 - 0x7fff5fc3bdef dyld 132.1 (???) <B633F790-4DDB-53CD-7ACF-2A3682BCEA9F> /usr/lib/dyld
    0x7fff80842000 - 0x7fff80863fff libresolv.9.dylib 40.0.0 (compatibility 1.0.0) <1AE68BBB-6536-125C-DE2A-13CA916D0EC4> /usr/lib/libresolv.9.dylib
    0x7fff83247000 - 0x7fff8324bff7 libmathCommon.A.dylib 315.0.0 (compatibility 1.0.0) <95718673-FEEE-B6ED-B127-BCDBDB60D4E5> /usr/lib/system/libmathCommon.A.dylib
    0x7fff841a2000 - 0x7fff84361ff7 libSystem.B.dylib 125.0.1 (compatibility 1.0.0) <CB9A4929-61AF-DE71-5635-133E9EC95783> /usr/lib/libSystem.B.dylib
    0x7fff87818000 - 0x7fff87895fef libstdc++.6.dylib 7.9.0 (compatibility 7.0.0) <35ECA411-2C08-FD7D-11B1-1B7A04921A5C> /usr/lib/libstdc++.6.dylib
    0x7fffffe00000 - 0x7fffffe01fff libSystem.B.dylib ??? (???) <CB9A4929-61AF-DE71-5635-133E9EC95783> /usr/lib/libSystem.B.dylib