Couchbase Server

standalone Moxi core dumped in malloc_consolidate(), at the end of removing a node

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Critical Critical
  • Resolution: Cannot Reproduce
  • Affects Version/s: 2010-12-16, 1.6.5.3
  • Fix Version/s: Backlog
  • Component/s: moxi
  • Security Level: Public
  • Labels:
    None

Description

Using stand alone moxi on 10.2.1.12
Running a load against cluster of 3 (12,13,14).
Removing 14.
Right when rebalancing ended, moxi crashed with:

[root@localhost ~]# /opt/membase/bin/moxi/moxi -u nobody -Z port_listen=11212,time_stats=1,usr=Administrator,pwd=password,default_bucket_name=default,downstream_max=1024,downstream_conn_max=16,connect_max_errors=3,connect_retry_interval=30000,connect_timeout=400,auth_timeout=100,cycle=200,downstream_conn_queue_timeout=200,downstream_timeout=5000,wait_queue_timeout=200 -z url=http://10.2.1.12:8091/pools/default/saslBucketsStreaming -p 0 -Y y -O stderr
Segmentation fault (core dumped)
Cores should be enabled on 10.2.1.12, but i don't know where to find them.

Since this is a standalone moxi i don't have logs.
I suspect that this happened right when the vubcket map changed.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Steve Yen added a comment -
Core was generated by `/opt/membase/bin/moxi/moxi -u nobody -Z port_listen=11212,time_stats=1,usr=Admi'.
Program terminated with signal 11, Segmentation fault.
#0 0x000000310f270454 in malloc_consolidate () from /lib64/libc.so.6
(gdb) thread all ap bt
No symbol "all" in current context.
(gdb) thread ap al bt

Thread 8 (Thread 1922):
#0 0x000000310f2d4018 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043f600 in epoll_dispatch (base=0x10b8b4a0, arg=0x10b8a730,
    tv=<value optimized out>) at epoll.c:198
#2 0x000000000043db31 in event_base_loop (base=0x10b8b4a0, flags=0)
    at event.c:516
#3 0x0000000000408f5b in main (argc=13, argv=0x7fff0656f8c8)
    at memcached.c:4966

Thread 7 (Thread 1924):
#0 0x000000310f2d4018 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043f600 in epoll_dispatch (base=0x10ba30a0, arg=0x10ba36d0,
    tv=<value optimized out>) at epoll.c:198
#2 0x000000000043db31 in event_base_loop (base=0x10ba30a0, flags=0)
    at event.c:516
#3 0x00000000004138d5 in worker_libevent (arg=<value optimized out>)
    at thread.c:272
#4 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0
#5 0x000000310f2d3c2d in clone () from /lib64/libc.so.6

Thread 6 (Thread 1925):
#0 0x000000310f2d4018 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043f600 in epoll_dispatch (base=0x10ba7b10, arg=0x10ba8140,
    tv=<value optimized out>) at epoll.c:198
#2 0x000000000043db31 in event_base_loop (base=0x10ba7b10, flags=0)
    at event.c:516
#3 0x00000000004138d5 in worker_libevent (arg=<value optimized out>)
    at thread.c:272
#4 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0
#5 0x000000310f2d3c2d in clone () from /lib64/libc.so.6

Thread 5 (Thread 1926):
#0 0x000000310f2d4018 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000043f600 in epoll_dispatch (base=0x10bac580, arg=0x10bacbb0,
    tv=<value optimized out>) at epoll.c:198
#2 0x000000000043db31 in event_base_loop (base=0x10bac580, flags=0)
    at event.c:516
---Type <return> to continue, or q <return> to quit---
#3 0x00000000004138d5 in worker_libevent (arg=<value optimized out>)
    at thread.c:272
#4 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0
#5 0x000000310f2d3c2d in clone () from /lib64/libc.so.6

Thread 4 (Thread 1927):
#0 0x000000310fe0ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()
   from /lib64/libpthread.so.0
#1 0x0000000000412b40 in assoc_maintenance_thread (arg=<value optimized out>)
    at assoc.c:220
#2 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0
#3 0x000000310f2d3c2d in clone () from /lib64/libc.so.6

Thread 3 (Thread 1928):
#0 0x000000310f2cae46 in poll () from /lib64/libc.so.6
#1 0x0000000000446e36 in Curl_socket_ready ()
#2 0x0000000000000000 in ?? ()

Thread 2 (Thread 1929):
#0 0x000000310f2c5f3b in read () from /lib64/libc.so.6
#1 0x000000310f26cc07 in _IO_new_file_underflow () from /lib64/libc.so.6
#2 0x000000310f26d5ce in _IO_default_uflow_internal () from /lib64/libc.so.6
#3 0x000000310f268e8b in getc () from /lib64/libc.so.6
#4 0x000000000042e2cf in check_stdin_thread (arg=<value optimized out>)
    at stdin_check.c:16
#5 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0
#6 0x000000310f2d3c2d in clone () from /lib64/libc.so.6

Thread 1 (Thread 1923):
#0 0x000000310f270454 in malloc_consolidate () from /lib64/libc.so.6
#1 0x000000310f272a1a in _int_malloc () from /lib64/libc.so.6
#2 0x000000310f274bee in malloc () from /lib64/libc.so.6
#3 0x00000000004123e9 in do_item_alloc (
    key=0x2aaaac01400c "sharon6m-dataload7331\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\37---Type <return> to continue, or q <return> to quit---
7\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377"..., nkey=21,
    flags=<value optimized out>, exptime=0, nbytes=1810) at items.c:98
#4 0x0000000000424491 in cproxy_process_a2b_downstream_nread (
    c=0x2aaaac004990) at cproxy_protocol_a2b.c:568
#5 0x000000000040c9b3 in complete_nread (c=0x2aaaac004990) at memcached.c:1938
#6 drive_machine (c=0x2aaaac004990) at memcached.c:3499
#7 0x000000000043dbfa in event_process_active (base=0x10b9e630, flags=0)
    at event.c:385
#8 event_base_loop (base=0x10b9e630, flags=0) at event.c:525
#9 0x00000000004138d5 in worker_libevent (arg=<value optimized out>)
    at thread.c:272
#10 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0
#11 0x000000310f2d3c2d in clone () from /lib64/libc.so.6
(gdb)
Show
Steve Yen added a comment - Core was generated by `/opt/membase/bin/moxi/moxi -u nobody -Z port_listen=11212,time_stats=1,usr=Admi'. Program terminated with signal 11, Segmentation fault. #0 0x000000310f270454 in malloc_consolidate () from /lib64/libc.so.6 (gdb) thread all ap bt No symbol "all" in current context. (gdb) thread ap al bt Thread 8 (Thread 1922): #0 0x000000310f2d4018 in epoll_wait () from /lib64/libc.so.6 #1 0x000000000043f600 in epoll_dispatch (base=0x10b8b4a0, arg=0x10b8a730,     tv=<value optimized out>) at epoll.c:198 #2 0x000000000043db31 in event_base_loop (base=0x10b8b4a0, flags=0)     at event.c:516 #3 0x0000000000408f5b in main (argc=13, argv=0x7fff0656f8c8)     at memcached.c:4966 Thread 7 (Thread 1924): #0 0x000000310f2d4018 in epoll_wait () from /lib64/libc.so.6 #1 0x000000000043f600 in epoll_dispatch (base=0x10ba30a0, arg=0x10ba36d0,     tv=<value optimized out>) at epoll.c:198 #2 0x000000000043db31 in event_base_loop (base=0x10ba30a0, flags=0)     at event.c:516 #3 0x00000000004138d5 in worker_libevent (arg=<value optimized out>)     at thread.c:272 #4 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0 #5 0x000000310f2d3c2d in clone () from /lib64/libc.so.6 Thread 6 (Thread 1925): #0 0x000000310f2d4018 in epoll_wait () from /lib64/libc.so.6 #1 0x000000000043f600 in epoll_dispatch (base=0x10ba7b10, arg=0x10ba8140,     tv=<value optimized out>) at epoll.c:198 #2 0x000000000043db31 in event_base_loop (base=0x10ba7b10, flags=0)     at event.c:516 #3 0x00000000004138d5 in worker_libevent (arg=<value optimized out>)     at thread.c:272 #4 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0 #5 0x000000310f2d3c2d in clone () from /lib64/libc.so.6 Thread 5 (Thread 1926): #0 0x000000310f2d4018 in epoll_wait () from /lib64/libc.so.6 #1 0x000000000043f600 in epoll_dispatch (base=0x10bac580, arg=0x10bacbb0,     tv=<value optimized out>) at epoll.c:198 #2 0x000000000043db31 in event_base_loop (base=0x10bac580, flags=0)     at event.c:516 ---Type <return> to continue, or q <return> to quit--- #3 0x00000000004138d5 in worker_libevent (arg=<value optimized out>)     at thread.c:272 #4 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0 #5 0x000000310f2d3c2d in clone () from /lib64/libc.so.6 Thread 4 (Thread 1927): #0 0x000000310fe0ab99 in pthread_cond_wait@@GLIBC_2.3.2 ()    from /lib64/libpthread.so.0 #1 0x0000000000412b40 in assoc_maintenance_thread (arg=<value optimized out>)     at assoc.c:220 #2 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0 #3 0x000000310f2d3c2d in clone () from /lib64/libc.so.6 Thread 3 (Thread 1928): #0 0x000000310f2cae46 in poll () from /lib64/libc.so.6 #1 0x0000000000446e36 in Curl_socket_ready () #2 0x0000000000000000 in ?? () Thread 2 (Thread 1929): #0 0x000000310f2c5f3b in read () from /lib64/libc.so.6 #1 0x000000310f26cc07 in _IO_new_file_underflow () from /lib64/libc.so.6 #2 0x000000310f26d5ce in _IO_default_uflow_internal () from /lib64/libc.so.6 #3 0x000000310f268e8b in getc () from /lib64/libc.so.6 #4 0x000000000042e2cf in check_stdin_thread (arg=<value optimized out>)     at stdin_check.c:16 #5 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0 #6 0x000000310f2d3c2d in clone () from /lib64/libc.so.6 Thread 1 (Thread 1923): #0 0x000000310f270454 in malloc_consolidate () from /lib64/libc.so.6 #1 0x000000310f272a1a in _int_malloc () from /lib64/libc.so.6 #2 0x000000310f274bee in malloc () from /lib64/libc.so.6 #3 0x00000000004123e9 in do_item_alloc (     key=0x2aaaac01400c "sharon6m-dataload7331\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\37---Type <return> to continue, or q <return> to quit--- 7\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377\377"..., nkey=21,     flags=<value optimized out>, exptime=0, nbytes=1810) at items.c:98 #4 0x0000000000424491 in cproxy_process_a2b_downstream_nread (     c=0x2aaaac004990) at cproxy_protocol_a2b.c:568 #5 0x000000000040c9b3 in complete_nread (c=0x2aaaac004990) at memcached.c:1938 #6 drive_machine (c=0x2aaaac004990) at memcached.c:3499 #7 0x000000000043dbfa in event_process_active (base=0x10b9e630, flags=0)     at event.c:385 #8 event_base_loop (base=0x10b9e630, flags=0) at event.c:525 #9 0x00000000004138d5 in worker_libevent (arg=<value optimized out>)     at thread.c:272 #10 0x000000310fe064a7 in start_thread () from /lib64/libpthread.so.0 #11 0x000000310f2d3c2d in clone () from /lib64/libc.so.6 (gdb)
Hide
Permalink
Steve Yen added a comment -
Need some valgrind'ing on moxi.
Show
Steve Yen added a comment - Need some valgrind'ing on moxi.
Hide
Permalink
Steve Yen added a comment -
See MB-1581 on moxi + valgrind
Show
Steve Yen added a comment - See MB-1581 on moxi + valgrind
Hide
Permalink
Steve Yen added a comment -
Haven't heard of any other instances of malloc_consolidate issues since this report.
Show
Steve Yen added a comment - Haven't heard of any other instances of malloc_consolidate issues since this report.

People

  • Assignee:
    Steve Yen
    Reporter:
    Sharon Barr
Vote (0)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved: