Home | Forums | Couchbase | Couchbase Server 1.8.x

MOXI - status 139 - segfault problems

No replies
Thu, 08/23/2012 - 04:10
JDILloyd
Offline
Joined: 04/18/2012
Groups: None

We have 2 couchbase 1.8.0 clusters, 1 of them is fine running a couchbase/membase bucket. The other cluster running a single memcached bucket keeps having issues. MOXI keeps exiting with 139 status (segfault) on every node in the cluster.

The systems in the cluster are all running CentOS 5.7, Intel Xeon E5620, 24GB RAM, 2x 600GB SAS Drives in RAID0.

Bucket Info
default Memcached 4 servers 414MB/23.4GB 0B

Any assistance on the issue would be greatly appreciated.

====================

UI Log

Port server moxi on node 'ns_1@10.33.148.204' exited with status 139. Restarting. Messages: 2012-08-23 04:19:07: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (8)
2012-08-23 04:19:07: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (8)	ns_port_server000	ns_1@10.33.148.204	11:22:54 - Thu Aug 23, 2012

ns_log

INFO REPORT  <5883.285.0>                                   2012-08-18 01:30:02
===============================================================================
 
menelaus_web streaming socket closed by client
 
INFO REPORT  <5883.292.0>                                   2012-08-18 01:30:02
===============================================================================
 
ns_log: logging ns_port_server:0:Port server moxi on node 'ns_1@10.33.148.204' exited with status 139. Restarting. Messages: 2012-08-16 11:02:24: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (8)
2012-08-16 11:02:24: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (8)
 
ERROR REPORT  <5883.292.0>                                  2012-08-18 01:30:02
===============================================================================
 
** Generic server <5883.292.0> terminating 
** Last message in was {#Port<5883.2185>,{exit_status,139}}
** When Server state == {state,#Port<5883.2185>,moxi,
                               {["2012-08-16 11:02:24: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (8)",
                                 "2012-08-16 11:02:24: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (8)",
                                 empty],
                                [empty,empty,empty,empty,empty,empty,empty,
                                 empty,empty,empty,empty,empty,empty,empty,
                                 empty,empty,empty,empty,empty,empty,empty,
                                 empty,empty,empty,empty,empty,empty,empty,
                                 empty,empty,empty,empty,empty,empty,empty,
                                 empty,empty,empty,empty,empty,empty,empty,
                                 empty,empty,empty,empty,empty,empty,empty,
                                 empty,empty,empty,empty,empty,empty,empty,
                                 empty,empty,empty,empty,empty,empty,empty,
                                 empty,empty,empty,empty,empty,empty,empty,
                                 empty,empty,empty,empty,empty,empty,empty,
                                 empty,empty,empty,empty,empty,empty,empty,
                                 empty,empty,empty,empty,empty,empty,empty,
                                 empty,empty,empty,empty,empty,empty]},
                               undefined,[],0,true}
** Reason for termination == 
** {abnormal,139}
 
CRASH REPORT  <5883.292.0>                                  2012-08-18 01:30:02
===============================================================================
Crashing process                                                               
   initial_call                           {ns_port_server,init,['Argument__1']}
   pid                                                             <5883.292.0>
   registered_name                                                           []
   error_info
         {exit,{abnormal,139},
              [{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}]}
   ancestors
         [<5883.291.0>,ns_port_sup,ns_server_sup,ns_server_cluster_sup,
         <5883.50.0>]
   messages                                  [{'EXIT',#Port<5883.2185>,normal}]
   links                                                         [<5883.291.0>]
   dictionary                                                                []
   trap_exit                                                               true
   status                                                               running
   heap_size                                                               1597
   stack_size                                                                24
   reductions                                                              1717
 
 
INFO REPORT  <5883.291.0>                                   2012-08-18 01:30:02
===============================================================================
 
Cushion managed supervisor for moxi failed:  {abnormal,139}
 
ERROR REPORT  <5883.291.0>                                  2012-08-18 01:30:02
===============================================================================
 
** Generic server <5883.291.0> terminating 
** Last message in was {die,{error,cushioned_supervisor,{abnormal,139}}}
** When Server state == {state,moxi,5000,{1345,136544,798588},undefined}
** Reason for termination == 
** {error,cushioned_supervisor,{abnormal,139}}
 
CRASH REPORT  <5883.291.0>                                  2012-08-18 01:30:02
===============================================================================
Crashing process                                                               
   initial_call                       {supervisor_cushion,init,['Argument__1']}
   pid                                                             <5883.291.0>
   registered_name                                                           []
   error_info
         {exit,{error,cushioned_supervisor,{abnormal,139}},
              [{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}]}
   ancestors      [ns_port_sup,ns_server_sup,ns_server_cluster_sup,<5883.50.0>]
   messages                                                                  []
   links                                                         [<5883.289.0>]
   dictionary                                                                []
   trap_exit                                                               true
   status                                                               running
   heap_size                                                                987
   stack_size                                                                24
   reductions                                                               287
 
 
SUPERVISOR REPORT  <5883.289.0>                             2012-08-18 01:30:02
===============================================================================
Reporting supervisor                                        {local,ns_port_sup}
 
Child process                                                                  
   errorContext                                                child_terminated
   reason                           {error,cushioned_supervisor,{abnormal,139}}
   pid                                                             <5883.291.0>
   name
         {moxi,"/opt/couchbase/bin/moxi",
              ["-Z",
               "port_listen=11211,default_bucket_name=default,downstream_max=1024,downstream_conn_max=4,connect_max_errors=5,connect_retry_interval=30000,connect_timeout=400,auth_timeout=100,cycle=200,downstream_conn_queue_timeout=200,downstream_timeout=5000,wait_queue_timeout=200",
               "-z",
               "url=http://127.0.0.1:8091/pools/default/saslBucketsStreaming",
               "-p","0","-Y","y","-O","stderr",[]],
              [{env,[{"EVENT_NOSELECT","1"},
                     {"MOXI_SASL_PLAIN_USR","jdicouch"},
                     {"MOXI_SASL_PLAIN_PWD","RBr0Kq)m"}]},
               use_stdio,exit_status,port_server_send_eol,stderr_to_stdout,
               stream]}
   mfargs
         {supervisor_cushion,start_link,
                            [moxi,5000,ns_port_server,start_link,
                             [moxi,"/opt/couchbase/bin/moxi",
                              ["-Z",
                               "port_listen=11211,default_bucket_name=default,downstream_max=1024,downstream_conn_max=4,connect_max_errors=5,connect_retry_interval=30000,connect_timeout=400,auth_timeout=100,cycle=200,downstream_conn_queue_timeout=200,downstream_timeout=5000,wait_queue_timeout=200",
                               "-z",
                               "url=http://127.0.0.1:8091/pools/default/saslBucketsStreaming",
                               "-p","0","-Y","y","-O","stderr",[]],
                              [{env,[{"EVENT_NOSELECT","1"},
                                     {"MOXI_SASL_PLAIN_USR","jdicouch"},
                                     {"MOXI_SASL_PLAIN_PWD","RBr0Kq)m"}]},
                               use_stdio,exit_status,port_server_send_eol,
                               stderr_to_stdout,stream]]]}
   restart_type                                                       permanent
   shutdown                                                               10000
   child_type                                                            worker
 
 
INFO REPORT  <5883.13438.20>                                2012-08-18 01:30:02
===============================================================================
 
starting ns_port_server with delay of 5000
 
PROGRESS REPORT  <5883.289.0>                               2012-08-18 01:30:02
===============================================================================
supervisor                                                  {local,ns_port_sup}
started
         [{pid,<5883.13438.20>},
         {name,
             {moxi,"/opt/couchbase/bin/moxi",
                 ["-Z",
                  "port_listen=11211,default_bucket_name=default,downstream_max=1024,downstream_conn_max=4,connect_max_errors=5,connect_retry_interval=30000,connect_timeout=400,auth_timeout=100,cycle=200,downstream_conn_queue_timeout=200,downstream_timeout=5000,wait_queue_timeout=200",
                  "-z",
                  "url=http://127.0.0.1:8091/pools/default/saslBucketsStreaming",
                  "-p","0","-Y","y","-O","stderr",[]],
                 [{env,
                      [{"EVENT_NOSELECT","1"},
                       {"MOXI_SASL_PLAIN_USR","jdicouch"},
                       {"MOXI_SASL_PLAIN_PWD","RBr0Kq)m"}]},
                  use_stdio,exit_status,port_server_send_eol,
                  stderr_to_stdout,stream]}},
         {mfargs,
             {supervisor_cushion,start_link,
                 [moxi,5000,ns_port_server,start_link,
                  [moxi,"/opt/couchbase/bin/moxi",
                   ["-Z",
                    "port_listen=11211,default_bucket_name=default,downstream_max=1024,downstream_conn_max=4,connect_max_errors=5,connect_retry_interval=30000,connect_timeout=400,auth_timeout=100,cycle=200,downstream_conn_queue_timeout=200,downstream_timeout=5000,wait_queue_timeout=200",
                    "-z",
                    "url=http://127.0.0.1:8091/pools/default/saslBucketsStreaming",
                    "-p","0","-Y","y","-O","stderr",[]],
                   [{env,
                        [{"EVENT_NOSELECT","1"},
                         {"MOXI_SASL_PLAIN_USR","jdicouch"},
                         {"MOXI_SASL_PLAIN_PWD","RBr0Kq)m"}]},
                    use_stdio,exit_status,port_server_send_eol,
                    stderr_to_stdout,stream]]]}},
         {restart_type,permanent},
         {shutdown,10000},
         {child_type,worker}]
 
 
INFO REPORT  <5883.13439.20>                                2012-08-18 01:30:03
===============================================================================
 
moxi<0.13439.20>: 2012-08-18 01:30:02: (cproxy_config.c.317) env: MOXI_SASL_PLAIN_USR (8)
moxi<0.13439.20>: 2012-08-18 01:30:02: (cproxy_config.c.326) env: MOXI_SASL_PLAIN_PWD (8)
 
INFO REPORT  <5883.254.0>                                   2012-08-18 01:30:16
===============================================================================
 
ns_1@10.33.148.204:<5883.254.0>:ns_config_rep:257: Pulling config from: 'ns_1@10.33.148.178'
 
 
INFO REPORT  <5883.281.0>                                   2012-08-18 01:30:24
===============================================================================
 
ns_1@10.33.148.204:<5883.281.0>:ns_doctor:86: Current node statuses:
[{'ns_1@10.33.148.177',
     [{last_heard,{1345,275024,454579}},
      {active_buckets,["default"]},
      {ready_buckets,["default"]},
      {replication,[{"default",1.0}]},
      {memory,
          [{total,207751368},
           {processes,145034632},
           {processes_used,145009392},
           {system,62716736},
           {atom,828337},
           {atom_used,825958},
           {binary,2757568},
           {code,7928454},
           {ets,40285496}]},
      {system_stats,
          [{cpu_utilization_rate,3.745318352059925},
           {swap_total,2147475456},
           {swap_used,0}]},
      {interesting_stats,
          [{curr_items,7504780},{curr_items_tot,0},{vb_replica_curr_items,0}]},
      {cluster_compatibility_version,1},
      {version,
          [{os_mon,"2.2.6"},
           {mnesia,"4.4.19"},
           {kernel,"2.14.4"},
           {sasl,"2.1.9.4"},
           {ns_server,"1.8.0r-55-g80f24f2-community"},
           {stdlib,"1.17.4"}]},
      {system_arch,"x86_64-unknown-linux-gnu"},
      {wall_clock,139481},
      {memory_data,{25268293632,3730534400,{<5972.13902.24>,68185248}}},
      {disk_data,
          [{"/",74693808,5},
           {"/dev/shm",12338032,0},
           {"/boot",126931,18},
           {"/mysql_data",576727680,1}]},
      {meminfo,
          <<"MemTotal:     24676068 kB\nMemFree:      21027784 kB\nBuffers:        420668 kB\nCached:        1445208 kB\nSwapCached:          0 kB\nActive:        2876172 kB\nInactive:       601852 kB\nHighTotal:           0 kB\nHighFree:            0 kB\nLowTotal:     24676068 kB\nLowFree:      21027784 kB\nSwapTotal:     2097144 kB\nSwapFree:      2097144 kB\nDirty:             284 kB\nWriteback:           0 kB\nAnonPages:     1612292 kB\nMapped:          12896 kB\nSlab:           126372 kB\nPageTables:       7284 kB\nNFS_Unstable:        0 kB\nBounce:              0 kB\nCommitLimit:  14435176 kB\nCommitted_AS:  2196176 kB\nVmallocTotal: 34359738367 kB\nVmallocUsed:    264044 kB\nVmallocChunk: 34359474039 kB\nHugePages_Total:     0\nHugePages_Free:      0\nHugePages_Rsvd:      0\nHugepagesize:     2048 kB\n">>},
      {system_memory_data,
          [{system_total_memory,25268293632},
           {free_swap,2147475456},
           {total_swap,2147475456},
           {cached_memory,1479892992},
           {buffered_memory,430764032},
           {free_memory,21532450816},
           {total_memory,25268293632}]},
      {statistics,
          [{wall_clock,{139471437,1}},
           {context_switches,{42034227,0}},
           {garbage_collection,{6176064,140740155731,0}},
           {io,{{input,24729104491},{output,10084517509}}},
           {reductions,{20079536940,1376291}},
           {run_queue,1},
           {runtime,{4578530,440}}]}]},
 {'ns_1@10.33.148.178',
     [{last_heard,{1345,275023,365160}},
      {active_buckets,["default"]},
      {ready_buckets,["default"]},
      {replication,[{"default",1.0}]},
      {memory,
          [{total,173623536},
           {processes,111887480},
           {processes_used,111857208},
           {system,61736056},
           {atom,817833},
           {atom_used,811002},
           {binary,1901320},
           {code,7720743},
           {ets,40383120}]},
      {system_stats,
          [{cpu_utilization_rate,6.125},
           {swap_total,2147475456},
           {swap_used,0}]},
      {interesting_stats,
          [{curr_items,8760664},{curr_items_tot,0},{vb_replica_curr_items,0}]},
      {cluster_compatibility_version,1},
      {version,
          [{os_mon,"2.2.6"},
           {mnesia,"4.4.19"},
           {kernel,"2.14.4"},
           {sasl,"2.1.9.4"},
           {ns_server,"1.8.0r-55-g80f24f2-community"},
           {stdlib,"1.17.4"}]},
      {system_arch,"x86_64-unknown-linux-gnu"},
      {wall_clock,139566},
      {memory_data,{25268293632,3900379136,{<5973.240.0>,49697096}}},
      {disk_data,
          [{"/",74693808,5},
           {"/dev/shm",12338032,0},
           {"/boot",126931,18},
           {"/mysql_data",576727680,1}]},
      {meminfo,
          <<"MemTotal:     24676068 kB\nMemFree:      20867732 kB\nBuffers:        424956 kB\nCached:        1428228 kB\nSwapCached:          0 kB\nActive:        3037664 kB\nInactive:       601484 kB\nHighTotal:           0 kB\nHighFree:            0 kB\nLowTotal:     24676068 kB\nLowFree:      20867732 kB\nSwapTotal:     2097144 kB\nSwapFree:      2097144 kB\nDirty:             228 kB\nWriteback:          52 kB\nAnonPages:     1786456 kB\nMapped:          13064 kB\nSlab:           125392 kB\nPageTables:       7460 kB\nNFS_Unstable:        0 kB\nBounce:              0 kB\nCommitLimit:  14435176 kB\nCommitted_AS:  2344760 kB\nVmallocTotal: 34359738367 kB\nVmallocUsed:    264044 kB\nVmallocChunk: 34359474039 kB\nHugePages_Total:     0\nHugePages_Free:      0\nHugePages_Rsvd:      0\nHugepagesize:     2048 kB\n">>},
      {system_memory_data,
          [{system_total_memory,25268293632},
           {free_swap,2147475456},
           {total_swap,2147475456},
           {cached_memory,1462505472},
           {buffered_memory,435154944},
           {free_memory,21368557568},
           {total_memory,25268293632}]},
      {statistics,
          [{wall_clock,{139561236,1}},
           {context_switches,{29997437,0}},
           {garbage_collection,{4969532,33358218826,0}},
           {io,{{input,7243274945},{output,9816674252}}},
           {reductions,{9518466746,983012}},
           {run_queue,0},
           {runtime,{2474080,300}}]}]},
 {'ns_1@10.33.148.203',
     [{last_heard,{1345,275023,63442}},
      {active_buckets,["default"]},
      {ready_buckets,["default"]},
      {replication,[{"default",1.0}]},
      {memory,
          [{total,149064712},
           {processes,83022240},
           {processes_used,82991968},
           {system,66042472},
           {atom,818641},
           {atom_used,811485},
           {binary,5407256},
           {code,7728335},
           {ets,41181896}]},
      {system_stats,
          [{cpu_utilization_rate,1.5},{swap_total,2147475456},{swap_used,0}]},
      {interesting_stats,
          [{curr_items,2191683},{curr_items_tot,0},{vb_replica_curr_items,0}]},
      {cluster_compatibility_version,1},
      {version,
          [{os_mon,"2.2.6"},
           {mnesia,"4.4.19"},
           {kernel,"2.14.4"},
           {sasl,"2.1.9.4"},
           {ns_server,"1.8.0r-55-g80f24f2-community"},
           {stdlib,"1.17.4"}]},
      {system_arch,"x86_64-unknown-linux-gnu"},
      {wall_clock,9311086},
      {memory_data,{25268293632,10517430272,{<5974.230.0>,36982608}}},
      {disk_data,
          [{"/",74693808,5},
           {"/dev/shm",12338032,0},
           {"/boot",126931,18},
           {"/mysql_data",576727680,1}]},
      {meminfo,
          <<"MemTotal:     24676068 kB\nMemFree:      14407744 kB\nBuffers:        455464 kB\nCached:        1598800 kB\nSwapCached:          0 kB\nActive:        9400084 kB\nInactive:       640324 kB\nHighTotal:           0 kB\nHighFree:            0 kB\nLowTotal:     24676068 kB\nLowFree:      14407744 kB\nSwapTotal:     2097144 kB\nSwapFree:      2097144 kB\nDirty:             288 kB\nWriteback:          16 kB\nAnonPages:     7986180 kB\nMapped:          14428 kB\nSlab:           171820 kB\nPageTables:      19452 kB\nNFS_Unstable:        0 kB\nBounce:              0 kB\nCommitLimit:  14435176 kB\nCommitted_AS:  8533788 kB\nVmallocTotal: 34359738367 kB\nVmallocUsed:    264044 kB\nVmallocChunk: 34359474039 kB\nHugePages_Total:     0\nHugePages_Free:      0\nHugePages_Rsvd:      0\nHugepagesize:     2048 kB\n">>},
      {system_memory_data,
          [{system_total_memory,25268293632},
           {free_swap,2147475456},
           {total_swap,2147475456},
           {cached_memory,1637171200},
           {buffered_memory,466395136},
           {free_memory,14753529856},
           {total_memory,25268293632}]},
      {statistics,
          [{wall_clock,{9311071159,0}},
           {context_switches,{1895623500,0}},
           {garbage_collection,{338380429,2015679429479,0}},
           {io,{{input,477798174960},{output,652588600399}}},
           {reductions,{619129924190,1000030}},
           {run_queue,0},
           {runtime,{159967490,330}}]}]},
 {'ns_1@10.33.148.204',
     [{last_heard,{1345,275019,762723}},
      {active_buckets,["default"]},
      {ready_buckets,["default"]},
      {replication,[{"default",1.0}]},
      {memory,
          [{total,416862384},
           {processes,81435472},
           {processes_used,81406000},
           {system,335426912},
           {atom,817833},
           {atom_used,811054},
           {binary,3103112},
           {code,7723975},
           {ets,40329472}]},
      {system_stats,[{cpu_utilization_rate,3.0},{swap_total,0},{swap_used,0}]},
      {interesting_stats,
          [{curr_items,7813899},{curr_items_tot,0},{vb_replica_curr_items,0}]},
      {cluster_compatibility_version,1},
      {version,
          [{os_mon,"2.2.6"},
           {mnesia,"4.4.19"},
           {kernel,"2.14.4"},
           {sasl,"2.1.9.4"},
           {ns_server,"1.8.0r-55-g80f24f2-community"},
           {stdlib,"1.17.4"}]},
      {system_arch,"x86_64-unknown-linux-gnu"},
      {wall_clock,138476},
      {memory_data,{25268293632,5637570560,{<5883.236.0>,26295216}}},
      {disk_data,
          [{"/",74693808,7},
           {"/dev/shm",12338032,0},
           {"/boot",126931,18},
           {"/mysql_data",576727680,1}]},
      {meminfo,
          <<"MemTotal:     24676068 kB\nMemFree:      19211572 kB\nBuffers:        453716 kB\nCached:        2974820 kB\nSwapCached:          0 kB\nActive:        3254740 kB\nInactive:      1998684 kB\nHighTotal:           0 kB\nHighFree:            0 kB\nLowTotal:     24676068 kB\nLowFree:      19211572 kB\nSwapTotal:           0 kB\nSwapFree:            0 kB\nDirty:             132 kB\nWriteback:           0 kB\nAnonPages:     1824556 kB\nMapped:          12760 kB\nSlab:           172992 kB\nPageTables:       6812 kB\nNFS_Unstable:        0 kB\nBounce:              0 kB\nCommitLimit:  12338032 kB\nCommitted_AS:  2367284 kB\nVmallocTotal: 34359738367 kB\nVmallocUsed:    263016 kB\nVmallocChunk: 34359475067 kB\nHugePages_Total:     0\nHugePages_Free:      0\nHugePages_Rsvd:      0\nHugepagesize:     2048 kB\n">>},
      {system_memory_data,
          [{system_total_memory,25268293632},
           {free_swap,0},
           {total_swap,0},
           {cached_memory,3046215680},
           {buffered_memory,464605184},
           {free_memory,19672649728},
           {total_memory,25268293632}]},
      {statistics,
          [{wall_clock,{138466668,0}},
           {context_switches,{29790194,0}},
           {garbage_collection,{4838566,33136589139,0}},
           {io,{{input,7175383106},{output,9725418306}}},
           {reductions,{9497703106,1029425}},
           {run_queue,0},
           {runtime,{2458070,280}}]}]}]
 
INFO REPORT  <5883.312.0>                                   2012-08-18 01:30:46
===============================================================================
 
ns_1@10.33.148.204:<5883.312.0>:stats_collector:83: Stats for bucket "default":
accepting_conns               1
auth_cmds                     218
auth_errors                   0
bucket_active_conns           1
bucket_conns                  55
bytes                         1033908004
bytes_read                    23345148221
bytes_written                 43775384583
cas_badval                    0
cas_hits                      0
cas_misses                    0
cmd_flush                     0
cmd_get                       254181520
cmd_set                       75827165
conn_yields                   0
connection_structures         75
curr_connections              65
curr_items                    7813020
daemon_connections            10
decr_hits                     3046115
decr_misses                   8255
delete_hits                   5733
delete_misses                 33353
engine_maxbytes               6291456000
evictions                     0
get_hits                      216147576
get_misses                    38033944
incr_hits                     46348
incr_misses                   50276
libevent                      2.0.11-stable
limit_maxbytes                67108864
listen_disabled_num           0
pid                           14642
pointer_size                  64
reclaimed                     5007052
rejected_conns                0
rusage_system                 4078.130030
rusage_user                   4174.720346
threads                       4
time                          1345275045
total_connections             229
total_items                   75827187
uptime                        138503
version                       UNKNOWN

Kernel log

moxi[18635] general protection rip:42d8c4 rsp:4270c130 error:0
moxi[18992]: segfault at 0000000000000000 rip 0000000000000000 rsp 0000000042c57290 error 14
moxi[20179]: segfault at 0000000000000004 rip 000000000042d8c4 rsp 0000000042c6a130 error 4
moxi[20463]: segfault at 0000000000016420 rip 0000000000016420 rsp 00000000415f0290 error 14
moxi[21312]: segfault at 0000000000000000 rip 0000000000000000 rsp 0000000043510290 error 14
moxi[23208]: segfault at 0000000000000000 rip 0000000000000000 rsp 0000000041c0d290 error 14
moxi[24253]: segfault at 0000000000000800 rip 000000000042d8c4 rsp 0000000043a03130 error 4
moxi[24547]: segfault at 0000400000000000 rip 000000000042d8c4 rsp 000000004425c130 error 4
moxi[25062]: segfault at 0000000040000000 rip 000000000042d8c4 rsp 0000000044442130 error 4
moxi[25470]: segfault at 0000000800000000 rip 000000000042d8c4 rsp 0000000042398130 error 4
moxi[25865] general protection rip:42d8c4 rsp:41f60130 error:0
moxi[26271]: segfault at 0000000000000080 rip 000000000042d8c4 rsp 0000000042041130 error 4
moxi[26806]: segfault at 0000000000001000 rip 000000000042d8c4 rsp 000000004336f130 error 4
moxi[27173] general protection rip:42d8c4 rsp:43a3e130 error:0
moxi[27529]: segfault at 0000000000000010 rip 000000000042d8c4 rsp 0000000040e84130 error 4
moxi[28244] general protection rip:42d8c4 rsp:42e90130 error:0
moxi[29269]: segfault at 0000000000000000 rip 0000000000000000 rsp 0000000044363290 error 14
moxi[32107]: segfault at 0000000000000002 rip 000000000042d8c4 rsp 0000000040dd8130 error 4
moxi[32539]: segfault at 0000000000020000 rip 0000000000020000 rsp 00000000429f6290 error 14
moxi[671]: segfault at 0000000000800000 rip 000000000042d8c4 rsp 000000004290d130 error 4
moxi[1322] general protection rip:42d8c4 rsp:43b54130 error:0
moxi[1744]: segfault at 0000008000000000 rip 000000000042d8c4 rsp 000000004139b130 error 4
moxi[2503] general protection rip:42d8c4 rsp:43a0c130 error:0
moxi[3121]: segfault at 0000000000000080 rip 000000000042d8c4 rsp 0000000043aa6130 error 4
moxi[3995]: segfault at 0000000080000000 rip 000000000042d8c4 rsp 0000000042175130 error 4
moxi[4793]: segfault at 00000000427641c0 rip 000000000042da4d rsp 0000000042762130 error 4
moxi[5393]: segfault at 0000000000000000 rip 0000000000000000 rsp 00000000445c8290 error 14
moxi[6383]: segfault at 0000000000000000 rip 0000000000000000 rsp 0000000041bef290 error 14
moxi[8152]: segfault at 0000000000000000 rip 0000000000000000 rsp 0000000042cac290 error 14
moxi[8536] general protection rip:42d8c4 rsp:4244e130 error:0
moxi[9608]: segfault at 0000000000000008 rip 000000000042d8c4 rsp 0000000042df6130 error 4
moxi[10050]: segfault at 0000000800000000 rip 000000000042d8c4 rsp 0000000041c97130 error 4
moxi[10397]: segfault at 0000000000000040 rip 000000000042d8c4 rsp 0000000043659130 error 4
moxi[11391]: segfault at 0000000000000000 rip 0000000000000000 rsp 0000000042b5c290 error 14
moxi[17182] general protection rip:42d8c4 rsp:42df2130 error:0
moxi[17659]: segfault at 0000000020000000 rip 0000000020000000 rsp 0000000041de4290 error 14
moxi[19911]: segfault at 0000000000004000 rip 0000000000004000 rsp 000000004318a290 error 14
moxi[21318]: segfault at 0000000000000000 rip 0000000000000000 rsp 0000000043d3a290 error 14
moxi[22075]: segfault at 0000000000010000 rip 000000000042d8c4 rsp 000000004280a130 error 4
moxi[22740]: segfault at 0000000000000210 rip 0000000000000210 rsp 0000000041fd7290 error 14
moxi[24921]: segfault at 0000010000000000 rip 000000000042d8c4 rsp 0000000043c37130 error 4
moxi[25582]: segfault at 0000000000000004 rip 000000000042d8c4 rsp 0000000042100130 error 4
moxi[26340]: segfault at 0000000000000000 rip 0000000000000000 rsp 000000004207c290 error 14
moxi[31739]: segfault at 0000000000000000 rip 0000000000402005 rsp 0000000042d44290 error 6
moxi[2781]: segfault at 0000010000000000 rip 0000010000000000 rsp 0000000043815290 error 14
moxi[5016]: segfault at 0000000046e821c0 rip 000000000042da4d rsp 0000000042e82130 error 4
moxi[9110]: segfault at 0000000000000000 rip 0000000000000000 rsp 00000000427e0290 error 14
moxi[11390]: segfault at 0000000000800000 rip 000000000042d8c4 rsp 0000000042b1b130 error 4
moxi[13176]: segfault at 0000000000000000 rip 0000000000000000 rsp 0000000042454290 error 14
moxi[16494]: segfault at 0000000000000800 rip 000000000042d8c4 rsp 0000000043643130 error 4
moxi[17617]: segfault at 0000000002000000 rip 000000000042d8c4 rsp 0000000043050130 error 4
moxi[22103]: segfault at 0000040000000000 rip 000000000042d8c4 rsp 00000000427d2130 error 4
moxi[24301]: segfault at 0000000000000000 rip 0000000000000000 rsp 00000000429c3290 error 14
moxi[28539]: segfault at 0000000000020000 rip 000000000042d8c4 rsp 00000000412c7130 error 4

*EDIT*
Some additional information:

when MOXI segfaults and "restarts" our PHP application continuously times out when connecting to memcache and is unable to connect again until we do a full restart of couchbase-server on that node.

memtest on all nodes came back clean.

strace on a live moxi process shows nothing other than MOXI is getting killed by SIGSEGV

We have had both clusters running for a couple of months, the memcache cluster had this issue to begin with and somehow fixed itself and has been fine up until a few days ago when the problem started up again randomly. Nothing afaik has changed with the app nor any of the other servers that rely on memcache.

Top