[MB-6638] beam.smp ( erl_bestfit_alloc.c:435 ) segmentation fault when rebalancing a new node to a cluster with 2 buckets and 1 design doc per bucket Created: 12/Sep/12  Updated: 10/Jan/13  Resolved: 24/Oct/12

Status: Closed
Project: Couchbase Server
Component/s: 3rd-party, ns_server, view-engine
Affects Version/s: 2.0-beta-2
Fix Version/s: 2.0-beta-2
Security Level: Public

Type: Bug Priority: Blocker
Reporter: Thuan Nguyen Assignee: damien
Resolution: Fixed Votes: 0
Labels: system-test
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment: centos 6.2 build 2.0.0-1708

Attachments: GZip Archive memcached.log.6.txt.gz     GZip Archive memcached.log.7.txt.gz    

 Description   
Cluster information:
- Load 9 million items to both buckets
- Cluster has 2 buckets, default (3GB) and saslbucket (3GB)
- Each bucket has one doc and 2 views for each doc (default d1 and saslbucket d11)

* Create cluster with 10 nodes installed couchbase server 2.0.0-1708

10.3.121.13
10.3.121.14
10.3.121.15
10.3.121.16
10.3.121.17
10.3.121.20
10.3.121.22
10.3.121.24
10.3.121.25
10.3.121.23
* Data path /data
* View path /data

Doing rebalanc in, out, failover.
Last failed was rebalance add 2 nodes, 23 and 24 into cluster.
During rebalance, node 23 suddenly down. Rebalance failed as expected.
Then later, memcached on node 13 was killed and restart.

[ns_server:debug,2012-09-12T17:06:05.705,ns_1@10.3.121.13:<0.28110.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_config_events,<0.28109.33>} exited with reason {timeout,
                                                                                   {gen_server,
                                                                                    call,
                                                                                    [ns_node_disco,
                                                                                     nodes_wanted]}}
[ns_server:debug,2012-09-12T17:06:05.712,ns_1@10.3.121.13:<0.28166.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_config_events,<0.28165.33>} exited with reason shutdown
[ns_server:error,2012-09-12T17:06:05.718,ns_1@10.3.121.13:<0.29417.33>:ns_memcached:verify_report_long_call:274]call {stats,<<>>} took too long: 5881903 us
[ns_server:debug,2012-09-12T17:06:05.720,ns_1@10.3.121.13:<0.28132.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_node_disco_events,<0.28130.33>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:05.732,ns_1@10.3.121.13:<0.28110.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_config_events,<0.28109.33>} event handler: ok
[ns_server:debug,2012-09-12T17:06:05.733,ns_1@10.3.121.13:<0.28166.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_config_events,<0.28165.33>} event handler: ok
[ns_server:debug,2012-09-12T17:06:05.739,ns_1@10.3.121.13:<0.5306.465>:compaction_daemon:do_chain_compactors:500]Got exit signal from parent: {'EXIT',<0.4974.465>,shutdown}
[ns_server:debug,2012-09-12T17:06:05.740,ns_1@10.3.121.13:<0.28132.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_node_disco_events,<0.28130.33>} event handler: ok
[ns_server:debug,2012-09-12T17:06:05.740,ns_1@10.3.121.13:ns_node_disco:ns_node_disco:init:103]Initting ns_node_disco with ['ns_1@10.3.121.15','ns_1@10.3.121.16',
                             'ns_1@10.3.121.17','ns_1@10.3.121.22',
                             'ns_1@10.3.121.20','ns_1@10.3.121.14',
                             'ns_1@10.3.121.25','ns_1@10.3.121.24']
[ns_server:debug,2012-09-12T17:06:05.741,ns_1@10.3.121.13:<0.28129.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_config_events,<0.28128.33>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:05.743,ns_1@10.3.121.13:<0.19269.466>:janitor_agent:new_style_query_vbucket_states_loop:111]Exception from query_vbucket_states of "saslbucket":'ns_1@10.3.121.23'
{'EXIT',{{nodedown,'ns_1@10.3.121.23'},
         {gen_server,call,
                     [{'janitor_agent-saslbucket','ns_1@10.3.121.23'},
                      query_vbucket_states,infinity]}}}
[ns_server:debug,2012-09-12T17:06:05.745,ns_1@10.3.121.13:<0.5309.465>:compaction_daemon:do_chain_compactors:500]Got exit signal from parent: {'EXIT',<0.5306.465>,shutdown}
[ns_server:debug,2012-09-12T17:06:05.746,ns_1@10.3.121.13:<0.28126.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_config_events,<0.28125.33>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:05.761,ns_1@10.3.121.13:<0.19269.466>:janitor_agent:new_style_query_vbucket_states_loop_next_step:116]Waiting for "saslbucket" on 'ns_1@10.3.121.23'
[ns_server:debug,2012-09-12T17:06:05.897,ns_1@10.3.121.13:<0.28129.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_config_events,<0.28128.33>} event handler: ok
[ns_server:debug,2012-09-12T17:06:05.898,ns_1@10.3.121.13:<0.14047.466>:compaction_daemon:do_spawn_view_index_compactor:742]Got exit signal from parent: {'EXIT',<0.5309.465>,shutdown}
[ns_server:debug,2012-09-12T17:06:05.903,ns_1@10.3.121.13:ns_cookie_manager:ns_cookie_manager:do_cookie_sync:115]ns_cookie_manager do_cookie_sync
[ns_server:debug,2012-09-12T17:06:05.903,ns_1@10.3.121.13:<0.28126.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_config_events,<0.28125.33>} event handler: ok
[couchdb:info,2012-09-12T17:06:05.906,ns_1@10.3.121.13:<0.5719.436>:couch_log:info:39]Set view `default`, replica group `_design/d1`, canceling compaction (pid <0.18999.466>)
[ns_server:debug,2012-09-12T17:06:06.000,ns_1@10.3.121.13:ns_cookie_manager:ns_cookie_manager:do_cookie_save:152]saving cookie to "/opt/couchbase/var/lib/couchbase/couchbase-server.cookie"
[ns_server:debug,2012-09-12T17:06:06.002,ns_1@10.3.121.13:<0.28122.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_stats_event,<0.28121.33>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:06.009,ns_1@10.3.121.13:<0.28122.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_stats_event,<0.28121.33>} event handler: ok
[ns_server:debug,2012-09-12T17:06:06.040,ns_1@10.3.121.13:<0.28120.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_tick_event,<0.28119.33>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:06.044,ns_1@10.3.121.13:<0.28648.34>:single_bucket_sup:top_loop:28]Delegating exit {'EXIT',<0.28117.33>,shutdown} to child supervisor: <0.28649.34>

[ns_server:debug,2012-09-12T17:06:06.047,ns_1@10.3.121.13:ns_cookie_manager:ns_cookie_manager:do_cookie_save:154]attempted to save cookie to "/opt/couchbase/var/lib/couchbase/couchbase-server.cookie": ok
[ns_server:debug,2012-09-12T17:06:06.074,ns_1@10.3.121.13:<0.28120.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_tick_event,<0.28119.33>} event handler: ok
[ns_server:debug,2012-09-12T17:06:06.074,ns_1@10.3.121.13:<0.19276.466>:ns_node_disco:do_nodes_wanted_updated_fun:202]ns_node_disco: nodes_wanted updated: ['ns_1@10.3.121.13','ns_1@10.3.121.14',
                                      'ns_1@10.3.121.15','ns_1@10.3.121.16',
                                      'ns_1@10.3.121.17','ns_1@10.3.121.20',
                                      'ns_1@10.3.121.22','ns_1@10.3.121.23',
                                      'ns_1@10.3.121.24','ns_1@10.3.121.25'], with cookie: oetfisqvqrrxvaat
[ns_server:debug,2012-09-12T17:06:06.079,ns_1@10.3.121.13:<0.28689.34>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_stats_event,<0.28688.34>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:06.080,ns_1@10.3.121.13:<0.28687.34>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_tick_event,<0.28686.34>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:06.083,ns_1@10.3.121.13:<0.28689.34>:ns_pubsub:do_subscribe_link:149]Deleting {ns_stats_event,<0.28688.34>} event handler: ok
[ns_server:debug,2012-09-12T17:06:06.084,ns_1@10.3.121.13:<0.28687.34>:ns_pubsub:do_subscribe_link:149]Deleting {ns_tick_event,<0.28686.34>} event handler: ok
[stats:warn,2012-09-12T17:06:06.087,ns_1@10.3.121.13:<0.29425.33>:stats_collector:latest_tick:201]Dropped 7 ticks
[rebalance:info,2012-09-12T17:06:06.091,ns_1@10.3.121.13:<0.25334.457>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!

[rebalance:info,2012-09-12T17:06:06.120,ns_1@10.3.121.13:<0.23919.457>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!

[ns_server:debug,2012-09-12T17:06:06.120,ns_1@10.3.121.13:<0.1936.453>:ns_pubsub:do_subscribe_link:149]Deleting {master_activity_events_ingress,<0.28014.33>} event handler: {error,
                                                                       module_not_found}
[rebalance:info,2012-09-12T17:06:06.128,ns_1@10.3.121.13:<0.23192.457>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!

[rebalance:info,2012-09-12T17:06:06.147,ns_1@10.3.121.13:<0.22947.457>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!

[rebalance:info,2012-09-12T17:06:06.201,ns_1@10.3.121.13:<0.21996.457>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!

[rebalance:info,2012-09-12T17:06:06.218,ns_1@10.3.121.13:<0.11912.457>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!

[rebalance:info,2012-09-12T17:06:06.223,ns_1@10.3.121.13:<0.10815.457>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!

[ns_server:debug,2012-09-12T17:06:06.228,ns_1@10.3.121.13:<0.28050.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {buckets_events,<0.28049.33>} exited with reason {shutdown,
                                                                                 {gen_server,
                                                                                  call,
                                                                                  ['ns_vbm_new_sup-default',
                                                                                   which_children,
                                                                                   infinity]}}
[ns_server:debug,2012-09-12T17:06:06.239,ns_1@10.3.121.13:<0.28050.33>:ns_pubsub:do_subscribe_link:149]Deleting {buckets_events,<0.28049.33>} event handler: ok
[user:info,2012-09-12T17:06:06.245,ns_1@10.3.121.13:'ns_memcached-default':ns_memcached:terminate:625]Shutting down bucket "default" on 'ns_1@10.3.121.13' for server shutdown
[ns_server:info,2012-09-12T17:06:06.261,ns_1@10.3.121.13:'ns_memcached-default':ns_memcached:terminate:636]This bucket shutdown is not due to bucket deletion. Doing nothing
[ns_server:debug,2012-09-12T17:06:06.276,ns_1@10.3.121.13:<0.28651.34>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_config_events,<0.28650.34>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:06.276,ns_1@10.3.121.13:<0.28663.34>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {mc_couch_events,<0.28650.34>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:06.276,ns_1@10.3.121.13:<0.28662.34>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_node_disco_events,<0.28650.34>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:06.276,ns_1@10.3.121.13:<0.28648.34>:single_bucket_sup:top_loop:24]per-bucket supervisor for "default" died with reason shutdown

[ns_server:debug,2012-09-12T17:06:06.279,ns_1@10.3.121.13:<0.28651.34>:ns_pubsub:do_subscribe_link:149]Deleting {ns_config_events,<0.28650.34>} event handler: ok
[ns_server:debug,2012-09-12T17:06:06.281,ns_1@10.3.121.13:<0.28663.34>:ns_pubsub:do_subscribe_link:149]Deleting {mc_couch_events,<0.28650.34>} event handler: ok
[ns_server:debug,2012-09-12T17:06:06.283,ns_1@10.3.121.13:<0.28662.34>:ns_pubsub:do_subscribe_link:149]Deleting {ns_node_disco_events,<0.28650.34>} event handler: ok
[ns_server:debug,2012-09-12T17:06:06.283,ns_1@10.3.121.13:<0.29393.33>:single_bucket_sup:top_loop:28]Delegating exit {'EXIT',<0.28117.33>,shutdown} to child supervisor: <0.29394.33>

[ns_server:debug,2012-09-12T17:06:06.299,ns_1@10.3.121.13:<0.29428.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_stats_event,<0.29427.33>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:06.300,ns_1@10.3.121.13:<0.29426.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_tick_event,<0.29425.33>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:06.305,ns_1@10.3.121.13:<0.29428.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_stats_event,<0.29427.33>} event handler: ok
[ns_server:debug,2012-09-12T17:06:06.307,ns_1@10.3.121.13:<0.29426.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_tick_event,<0.29425.33>} event handler: ok
[rebalance:info,2012-09-12T17:06:06.310,ns_1@10.3.121.13:<0.11978.466>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!

[ns_server:info,2012-09-12T17:06:06.323,ns_1@10.3.121.13:ns_port_memcached:ns_port_server:log:169]memcached<0.28094.33>: Thu Sep 13 00:06:06.117323 3: TAP (Consumer) eq_tapq:anon_509 - disconnected
memcached<0.28094.33>: Thu Sep 13 00:06:06.123981 3: TAP (Consumer) eq_tapq:anon_1652 - disconnected
memcached<0.28094.33>: Thu Sep 13 00:06:06.144392 3: TAP (Consumer) eq_tapq:anon_480 - disconnected
memcached<0.28094.33>: Thu Sep 13 00:06:06.151955 3: TAP (Consumer) eq_tapq:anon_488 - disconnected
memcached<0.28094.33>: Thu Sep 13 00:06:06.203038 3: TAP (Consumer) eq_tapq:anon_504 - disconnected
memcached<0.28094.33>: Thu Sep 13 00:06:06.221683 3: TAP (Consumer) eq_tapq:anon_719 - disconnected
memcached<0.28094.33>: Thu Sep 13 00:06:06.227307 3: TAP (Consumer) eq_tapq:anon_499 - disconnected


memcached log from node 13 at the time it was killed

Thu Sep 13 00:06:07.606021 3: Schedule cleanup of "eq_tapq:anon_20"
Thu Sep 13 00:06:07.606058 3: Schedule cleanup of "eq_tapq:anon_34"
Thu Sep 13 00:06:07.606083 3: Schedule cleanup of "eq_tapq:anon_36"
Thu Sep 13 00:06:07.606115 3: Schedule cleanup of "eq_tapq:anon_40"
Thu Sep 13 00:06:07.606142 3: Schedule cleanup of "eq_tapq:anon_730"
Thu Sep 13 00:06:07.606165 3: Schedule cleanup of "eq_tapq:anon_2425"
Thu Sep 13 00:06:21.096648 3: Had to wait 13 s for shutdown
Thu Sep 13 00:06:31.465158 3: Shutting down tap connections!
Thu Sep 13 00:06:31.465199 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.15"
Thu Sep 13 00:06:31.465278 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.16"
Thu Sep 13 00:06:31.465357 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.22"
Thu Sep 13 00:06:31.465405 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.17"
Thu Sep 13 00:06:31.465414 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.15 - Clear the tap queues by force
Thu Sep 13 00:06:31.465441 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.20"
Thu Sep 13 00:06:31.465471 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.14"
Thu Sep 13 00:06:31.465492 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.25"
Thu Sep 13 00:06:31.465517 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.24"
Thu Sep 13 00:06:31.465604 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.16 - Clear the tap queues by force
Thu Sep 13 00:06:31.467071 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.22 - Clear the tap queues by force
Thu Sep 13 00:06:31.467363 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.20 - Clear the tap queues by force
Thu Sep 13 00:06:31.467606 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.24 - Clear the tap queues by force
Thu Sep 13 00:06:31.467710 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.25 - Clear the tap queues by force
Thu Sep 13 00:06:31.467917 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.17 - Clear the tap queues by force
Thu Sep 13 00:06:31.468102 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.14 - Clear the tap queues by force


Link to diags of all nodes https://s3.amazonaws.com/packages.couchbase/diag-logs/orange/201209/10nodes-1708-memcached-killed-after-reb-failed-20120912.tgz

Link to atop of node 13 https://s3.amazonaws.com/packages.couchbase/atop-files/orange/201209/atop-node13-20120912


 Comments   
Comment by Chiyoung Seo [ 13/Sep/12 ]
Node 10.3.121.23, which was newly added, became unreachable from 16:57:37 to 17:38:36. I didn't see any logs in the diags on this node during this period:

[views:info,2012-09-12T16:57:37.570,ns_1@10.3.121.23:'capi_set_view_manager-saslbucket':capi_set_view_manager:apply_index_states:464]
Calling couch_set_view:add_replica_partitions([<<"saslbucket">>,
                                               <<"_design/d11">>,
                                               [18,19,32,33,34,35,44,45,46,
                                                105,106,107,114,120,152,153,
                                                158,159,160,181,182,198,199,
                                                216,217,224,225,226,227,228,
                                                229,230,242,245,253,276,277,
                                                278,279,280,281,282,283,300,
                                                301,302,303,324,356,357,358,
                                                359,360,361,381,382,383,384,
                                                385,443,444,445,451,452,453,
                                                454,455,463,464,465,466,467,
                                                468,481,482,483,506,507,508,
                                                520,521,522,523,524,527,552,
                                                553,554,565,574,575,576,577,
                                                578,579,591,592,594,595,596,
                                                597,602,603,604,619,620,621,
                                                627,628,629,632,633,634,635,
                                                636,637,656,657,729,730,731,
                                                732,749,750,761,808,809,810,
                                                811,812,836,842,848,849,850,
                                                851,873,874,875,907]])
[ns_server:info,2012-09-12T17:38:36.992,nonode@nohost:<0.60.0>:ns_server:init_logging:198]Started & configured logging
[ns_server:info,2012-09-12T17:38:37.003,nonode@nohost:<0.60.0>:ns_server:log_pending:30]Static config terms:


 
Comment by Chiyoung Seo [ 13/Sep/12 ]
Alk, please see my comment and advise me why this can happen.
Comment by Karan Kumar (Inactive) [ 13/Sep/12 ]
I was able to figure out some more stuff.
The beam.smp segfaulted around this time. Which caused all sorts of query/rebalance failures.

Sep 12 16:57:37 orange-3006 kernel: beam.smp[25844]: segfault at 10000000f ip 000000000043ef50 sp 00007f7139064d48 error 6 in beam.smp[400000+1d6000]
Comment by Karan Kumar (Inactive) [ 13/Sep/12 ]
Unfortunately, I dont see any core file on the system.

Even though, cores are enabled on the machine.
Comment by Aleksey Kondratenko [ 13/Sep/12 ]
We'll need core dump here.
Comment by Karan Kumar (Inactive) [ 13/Sep/12 ]
We are hitting this again.

But for some reason the cores are not getting generated.

beam.smp[16816]: segfault at 0 ip 00000030af08847b sp 00007f011e38fdd8 error 4 in libc-2.12.so[30af000000+186000]
Comment by Karan Kumar (Inactive) [ 13/Sep/12 ]
Increasing this to blocker
Hitting this more often than not
Comment by Karan Kumar (Inactive) [ 13/Sep/12 ]
ulimit for all users have now been set to unlimited. and kernel.core_pattern = /data/core.%e.%p on all the nodes.
Comment by Karan Kumar (Inactive) [ 14/Sep/12 ]
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-50.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp...done.
[New Thread 28408]
[New Thread 28409]
[New Thread 28410]
[New Thread 28412]
[New Thread 28414]
[New Thread 28420]
[New Thread 28979]
[New Thread 28398]
[New Thread 28400]
[New Thread 28425]
[New Thread 28413]
[New Thread 28416]
[New Thread 28417]
[New Thread 28433]
[New Thread 28589]
[New Thread 29574]
[New Thread 28401]
[New Thread 28405]
[New Thread 28406]
[New Thread 28424]
[New Thread 28415]
[New Thread 28419]
[New Thread 29407]
[New Thread 29576]
[New Thread 28411]
[New Thread 28418]
[New Thread 28421]
[New Thread 28403]
[New Thread 28407]
[New Thread 28423]
[New Thread 28422]
Reading symbols from /lib64/libutil.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libutil.so.1
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libncurses.so.5...(no debugging symbols found)...done.
Loaded symbols for /lib64/libncurses.so.5
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libtinfo.so.5...(no debugging symbols found)...done.
Loaded symbols for /lib64/libtinfo.so.5
Reading symbols from /opt/couchbase/lib/erlang/lib/crypto-2.0.4/priv/lib/crypto.so...done.
Loaded symbols for /opt/couchbase/lib/erlang/lib/crypto-2.0.4/priv/lib/crypto.so
Reading symbols from /usr/lib64/libcrypto.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.6
Reading symbols from /usr/lib64/libssl.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl.so.6
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /opt/couchbase/lib/libv8.so...done.
Loaded symbols for /opt/couchbase/lib/libv8.so
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libfreebl3.so...(no debugging symbols found)...done.
Loaded symbols for /lib64/libfreebl3.so
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/ejson-0.1.0/priv/ejson.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/ejson-0.1.0/priv/ejson.so
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/snappy-1.0.4/priv/snappy_nif.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/snappy-1.0.4/priv/snappy_nif.so
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/mapreduce-1.0/priv/mapreduce_nif.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/mapreduce-1.0/priv/mapreduce_nif.so
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_icu_driver.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_icu_driver.so
Reading symbols from /opt/couchbase/lib/libicuuc.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicudata.so.44
Reading symbols from /opt/couchbase/lib/libicui18n.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicui18n.so.44
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_ejson_compare.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_ejson_compare.so
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/couch_view_parser-1.0/priv/couch_view_parser_nif.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/couch_view_parser-1.0/priv/couch_view_parser_nif.so
Core was generated by `/opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp -A 16 -sbt u -P 327680 -K tru'.
Program terminated with signal 11, Segmentation fault.
#0 0x000000000043ec6b in tree_delete (allctr=<value optimized out>, del=0x12c6a80, flags=<value optimized out>) at beam/erl_bestfit_alloc.c:435
435 beam/erl_bestfit_alloc.c: No such file or directory.
in beam/erl_bestfit_alloc.c
Missing separate debuginfos, use: debuginfo-install couchbase-server-2.0.0-1717.x86_64
(gdb) t a a bt

Thread 31 (Thread 0x7f4afea01700 (LWP 28422)):
#0 0x000000397260804d in pthread_join () from /lib64/libpthread.so.0
#1 0x00007f4af6371887 in v8::internal::ContextSwitcher::StopPreemption () at src/v8threads.cc:445
#2 0x00007f4af424b7a3 in destroyContext (ctx=0x7f4aac7b4f18) at mapreduce.cc:354
#3 0x00000000005138fb in nif_resource_dtor (bin=<value optimized out>) at beam/erl_nif.c:1217
#4 0x000000000048904b in erts_bin_free (offheap=<value optimized out>) at beam/erl_binary.h:303
#5 erts_cleanup_offheap (offheap=<value optimized out>) at beam/erl_message.c:174
#6 0x0000000000498a81 in delete_process (p=0x7f4aa286d568, pix_lock=<value optimized out>) at beam/erl_process.c:6621
#7 continue_exit_process (p=0x7f4aa286d568, pix_lock=<value optimized out>) at beam/erl_process.c:7663
#8 0x000000000051db94 in terminate_proc (c_p=0x7f4aa286d568, pc=<value optimized out>, reg=<value optimized out>, bf=<value optimized out>) at beam/beam_emu.c:5633
#9 handle_error (c_p=0x7f4aa286d568, pc=<value optimized out>, reg=<value optimized out>, bf=<value optimized out>) at beam/beam_emu.c:5508
#10 0x0000000000520cdf in process_main () at beam/beam_emu.c:3273
#11 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b000eafc0) at beam/erl_process.c:3782
#12 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200a0) at pthread/ethread.c:106
#13 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#14 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 30 (Thread 0x7f4afe000700 (LWP 28423)):
#0 0x0000000000522f94 in process_main () at x86_64-unknown-linux-gnu/opt/smp/beam_hot.h:919
#1 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b000f9080) at beam/erl_process.c:3782
#2 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200a0) at pthread/ethread.c:106
#3 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#4 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 29 (Thread 0x7f4afec23700 (LWP 28407)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8138) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8138) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8138) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 28 (Thread 0x7f4affe69700 (LWP 28403)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x0000000000462ad9 in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:697
#3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:472
#4 sys_msg_dispatcher_func (unused=<value optimized out>) at beam/erl_trace.c:3238
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200d0) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 27 (Thread 0x7f4afea47700 (LWP 28421)):
#0 0x000000397260f03d in waitpid () from /lib64/libpthread.so.0
#1 0x000000000053906f in child_waiter (unused=<value optimized out>) at sys/unix/sys.c:2821
#2 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620080) at pthread/ethread.c:106
#3 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#4 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x7f4afeaad700 (LWP 28418)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8660) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8660) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8660) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 25 (Thread 0x7f4afeb9b700 (LWP 28411)):
---Type <return> to continue, or q <return> to quit---
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8318) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8318) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8318) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x7f4aaed44700 (LWP 29576)):
#0 0x0000003971eab15d in nanosleep () from /lib64/libc.so.6
#1 0x0000003971edf124 in usleep () from /lib64/libc.so.6
#2 0x00007f4af6371848 in v8::internal::ContextSwitcher::Run (this=0x7f4af814d8d0) at src/v8threads.cc:457
#3 0x00007f4af62f1580 in v8::internal::ThreadEntry (arg=0x7f4af814d8d0) at src/platform-linux.cc:745
#4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x7f4a94844700 (LWP 29407)):
#0 0x0000003971eab15d in nanosleep () from /lib64/libc.so.6
#1 0x0000003971edf124 in usleep () from /lib64/libc.so.6
#2 0x00007f4af6371848 in v8::internal::ContextSwitcher::Run (this=0x7f4ab41b09d0) at src/v8threads.cc:457
#3 0x00007f4af62f1580 in v8::internal::ThreadEntry (arg=0x7f4ab41b09d0) at src/platform-linux.cc:745
#4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7f4afea8b700 (LWP 28419)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d86d8) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d86d8) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d86d8) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7f4afeb13700 (LWP 28415)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d84f8) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d84f8) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d84f8) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7f4afd5ff700 (LWP 28424)):
#0 do_erts_alcu_free (type=<value optimized out>, unused=<value optimized out>, p=0x7f4aaf0b8608) at beam/erl_alloc_util.c:2894
#1 erts_alcu_free_thr_pref (type=<value optimized out>, unused=<value optimized out>, p=0x7f4aaf0b8608) at beam/erl_alloc_util.c:2949
#2 0x0000000000489022 in erts_free (offheap=<value optimized out>) at beam/erl_alloc.h:226
#3 erts_bin_free (offheap=<value optimized out>) at beam/erl_binary.h:307
#4 erts_cleanup_offheap (offheap=<value optimized out>) at beam/erl_message.c:174
#5 0x0000000000498a81 in delete_process (p=0x7f4aac311310, pix_lock=<value optimized out>) at beam/erl_process.c:6621
#6 continue_exit_process (p=0x7f4aac311310, pix_lock=<value optimized out>) at beam/erl_process.c:7663
#7 0x000000000051db94 in terminate_proc (c_p=0x7f4aac311310, pc=<value optimized out>, reg=<value optimized out>, bf=<value optimized out>) at beam/beam_emu.c:5633
#8 handle_error (c_p=0x7f4aac311310, pc=<value optimized out>, reg=<value optimized out>, bf=<value optimized out>) at beam/beam_emu.c:5508
#9 0x0000000000520cdf in process_main () at beam/beam_emu.c:3273
#10 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b00107140) at beam/erl_process.c:3782
#11 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200a0) at pthread/ethread.c:106
#12 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#13 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7f4afec45700 (LWP 28406)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d80c0) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d80c0) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d80c0) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7f4afec67700 (LWP 28405)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8048) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8048) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8048) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7f4b0017c700 (LWP 28401)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x000000000046a1cf in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:697
#3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:472
#4 emergency_watchdog (unused=<value optimized out>) at beam/utils.c:3945
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200d0) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f4a93e43700 (LWP 29574)):
#0 0x0000003971eab15d in nanosleep () from /lib64/libc.so.6
#1 0x0000003971edf124 in usleep () from /lib64/libc.so.6
#2 0x00007f4af6371848 in v8::internal::ContextSwitcher::Run (this=0x7f4aa8091b00) at src/v8threads.cc:457
#3 0x00007f4af62f1580 in v8::internal::ThreadEntry (arg=0x7f4aa8091b00) at src/platform-linux.cc:745
#4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7f4aaed55700 (LWP 28589)):
#0 0x000000397260d6c0 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f4af62f118d in v8::internal::LinuxSemaphore::Wait (this=0x7f4aa8009b10) at src/platform-linux.cc:869
#2 0x00007f4af630845c in v8::internal::RuntimeProfiler::WaitForSomeIsolateToEnterJS () at src/runtime-profiler.cc:386
#3 0x00007f4af62f2136 in v8::internal::SignalSender::Run (this=0x7f4aa8009470) at src/platform-linux.cc:1126
#4 0x00007f4af62f1580 in v8::internal::ThreadEntry (arg=0x7f4aa8009470) at src/platform-linux.cc:745
#5 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#6 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7f4aaffff700 (LWP 28433)):
#0 0x0000003971eab15d in nanosleep () from /lib64/libc.so.6
#1 0x0000003971edf124 in usleep () from /lib64/libc.so.6
#2 0x00007f4af4249397 in terminatorLoop (args=<value optimized out>) at mapreduce_nif.cc:445
#3 0x0000000000584d89 in thr_wrapper (vtwd=0x7f4afea00ae0) at pthread/ethread.c:106
#4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7f4afeacf700 (LWP 28417)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d85e8) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d85e8) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d85e8) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

---Type <return> to continue, or q <return> to quit---
Thread 12 (Thread 0x7f4afeaf1700 (LWP 28416)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8570) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8570) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8570) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f4afeb57700 (LWP 28413)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8408) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8408) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8408) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f4afcbfe700 (LWP 28425)):
#0 erts_smp_proc_raw_trylock__ (c_p=0x7f4aac2e15d0, ref=139958707146706) at beam/erl_process_lock.h:422
#1 erts_smp_proc_lock__ (c_p=0x7f4aac2e15d0, ref=139958707146706) at beam/erl_process_lock.h:455
#2 erts_smp_proc_lock (c_p=0x7f4aac2e15d0, ref=139958707146706) at beam/erl_process_lock.h:672
#3 demonitor (c_p=0x7f4aac2e15d0, ref=139958707146706) at beam/bif.c:377
#4 0x0000000000477c30 in demonitor_2 (A__p=0x7f4aac2e15d0, A_1=139958707146706, A_2=<value optimized out>) at beam/bif.c:505
#5 0x0000000000521927 in process_main () at beam/beam_emu.c:2542
#6 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b00115200) at beam/erl_process.c:3782
#7 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200a0) at pthread/ethread.c:106
#8 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#9 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f4b00b7d700 (LWP 28400)):
#0 0x000000397260e4ed in read () from /lib64/libpthread.so.0
#1 0x000000000053810e in signal_dispatcher_thread_func (unused=<value optimized out>) at sys/unix/sys.c:2906
#2 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff3161ffc0) at pthread/ethread.c:106
#3 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#4 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f4b00ea9700 (LWP 28398)):
#0 0x0000003971ede8b3 in select () from /lib64/libc.so.6
#1 0x0000000000537ba0 in erts_sys_main_thread () at sys/unix/sys.c:3049
#2 0x0000000000444761 in erl_start (argc=54, argv=<value optimized out>) at beam/erl_init.c:1441
#3 0x0000000000426219 in main (argc=<value optimized out>, argv=<value optimized out>) at sys/unix/erl_main.c:29

Thread 7 (Thread 0x7f4aae343700 (LWP 28979)):
#0 0x0000003971eab15d in nanosleep () from /lib64/libc.so.6
#1 0x0000003971edf124 in usleep () from /lib64/libc.so.6
#2 0x00007f4af6371848 in v8::internal::ContextSwitcher::Run (this=0x7f4aa4030140) at src/v8threads.cc:457
#3 0x00007f4af62f1580 in v8::internal::ThreadEntry (arg=0x7f4aa4030140) at src/platform-linux.cc:745
#4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f4afea69700 (LWP 28420)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8750) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8750) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8750) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

---Type <return> to continue, or q <return> to quit---
Thread 5 (Thread 0x7f4afeb35700 (LWP 28414)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8480) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8480) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8480) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f4afeb79700 (LWP 28412)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8390) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8390) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8390) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f4afebbd700 (LWP 28410)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d82a0) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d82a0) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d82a0) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f4afebdf700 (LWP 28409)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8228) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8228) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8228) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f4afec01700 (LWP 28408)):
#0 0x000000000043ec6b in tree_delete (allctr=<value optimized out>, del=0x12c6a80, flags=<value optimized out>) at beam/erl_bestfit_alloc.c:435
#1 0x0000000000437091 in mbc_free (allctr=0x12a86c0, p=<value optimized out>) at beam/erl_alloc_util.c:867
#2 0x000000000043833f in do_erts_alcu_free (type=<value optimized out>, unused=<value optimized out>, p=0x12c6bb8) at beam/erl_alloc_util.c:2898
#3 erts_alcu_free_thr_pref (type=<value optimized out>, unused=<value optimized out>, p=0x12c6bb8) at beam/erl_alloc_util.c:2949
#4 0x000000000054ca48 in file_async_ready (e=0x12c3230, data=0x12c6bb8) at drivers/common/efile_drv.c:2117
#5 0x0000000000480094 in async_ready (p=0x7f4afeca4e90, data=<value optimized out>) at beam/io.c:2445
#6 0x00000000004e906f in async_main (arg=0x7f4aff3d81b0) at beam/erl_async.c:296
#7 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#8 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#9 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Comment by Karan Kumar (Inactive) [ 14/Sep/12 ]
https://friendpaste.com/6ctSNIR61xOtG8HSieJ4qA
Comment by Aleksey Kondratenko [ 14/Sep/12 ]
Another beam crash in some memory management code in beam. Perhaps duplicate of some other crash we have
Comment by Thuan Nguyen [ 14/Sep/12 ]
Raw erlang core dump is here https://s3.amazonaws.com/packages.couchbase/core-dump/orange/201209/core.beam.smp.28398.gz
Comment by Aleksey Kondratenko [ 14/Sep/12 ]
Karan, what you want me to do with that?
Comment by Aleksey Kondratenko [ 14/Sep/12 ]
I've heard you were looking for core dump?
Comment by Karan Kumar (Inactive) [ 14/Sep/12 ]
Ahh.... If we have duplicate of this bug.. somewhere we can close it..
But the segfault's of any component IMHO are quite severe
Comment by Karan Kumar (Inactive) [ 17/Sep/12 ]
Another beam.smp segfault:-

#0 right_rotate (allctr=<value optimized out>, del=<value optimized out>, flags=<value optimized out>) at beam/erl_bestfit_alloc.c:252
252 beam/erl_bestfit_alloc.c: No such file or directory.
in beam/erl_bestfit_alloc.c
Missing separate debuginfos, use: debuginfo-install couchbase-server-2.0.0-1728.x86_64
(gdb) t a a bt

Thread 28 (Thread 0x7f4b54ff3700 (LWP 6632)):
#0 0x000000344ee0ebad in fsync () from /lib64/libpthread.so.0
#1 0x000000000053b8db in efile_fsync (errInfo=0xa9744c, fd=<value optimized out>) at drivers/unix/unix_efile.c:806
#2 0x0000000000550e18 in invoke_fsync (data=0xa97420) at drivers/common/efile_drv.c:940
#3 0x00000000004e8f55 in async_main (arg=0x7f4b557a8138) at beam/erl_async.c:281
#4 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#5 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#6 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

Thread 27 (Thread 0x7f4b54dd1700 (LWP 6647)):
#0 0x000000344ee0dff4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000344ee09328 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x000000344ee091f7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x000000000043811f in ethr_mutex_lock (type=<value optimized out>, extra=<value optimized out>, size=280) at ../include/internal/ethr_mutex.h:589
#4 erts_mtx_lock (type=<value optimized out>, extra=<value optimized out>, size=280) at beam/erl_threads.h:622
#5 erts_alcu_alloc_thr_pref (type=<value optimized out>, extra=<value optimized out>, size=280) at beam/erl_alloc_util.c:2864
#6 0x000000000054fa5b in file_outputv (e=0xa8fb90, ev=0x7f4b54dd0c10) at drivers/common/efile_drv.c:2563
#7 0x0000000000483c78 in erts_write_to_port (caller_id=<value optimized out>, p=0x7f4b54dd0a10, list=139960099274753) at beam/io.c:1159
#8 0x00000000004f5301 in do_port_command (A__p=0x7f4b029c3060, A_1=638167, A_2=139961559834186, A_3=<value optimized out>, flags=0) at beam/erl_bif_port.c:181
#9 0x0000000000521927 in process_main () at beam/beam_emu.c:2542
#10 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b564bafc0) at beam/erl_process.c:3782
#11 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5fc0) at pthread/ethread.c:106
#12 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#13 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

Thread 26 (Thread 0x7f4b539cf700 (LWP 6649)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000585463 in wait__ (e=0xa19c60) at pthread/ethr_event.c:185
#2 ethr_event_wait (e=0xa19c60) at pthread/ethr_event.c:218
#3 0x000000000049821d in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f4b564d7140, rq=0x7f4b564ba7c0) at beam/erl_threads.h:1809
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4b564d7140, rq=0x7f4b564ba7c0) at beam/erl_process.c:1076
#5 0x000000000049a248 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:5421
#6 0x000000000051e65f in process_main () at beam/beam_emu.c:1225
#7 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b564d7140) at beam/erl_process.c:3782
#8 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5fc0) at pthread/ethread.c:106
#9 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#10 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

Thread 25 (Thread 0x7f4b543d0700 (LWP 6648)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000585463 in wait__ (e=0xa19ba0) at pthread/ethr_event.c:185
#2 ethr_event_wait (e=0xa19ba0) at pthread/ethr_event.c:218
#3 0x000000000049821d in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f4b564c9080, rq=0x7f4b564ba5c0) at beam/erl_threads.h:1809
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4b564c9080, rq=0x7f4b564ba5c0) at beam/erl_process.c:1076
#5 0x000000000049a248 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:5421
#6 0x000000000051e65f in process_main () at beam/beam_emu.c:1225
#7 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b564c9080) at beam/erl_process.c:3782
#8 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5fc0) at pthread/ethread.c:106
#9 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#10 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

Thread 24 (Thread 0x7f4b52fce700 (LWP 6650)):
#0 0x000000344e6e62c3 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000053de47 in check_fd_events (ps=0x7f4b570012c8, pr=0x7f4b52fcd360, len=0x7f4b52fcdb7c, utvp=0x7f4b52fcdb60) at sys/common/erl_poll.c:1918
#2 erts_poll_wait_kp (ps=0x7f4b570012c8, pr=0x7f4b52fcd360, len=0x7f4b52fcdb7c, utvp=0x7f4b52fcdb60) at sys/common/erl_poll.c:2053
#3 0x0000000000540efe in erts_check_io_kp (do_wait=<value optimized out>) at sys/common/erl_check_io.c:1161
#4 0x0000000000498551 in scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4b564e5200, rq=0x7f4b564ba9c0) at beam/erl_process.c:1221
#5 0x000000000049a248 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:5421
#6 0x000000000051e65f in process_main () at beam/beam_emu.c:1225
#7 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b564e5200) at beam/erl_process.c:3782
#8 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5fc0) at pthread/ethread.c:106
#9 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#10 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x7f4afdee0700 (LWP 7076)):
#0 0x000000344ee0d6c0 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f4b50a6718d in v8::internal::LinuxSemaphore::Wait (this=0x7f4b08009b10) at src/platform-linux.cc:869
#2 0x00007f4b50a7e45c in v8::internal::RuntimeProfiler::WaitForSomeIsolateToEnterJS () at src/runtime-profiler.cc:386
#3 0x00007f4b50a68136 in v8::internal::SignalSender::Run (this=0x7f4af80019c0) at src/platform-linux.cc:1126
#4 0x00007f4b50a67580 in v8::internal::ThreadEntry (arg=0x7f4af80019c0) at src/platform-linux.cc:745
#5 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#6 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7f4b54f27700 (LWP 6638)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8408) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8408) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8408) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7f4b5654c700 (LWP 6628)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x000000000046a1cf in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:697
#3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:472
#4 emergency_watchdog (unused=<value optimized out>) at beam/utils.c:3945
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5ff0) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7f4b54e9f700 (LWP 6642)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a85e8) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a85e8) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a85e8) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7f4b54faf700 (LWP 6634)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8228) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8228) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8228) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7f4b54ee3700 (LWP 6640)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a84f8) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a84f8) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a84f8) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7f4b56239700 (LWP 6629)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x0000000000462ad9 in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:697
#3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:472
#4 sys_msg_dispatcher_func (unused=<value optimized out>) at beam/erl_trace.c:3238
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5ff0) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f4b54f8d700 (LWP 6635)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a82a0) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a82a0) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a82a0) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

Thread 15 (Thread 0x7f4b0062c700 (LWP 6660)):
#0 0x000000344e6ab15d in nanosleep () from /lib64/libc.so.6
#1 0x000000344e6df124 in usleep () from /lib64/libc.so.6
#2 0x00007f4b00631397 in terminatorLoop (args=<value optimized out>) at mapreduce_nif.cc:445
#3 0x0000000000584d89 in thr_wrapper (vtwd=0x7f4b54dd0ae0) at pthread/ethread.c:106
#4 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7f4b54f05700 (LWP 6639)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8480) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8480) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8480) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7f4b55015700 (LWP 6631)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a80c0) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a80c0) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a80c0) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7f4af3171700 (LWP 7145)):
#0 0x000000344e6ab15d in nanosleep () from /lib64/libc.so.6
#1 0x000000344e6df124 in usleep () from /lib64/libc.so.6
#2 0x00007f4b50ae7848 in v8::internal::ContextSwitcher::Run (this=0x7f4b0414ace0) at src/v8threads.cc:457
#3 0x00007f4b50a67580 in v8::internal::ThreadEntry (arg=0x7f4b0414ace0) at src/platform-linux.cc:745
#4 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f4b56f4d700 (LWP 6627)):
#0 0x000000344ee0e4ed in read () from /lib64/libpthread.so.0
#1 0x000000000053810e in signal_dispatcher_thread_func (unused=<value optimized out>) at sys/unix/sys.c:2906
#2 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5ee0) at pthread/ethread.c:106
#3 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#4 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f4b54ec1700 (LWP 6641)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8570) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8570) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8570) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

Thread 9 (Thread 0x7f4b54f49700 (LWP 6637)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8390) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8390) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8390) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f4b57279700 (LWP 6626)):
#0 0x000000344e6de8b3 in select () from /lib64/libc.so.6
#1 0x0000000000537ba0 in erts_sys_main_thread () at sys/unix/sys.c:3049
#2 0x0000000000444761 in erl_start (argc=54, argv=<value optimized out>) at beam/erl_init.c:1441
#3 0x0000000000426219 in main (argc=<value optimized out>, argv=<value optimized out>) at sys/unix/erl_main.c:29

Thread 7 (Thread 0x7f4b54f6b700 (LWP 6636)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8318) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8318) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8318) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f4b54fd1700 (LWP 6633)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a81b0) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a81b0) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a81b0) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f4b54e17700 (LWP 6646)):
#0 0x000000344ee0f03d in waitpid () from /lib64/libpthread.so.0
#1 0x000000000053906f in child_waiter (unused=<value optimized out>) at sys/unix/sys.c:2821
#2 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5fa0) at pthread/ethread.c:106
#3 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#4 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f4b55037700 (LWP 6630)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8048) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8048) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8048) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f4b54e5b700 (LWP 6644)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a86d8) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a86d8) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a86d8) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f4b54e39700 (LWP 6645)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8750) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8750) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8750) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f4b54e7d700 (LWP 6643)):
#0 right_rotate (allctr=<value optimized out>, del=<value optimized out>, flags=<value optimized out>) at beam/erl_bestfit_alloc.c:252
#1 tree_delete (allctr=<value optimized out>, del=<value optimized out>, flags=<value optimized out>) at beam/erl_bestfit_alloc.c:496
#2 0x00000000004370db in mbc_free (allctr=0xa7c2c0, p=<value optimized out>) at beam/erl_alloc_util.c:883
#3 0x000000000043833f in do_erts_alcu_free (type=<value optimized out>, unused=<value optimized out>, p=0xa87a38) at beam/erl_alloc_util.c:2898
#4 erts_alcu_free_thr_pref (type=<value optimized out>, unused=<value optimized out>, p=0xa87a38) at beam/erl_alloc_util.c:2949
#5 0x000000000054cac0 in file_async_ready (e=0xa84238, data=0xa87a38) at drivers/common/efile_drv.c:2091
#6 0x0000000000480094 in async_ready (p=0x7f4b553521e0, data=<value optimized out>) at beam/io.c:2445
#7 0x00000000004e906f in async_main (arg=0x7f4b557a8660) at beam/erl_async.c:296
#8 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#9 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#10 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Comment by Thuan Nguyen [ 17/Sep/12 ]
Raw erlang core file is here https://s3.amazonaws.com/packages.couchbase/core-dump/orange/201209/core.beam.smp.6626.gz
Comment by Karan Kumar (Inactive) [ 17/Sep/12 ]
Another crash.

We are seeing it most deterministically.
https://friendpaste.com/55MrWtpTIufnEMRrxXbLsI
Comment by Thuan Nguyen [ 27/Sep/12 ]
Got another erlang segfault during swap rebalance in new system test setup with all SSD drive and consistent view enable
Erlang crashed on node added to cluster

Link to stack trace of erlang crash node https://friendpaste.com/58uGMaPS8xo2tcLsDSTTTM
Comment by Thuan Nguyen [ 11/Oct/12 ]
Stack trace erlang crash on build 2.0.0-1832 on system test

https://friendpaste.com/59jNoNCWLSrPOyTCQK76Rq
Comment by Filipe Manana [ 11/Oct/12 ]
Tony, do you think you can save all the files (database, indexes, etc) from the moment the crash happens?
I think it's more helpful rather than pasting a stack trace everytime it happens.
Comment by Filipe Manana [ 11/Oct/12 ]
Tony, does this always happen as well on the same machine (physical or virtual) ?
Comment by Thuan Nguyen [ 11/Oct/12 ]
Crash did not happen on the same vm. It happened not only in our vms but also in ec2 too as I did test last time
Comment by Thuan Nguyen [ 12/Oct/12 ]
Hit erlang crash on toybuild 10112 on system test.
Link to stack trace on node 39 https://friendpaste.com/59jNoNCWLSrPOyTCQKWoCO
Core dump is on /data on node 39.

Cluster information:
- 8 centos 6.2 64bit server with 4 cores CPU
- Each server has 32 GB RAM and 400 GB SSD disk.
- 24.8 GB RAM for couchbase server at each node
- SSD disk format ext4 on /data
- Each server has its own SSD drive, no disk sharing with other server.
- Create cluster with 6 nodes installed couchbase server 2.0.0-10112 (toy build)
- Cluster has 2 buckets, default (12GB) and saslbucket (12GB).
- Each bucket has one doc and 2 views for each doc (default d1 and saslbucket d11)
- Disable consistent view on cluster.


10.6.2.37
10.6.2.38
10.6.2.39
10.6.2.40
10.6.2.44
10.6.2.45
10.6.2.42
10.6.2.43

* Load 14 million items to both bucket. Each key has size from 512 bytes to 1024 bytes
* Queries all 4 views from 2 docs
* Mutate 14 million items with key size from 1500 to 1024 bytes

* Do swap rebalance 2 nodes at the same time.
* Cluster is left as in crashed state to investigate
* Link to collect info of all nodes https://s3.amazonaws.com/packages.couchbase/collect_info/orange/2_0_0/201210/8nodes-col-toybuild-10112-beam-segfault-20121012-003212.tgz

Comment by damien [ 12/Oct/12 ]
Right now we think this might be related to another bug spotted on Erlang VMs. When started with async threads and using "raw" file descriptors, if the process that opened the file is shutdown or crashes abnormally, the the file descriptor is leaked.

one possible fix is to change the erlang startup parameters to turn off async file io, so this:
> erl +A 16 +sbt u +P 327680 +K true
becomes:
> erl +sbt u +P 327680 +K true
Comment by Aleksey Kondratenko [ 12/Oct/12 ]
Just keep in mind that without async io we'll have _massive_ timeouts all over the place. We specifically _had exactly this problem_ in early days of 1.6.0.
Comment by Farshid Ghods (Inactive) [ 17/Oct/12 ]
according to karan he has not seen any crash or rebalance timeouts on cluster where there are lot of rebalancing
1- 2.0 cluster doing views , 2 buckets , 30M items , 2 ddocs , 2 views , 8 nodes under 30k ops/sec and 300 queries per second
2- 20+ node cluster running 1.8.x key value use cases

performance team is also running more performance tests to verify that +s12 works ( without +a) option

we have not yet run xdcr system tests with these settings.
Comment by Ketaki Gangal [ 19/Oct/12 ]
Tested xdcr-functional tests with the setting +S 120:120, it works fine.
Comment by Steve Yen [ 19/Oct/12 ]
Notes from mtg with damien, alk, farshid, sriram, sharon, aaron, yaseen, aaron

priority: key-value use-case cannot regress from 1.8.1

plan A - more testing with no-async (farshid).
* +S 120:120 (not 12:12)
* QE run tests on real disk & durable configuration setting.
* look for timeouts in the U/I.
* windows.
* use physical disk boxes from perf team.

plan B - track down root cause.
* QE will run system tests against toy-build from siri, on VMs ok.
  * instrument memory calls, look for culprit - toy build (siri).
* idea - hammer NIF erlang unit tests repeatedly overnight (alk).
* idea - try "A1" (try this if plan A doesn't work).
  + frees up scheduler threads.
  + and single async-i/o thread might mitigate crashes.
  - this will serialize I/O even if we have multiple disks.

plan C - separate processes (if plan A doesn't work, and alk's team has time).
+ should fix the problem.
- costly.
- some tight coupling between some modules.

plan D - run async for key-value use case, sync for view use case.

-----------
by the way, 1.8.1 used async threads.

async threads mode can leak file descriptors.
* this was some different issue.
Comment by Aleksey Kondratenko [ 19/Oct/12 ]
Just verified perf team's RHEL 5.8 boxes. Indeed RHEL 5.8 even on ext3 has barriers ON by default. That's good for me.

I tested with my tool and I'm seeing typical performance from 7.2k RPMs disk.
Comment by Ronnie Sun (Inactive) [ 19/Oct/12 ]
btw, the test we were running used +S 16:16 instead of 12:12
Comment by Aleksey Kondratenko [ 19/Oct/12 ]
Tested on /data2 partition that's on lvm. And found that barriers actually don't work.

So system tests physical box is NOT ok. Looks like older (or maybe even modern) LVM does not provide durable disk configuration.
Comment by damien [ 23/Oct/12 ]
Mounting evidence suggests this bug is caused by the +A erlang startup settings, which turns on async IO for port drivers by using a pool of threads to perform the IO. We still don't understand the root cause, but it appears there is a race condition/cache coherency problem with how port drivers are freed in the VM.

Reassigning to Alk, as he will check in the change to disable the async threads and bump up the # of schedulers to mitigate problems with timeouts due to blocking IO.
Comment by Aleksey Kondratenko [ 23/Oct/12 ]
We merged "async off" commit.
Comment by kzeller [ 12/Nov/12 ]
Added to RN : Couchbase Server had intermittently crashed
during rebalance due to Erlang virtual machine
issues; we now
disable asynchronous threads and perform garbage collection
more often to avoid timeouts and process crashes.
Comment by Thuan Nguyen [ 27/Dec/12 ]
Integrated in github-couchdb-preview #544 (See [http://qa.hq.northscale.net/job/github-couchdb-preview/544/])
    MB-6638: use fork of file_sorter that doesn't use compression (Revision 6bbe1cf89b2f6b5c9cf098b81c5ea60d339f8f0a)

     Result = SUCCESS
Aliaksey Kandratsenka :
Files :
* src/couch_set_view/src/couch_set_view_updater.erl
* src/couchdb/couch_db_updater.erl
* src/couchdb/Makefile.am
* Makefile.am
* src/couchdb/file_sorter_2.erl
Generated at Wed Aug 20 03:52:21 CDT 2014 using JIRA 5.2.4#845-sha1:c9f4cc41abe72fb236945343a1f485c2c844dac9.