Details
-
Type:
Bug
-
Status:
Closed
-
Priority:
Blocker
-
Resolution: Fixed
-
Affects Version/s: 2.0-beta-2
-
Fix Version/s: 2.0-beta-2
-
Component/s: 3rd-party, ns_server, view-engine
-
Security Level: Public
-
Labels:
-
Environment:centos 6.2 build 2.0.0-1708
Description
Cluster information:
- Load 9 million items to both buckets
- Cluster has 2 buckets, default (3GB) and saslbucket (3GB)
- Each bucket has one doc and 2 views for each doc (default d1 and saslbucket d11)
* Create cluster with 10 nodes installed couchbase server 2.0.0-1708
10.3.121.13
10.3.121.14
10.3.121.15
10.3.121.16
10.3.121.17
10.3.121.20
10.3.121.22
10.3.121.24
10.3.121.25
10.3.121.23
* Data path /data
* View path /data
Doing rebalanc in, out, failover.
Last failed was rebalance add 2 nodes, 23 and 24 into cluster.
During rebalance, node 23 suddenly down. Rebalance failed as expected.
Then later, memcached on node 13 was killed and restart.
[ns_server:debug,2012-09-12T17:06:05.705,ns_1@10.3.121.13:<0.28110.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_config_events,<0.28109.33>} exited with reason {timeout,
{gen_server,
call,
[ns_node_disco,
nodes_wanted]}}
[ns_server:debug,2012-09-12T17:06:05.712,ns_1@10.3.121.13:<0.28166.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_config_events,<0.28165.33>} exited with reason shutdown
[ns_server:error,2012-09-12T17:06:05.718,ns_1@10.3.121.13:<0.29417.33>:ns_memcached:verify_report_long_call:274]call {stats,<<>>} took too long: 5881903 us
[ns_server:debug,2012-09-12T17:06:05.720,ns_1@10.3.121.13:<0.28132.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_node_disco_events,<0.28130.33>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:05.732,ns_1@10.3.121.13:<0.28110.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_config_events,<0.28109.33>} event handler: ok
[ns_server:debug,2012-09-12T17:06:05.733,ns_1@10.3.121.13:<0.28166.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_config_events,<0.28165.33>} event handler: ok
[ns_server:debug,2012-09-12T17:06:05.739,ns_1@10.3.121.13:<0.5306.465>:compaction_daemon:do_chain_compactors:500]Got exit signal from parent: {'EXIT',<0.4974.465>,shutdown}
[ns_server:debug,2012-09-12T17:06:05.740,ns_1@10.3.121.13:<0.28132.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_node_disco_events,<0.28130.33>} event handler: ok
[ns_server:debug,2012-09-12T17:06:05.740,ns_1@10.3.121.13:ns_node_disco:ns_node_disco:init:103]Initting ns_node_disco with ['ns_1@10.3.121.15','ns_1@10.3.121.16',
'ns_1@10.3.121.17','ns_1@10.3.121.22',
'ns_1@10.3.121.20','ns_1@10.3.121.14',
'ns_1@10.3.121.25','ns_1@10.3.121.24']
[ns_server:debug,2012-09-12T17:06:05.741,ns_1@10.3.121.13:<0.28129.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_config_events,<0.28128.33>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:05.743,ns_1@10.3.121.13:<0.19269.466>:janitor_agent:new_style_query_vbucket_states_loop:111]Exception from query_vbucket_states of "saslbucket":'ns_1@10.3.121.23'
{'EXIT',{{nodedown,'ns_1@10.3.121.23'},
{gen_server,call,
[{'janitor_agent-saslbucket','ns_1@10.3.121.23'},
query_vbucket_states,infinity]}}}
[ns_server:debug,2012-09-12T17:06:05.745,ns_1@10.3.121.13:<0.5309.465>:compaction_daemon:do_chain_compactors:500]Got exit signal from parent: {'EXIT',<0.5306.465>,shutdown}
[ns_server:debug,2012-09-12T17:06:05.746,ns_1@10.3.121.13:<0.28126.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_config_events,<0.28125.33>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:05.761,ns_1@10.3.121.13:<0.19269.466>:janitor_agent:new_style_query_vbucket_states_loop_next_step:116]Waiting for "saslbucket" on 'ns_1@10.3.121.23'
[ns_server:debug,2012-09-12T17:06:05.897,ns_1@10.3.121.13:<0.28129.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_config_events,<0.28128.33>} event handler: ok
[ns_server:debug,2012-09-12T17:06:05.898,ns_1@10.3.121.13:<0.14047.466>:compaction_daemon:do_spawn_view_index_compactor:742]Got exit signal from parent: {'EXIT',<0.5309.465>,shutdown}
[ns_server:debug,2012-09-12T17:06:05.903,ns_1@10.3.121.13:ns_cookie_manager:ns_cookie_manager:do_cookie_sync:115]ns_cookie_manager do_cookie_sync
[ns_server:debug,2012-09-12T17:06:05.903,ns_1@10.3.121.13:<0.28126.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_config_events,<0.28125.33>} event handler: ok
[couchdb:info,2012-09-12T17:06:05.906,ns_1@10.3.121.13:<0.5719.436>:couch_log:info:39]Set view `default`, replica group `_design/d1`, canceling compaction (pid <0.18999.466>)
[ns_server:debug,2012-09-12T17:06:06.000,ns_1@10.3.121.13:ns_cookie_manager:ns_cookie_manager:do_cookie_save:152]saving cookie to "/opt/couchbase/var/lib/couchbase/couchbase-server.cookie"
[ns_server:debug,2012-09-12T17:06:06.002,ns_1@10.3.121.13:<0.28122.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_stats_event,<0.28121.33>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:06.009,ns_1@10.3.121.13:<0.28122.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_stats_event,<0.28121.33>} event handler: ok
[ns_server:debug,2012-09-12T17:06:06.040,ns_1@10.3.121.13:<0.28120.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_tick_event,<0.28119.33>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:06.044,ns_1@10.3.121.13:<0.28648.34>:single_bucket_sup:top_loop:28]Delegating exit {'EXIT',<0.28117.33>,shutdown} to child supervisor: <0.28649.34>
[ns_server:debug,2012-09-12T17:06:06.047,ns_1@10.3.121.13:ns_cookie_manager:ns_cookie_manager:do_cookie_save:154]attempted to save cookie to "/opt/couchbase/var/lib/couchbase/couchbase-server.cookie": ok
[ns_server:debug,2012-09-12T17:06:06.074,ns_1@10.3.121.13:<0.28120.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_tick_event,<0.28119.33>} event handler: ok
[ns_server:debug,2012-09-12T17:06:06.074,ns_1@10.3.121.13:<0.19276.466>:ns_node_disco:do_nodes_wanted_updated_fun:202]ns_node_disco: nodes_wanted updated: ['ns_1@10.3.121.13','ns_1@10.3.121.14',
'ns_1@10.3.121.15','ns_1@10.3.121.16',
'ns_1@10.3.121.17','ns_1@10.3.121.20',
'ns_1@10.3.121.22','ns_1@10.3.121.23',
'ns_1@10.3.121.24','ns_1@10.3.121.25'], with cookie: oetfisqvqrrxvaat
[ns_server:debug,2012-09-12T17:06:06.079,ns_1@10.3.121.13:<0.28689.34>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_stats_event,<0.28688.34>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:06.080,ns_1@10.3.121.13:<0.28687.34>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_tick_event,<0.28686.34>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:06.083,ns_1@10.3.121.13:<0.28689.34>:ns_pubsub:do_subscribe_link:149]Deleting {ns_stats_event,<0.28688.34>} event handler: ok
[ns_server:debug,2012-09-12T17:06:06.084,ns_1@10.3.121.13:<0.28687.34>:ns_pubsub:do_subscribe_link:149]Deleting {ns_tick_event,<0.28686.34>} event handler: ok
[stats:warn,2012-09-12T17:06:06.087,ns_1@10.3.121.13:<0.29425.33>:stats_collector:latest_tick:201]Dropped 7 ticks
[rebalance:info,2012-09-12T17:06:06.091,ns_1@10.3.121.13:<0.25334.457>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!
[rebalance:info,2012-09-12T17:06:06.120,ns_1@10.3.121.13:<0.23919.457>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!
[ns_server:debug,2012-09-12T17:06:06.120,ns_1@10.3.121.13:<0.1936.453>:ns_pubsub:do_subscribe_link:149]Deleting {master_activity_events_ingress,<0.28014.33>} event handler: {error,
module_not_found}
[rebalance:info,2012-09-12T17:06:06.128,ns_1@10.3.121.13:<0.23192.457>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!
[rebalance:info,2012-09-12T17:06:06.147,ns_1@10.3.121.13:<0.22947.457>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!
[rebalance:info,2012-09-12T17:06:06.201,ns_1@10.3.121.13:<0.21996.457>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!
[rebalance:info,2012-09-12T17:06:06.218,ns_1@10.3.121.13:<0.11912.457>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!
[rebalance:info,2012-09-12T17:06:06.223,ns_1@10.3.121.13:<0.10815.457>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!
[ns_server:debug,2012-09-12T17:06:06.228,ns_1@10.3.121.13:<0.28050.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {buckets_events,<0.28049.33>} exited with reason {shutdown,
{gen_server,
call,
['ns_vbm_new_sup-default',
which_children,
infinity]}}
[ns_server:debug,2012-09-12T17:06:06.239,ns_1@10.3.121.13:<0.28050.33>:ns_pubsub:do_subscribe_link:149]Deleting {buckets_events,<0.28049.33>} event handler: ok
[user:info,2012-09-12T17:06:06.245,ns_1@10.3.121.13:'ns_memcached-default':ns_memcached:terminate:625]Shutting down bucket "default" on 'ns_1@10.3.121.13' for server shutdown
[ns_server:info,2012-09-12T17:06:06.261,ns_1@10.3.121.13:'ns_memcached-default':ns_memcached:terminate:636]This bucket shutdown is not due to bucket deletion. Doing nothing
[ns_server:debug,2012-09-12T17:06:06.276,ns_1@10.3.121.13:<0.28651.34>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_config_events,<0.28650.34>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:06.276,ns_1@10.3.121.13:<0.28663.34>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {mc_couch_events,<0.28650.34>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:06.276,ns_1@10.3.121.13:<0.28662.34>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_node_disco_events,<0.28650.34>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:06.276,ns_1@10.3.121.13:<0.28648.34>:single_bucket_sup:top_loop:24]per-bucket supervisor for "default" died with reason shutdown
[ns_server:debug,2012-09-12T17:06:06.279,ns_1@10.3.121.13:<0.28651.34>:ns_pubsub:do_subscribe_link:149]Deleting {ns_config_events,<0.28650.34>} event handler: ok
[ns_server:debug,2012-09-12T17:06:06.281,ns_1@10.3.121.13:<0.28663.34>:ns_pubsub:do_subscribe_link:149]Deleting {mc_couch_events,<0.28650.34>} event handler: ok
[ns_server:debug,2012-09-12T17:06:06.283,ns_1@10.3.121.13:<0.28662.34>:ns_pubsub:do_subscribe_link:149]Deleting {ns_node_disco_events,<0.28650.34>} event handler: ok
[ns_server:debug,2012-09-12T17:06:06.283,ns_1@10.3.121.13:<0.29393.33>:single_bucket_sup:top_loop:28]Delegating exit {'EXIT',<0.28117.33>,shutdown} to child supervisor: <0.29394.33>
[ns_server:debug,2012-09-12T17:06:06.299,ns_1@10.3.121.13:<0.29428.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_stats_event,<0.29427.33>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:06.300,ns_1@10.3.121.13:<0.29426.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_tick_event,<0.29425.33>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:06.305,ns_1@10.3.121.13:<0.29428.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_stats_event,<0.29427.33>} event handler: ok
[ns_server:debug,2012-09-12T17:06:06.307,ns_1@10.3.121.13:<0.29426.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_tick_event,<0.29425.33>} event handler: ok
[rebalance:info,2012-09-12T17:06:06.310,ns_1@10.3.121.13:<0.11978.466>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!
[ns_server:info,2012-09-12T17:06:06.323,ns_1@10.3.121.13:ns_port_memcached:ns_port_server:log:169]memcached<0.28094.33>: Thu Sep 13 00:06:06.117323 3: TAP (Consumer) eq_tapq:anon_509 - disconnected
memcached<0.28094.33>: Thu Sep 13 00:06:06.123981 3: TAP (Consumer) eq_tapq:anon_1652 - disconnected
memcached<0.28094.33>: Thu Sep 13 00:06:06.144392 3: TAP (Consumer) eq_tapq:anon_480 - disconnected
memcached<0.28094.33>: Thu Sep 13 00:06:06.151955 3: TAP (Consumer) eq_tapq:anon_488 - disconnected
memcached<0.28094.33>: Thu Sep 13 00:06:06.203038 3: TAP (Consumer) eq_tapq:anon_504 - disconnected
memcached<0.28094.33>: Thu Sep 13 00:06:06.221683 3: TAP (Consumer) eq_tapq:anon_719 - disconnected
memcached<0.28094.33>: Thu Sep 13 00:06:06.227307 3: TAP (Consumer) eq_tapq:anon_499 - disconnected
memcached log from node 13 at the time it was killed
Thu Sep 13 00:06:07.606021 3: Schedule cleanup of "eq_tapq:anon_20"
Thu Sep 13 00:06:07.606058 3: Schedule cleanup of "eq_tapq:anon_34"
Thu Sep 13 00:06:07.606083 3: Schedule cleanup of "eq_tapq:anon_36"
Thu Sep 13 00:06:07.606115 3: Schedule cleanup of "eq_tapq:anon_40"
Thu Sep 13 00:06:07.606142 3: Schedule cleanup of "eq_tapq:anon_730"
Thu Sep 13 00:06:07.606165 3: Schedule cleanup of "eq_tapq:anon_2425"
Thu Sep 13 00:06:21.096648 3: Had to wait 13 s for shutdown
Thu Sep 13 00:06:31.465158 3: Shutting down tap connections!
Thu Sep 13 00:06:31.465199 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.15"
Thu Sep 13 00:06:31.465278 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.16"
Thu Sep 13 00:06:31.465357 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.22"
Thu Sep 13 00:06:31.465405 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.17"
Thu Sep 13 00:06:31.465414 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.15 - Clear the tap queues by force
Thu Sep 13 00:06:31.465441 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.20"
Thu Sep 13 00:06:31.465471 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.14"
Thu Sep 13 00:06:31.465492 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.25"
Thu Sep 13 00:06:31.465517 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.24"
Thu Sep 13 00:06:31.465604 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.16 - Clear the tap queues by force
Thu Sep 13 00:06:31.467071 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.22 - Clear the tap queues by force
Thu Sep 13 00:06:31.467363 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.20 - Clear the tap queues by force
Thu Sep 13 00:06:31.467606 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.24 - Clear the tap queues by force
Thu Sep 13 00:06:31.467710 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.25 - Clear the tap queues by force
Thu Sep 13 00:06:31.467917 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.17 - Clear the tap queues by force
Thu Sep 13 00:06:31.468102 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.14 - Clear the tap queues by force
Link to diags of all nodes https://s3.amazonaws.com/packages.couchbase/diag-logs/orange/201209/10nodes-1708-memcached-killed-after-reb-failed-20120912.tgz
Link to atop of node 13 https://s3.amazonaws.com/packages.couchbase/atop-files/orange/201209/atop-node13-20120912
- Load 9 million items to both buckets
- Cluster has 2 buckets, default (3GB) and saslbucket (3GB)
- Each bucket has one doc and 2 views for each doc (default d1 and saslbucket d11)
* Create cluster with 10 nodes installed couchbase server 2.0.0-1708
10.3.121.13
10.3.121.14
10.3.121.15
10.3.121.16
10.3.121.17
10.3.121.20
10.3.121.22
10.3.121.24
10.3.121.25
10.3.121.23
* Data path /data
* View path /data
Doing rebalanc in, out, failover.
Last failed was rebalance add 2 nodes, 23 and 24 into cluster.
During rebalance, node 23 suddenly down. Rebalance failed as expected.
Then later, memcached on node 13 was killed and restart.
[ns_server:debug,2012-09-12T17:06:05.705,ns_1@10.3.121.13:<0.28110.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_config_events,<0.28109.33>} exited with reason {timeout,
{gen_server,
call,
[ns_node_disco,
nodes_wanted]}}
[ns_server:debug,2012-09-12T17:06:05.712,ns_1@10.3.121.13:<0.28166.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_config_events,<0.28165.33>} exited with reason shutdown
[ns_server:error,2012-09-12T17:06:05.718,ns_1@10.3.121.13:<0.29417.33>:ns_memcached:verify_report_long_call:274]call {stats,<<>>} took too long: 5881903 us
[ns_server:debug,2012-09-12T17:06:05.720,ns_1@10.3.121.13:<0.28132.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_node_disco_events,<0.28130.33>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:05.732,ns_1@10.3.121.13:<0.28110.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_config_events,<0.28109.33>} event handler: ok
[ns_server:debug,2012-09-12T17:06:05.733,ns_1@10.3.121.13:<0.28166.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_config_events,<0.28165.33>} event handler: ok
[ns_server:debug,2012-09-12T17:06:05.739,ns_1@10.3.121.13:<0.5306.465>:compaction_daemon:do_chain_compactors:500]Got exit signal from parent: {'EXIT',<0.4974.465>,shutdown}
[ns_server:debug,2012-09-12T17:06:05.740,ns_1@10.3.121.13:<0.28132.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_node_disco_events,<0.28130.33>} event handler: ok
[ns_server:debug,2012-09-12T17:06:05.740,ns_1@10.3.121.13:ns_node_disco:ns_node_disco:init:103]Initting ns_node_disco with ['ns_1@10.3.121.15','ns_1@10.3.121.16',
'ns_1@10.3.121.17','ns_1@10.3.121.22',
'ns_1@10.3.121.20','ns_1@10.3.121.14',
'ns_1@10.3.121.25','ns_1@10.3.121.24']
[ns_server:debug,2012-09-12T17:06:05.741,ns_1@10.3.121.13:<0.28129.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_config_events,<0.28128.33>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:05.743,ns_1@10.3.121.13:<0.19269.466>:janitor_agent:new_style_query_vbucket_states_loop:111]Exception from query_vbucket_states of "saslbucket":'ns_1@10.3.121.23'
{'EXIT',{{nodedown,'ns_1@10.3.121.23'},
{gen_server,call,
[{'janitor_agent-saslbucket','ns_1@10.3.121.23'},
query_vbucket_states,infinity]}}}
[ns_server:debug,2012-09-12T17:06:05.745,ns_1@10.3.121.13:<0.5309.465>:compaction_daemon:do_chain_compactors:500]Got exit signal from parent: {'EXIT',<0.5306.465>,shutdown}
[ns_server:debug,2012-09-12T17:06:05.746,ns_1@10.3.121.13:<0.28126.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_config_events,<0.28125.33>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:05.761,ns_1@10.3.121.13:<0.19269.466>:janitor_agent:new_style_query_vbucket_states_loop_next_step:116]Waiting for "saslbucket" on 'ns_1@10.3.121.23'
[ns_server:debug,2012-09-12T17:06:05.897,ns_1@10.3.121.13:<0.28129.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_config_events,<0.28128.33>} event handler: ok
[ns_server:debug,2012-09-12T17:06:05.898,ns_1@10.3.121.13:<0.14047.466>:compaction_daemon:do_spawn_view_index_compactor:742]Got exit signal from parent: {'EXIT',<0.5309.465>,shutdown}
[ns_server:debug,2012-09-12T17:06:05.903,ns_1@10.3.121.13:ns_cookie_manager:ns_cookie_manager:do_cookie_sync:115]ns_cookie_manager do_cookie_sync
[ns_server:debug,2012-09-12T17:06:05.903,ns_1@10.3.121.13:<0.28126.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_config_events,<0.28125.33>} event handler: ok
[couchdb:info,2012-09-12T17:06:05.906,ns_1@10.3.121.13:<0.5719.436>:couch_log:info:39]Set view `default`, replica group `_design/d1`, canceling compaction (pid <0.18999.466>)
[ns_server:debug,2012-09-12T17:06:06.000,ns_1@10.3.121.13:ns_cookie_manager:ns_cookie_manager:do_cookie_save:152]saving cookie to "/opt/couchbase/var/lib/couchbase/couchbase-server.cookie"
[ns_server:debug,2012-09-12T17:06:06.002,ns_1@10.3.121.13:<0.28122.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_stats_event,<0.28121.33>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:06.009,ns_1@10.3.121.13:<0.28122.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_stats_event,<0.28121.33>} event handler: ok
[ns_server:debug,2012-09-12T17:06:06.040,ns_1@10.3.121.13:<0.28120.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_tick_event,<0.28119.33>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:06.044,ns_1@10.3.121.13:<0.28648.34>:single_bucket_sup:top_loop:28]Delegating exit {'EXIT',<0.28117.33>,shutdown} to child supervisor: <0.28649.34>
[ns_server:debug,2012-09-12T17:06:06.047,ns_1@10.3.121.13:ns_cookie_manager:ns_cookie_manager:do_cookie_save:154]attempted to save cookie to "/opt/couchbase/var/lib/couchbase/couchbase-server.cookie": ok
[ns_server:debug,2012-09-12T17:06:06.074,ns_1@10.3.121.13:<0.28120.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_tick_event,<0.28119.33>} event handler: ok
[ns_server:debug,2012-09-12T17:06:06.074,ns_1@10.3.121.13:<0.19276.466>:ns_node_disco:do_nodes_wanted_updated_fun:202]ns_node_disco: nodes_wanted updated: ['ns_1@10.3.121.13','ns_1@10.3.121.14',
'ns_1@10.3.121.15','ns_1@10.3.121.16',
'ns_1@10.3.121.17','ns_1@10.3.121.20',
'ns_1@10.3.121.22','ns_1@10.3.121.23',
'ns_1@10.3.121.24','ns_1@10.3.121.25'], with cookie: oetfisqvqrrxvaat
[ns_server:debug,2012-09-12T17:06:06.079,ns_1@10.3.121.13:<0.28689.34>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_stats_event,<0.28688.34>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:06.080,ns_1@10.3.121.13:<0.28687.34>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_tick_event,<0.28686.34>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:06.083,ns_1@10.3.121.13:<0.28689.34>:ns_pubsub:do_subscribe_link:149]Deleting {ns_stats_event,<0.28688.34>} event handler: ok
[ns_server:debug,2012-09-12T17:06:06.084,ns_1@10.3.121.13:<0.28687.34>:ns_pubsub:do_subscribe_link:149]Deleting {ns_tick_event,<0.28686.34>} event handler: ok
[stats:warn,2012-09-12T17:06:06.087,ns_1@10.3.121.13:<0.29425.33>:stats_collector:latest_tick:201]Dropped 7 ticks
[rebalance:info,2012-09-12T17:06:06.091,ns_1@10.3.121.13:<0.25334.457>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!
[rebalance:info,2012-09-12T17:06:06.120,ns_1@10.3.121.13:<0.23919.457>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!
[ns_server:debug,2012-09-12T17:06:06.120,ns_1@10.3.121.13:<0.1936.453>:ns_pubsub:do_subscribe_link:149]Deleting {master_activity_events_ingress,<0.28014.33>} event handler: {error,
module_not_found}
[rebalance:info,2012-09-12T17:06:06.128,ns_1@10.3.121.13:<0.23192.457>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!
[rebalance:info,2012-09-12T17:06:06.147,ns_1@10.3.121.13:<0.22947.457>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!
[rebalance:info,2012-09-12T17:06:06.201,ns_1@10.3.121.13:<0.21996.457>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!
[rebalance:info,2012-09-12T17:06:06.218,ns_1@10.3.121.13:<0.11912.457>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!
[rebalance:info,2012-09-12T17:06:06.223,ns_1@10.3.121.13:<0.10815.457>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!
[ns_server:debug,2012-09-12T17:06:06.228,ns_1@10.3.121.13:<0.28050.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {buckets_events,<0.28049.33>} exited with reason {shutdown,
{gen_server,
call,
['ns_vbm_new_sup-default',
which_children,
infinity]}}
[ns_server:debug,2012-09-12T17:06:06.239,ns_1@10.3.121.13:<0.28050.33>:ns_pubsub:do_subscribe_link:149]Deleting {buckets_events,<0.28049.33>} event handler: ok
[user:info,2012-09-12T17:06:06.245,ns_1@10.3.121.13:'ns_memcached-default':ns_memcached:terminate:625]Shutting down bucket "default" on 'ns_1@10.3.121.13' for server shutdown
[ns_server:info,2012-09-12T17:06:06.261,ns_1@10.3.121.13:'ns_memcached-default':ns_memcached:terminate:636]This bucket shutdown is not due to bucket deletion. Doing nothing
[ns_server:debug,2012-09-12T17:06:06.276,ns_1@10.3.121.13:<0.28651.34>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_config_events,<0.28650.34>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:06.276,ns_1@10.3.121.13:<0.28663.34>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {mc_couch_events,<0.28650.34>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:06.276,ns_1@10.3.121.13:<0.28662.34>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_node_disco_events,<0.28650.34>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:06.276,ns_1@10.3.121.13:<0.28648.34>:single_bucket_sup:top_loop:24]per-bucket supervisor for "default" died with reason shutdown
[ns_server:debug,2012-09-12T17:06:06.279,ns_1@10.3.121.13:<0.28651.34>:ns_pubsub:do_subscribe_link:149]Deleting {ns_config_events,<0.28650.34>} event handler: ok
[ns_server:debug,2012-09-12T17:06:06.281,ns_1@10.3.121.13:<0.28663.34>:ns_pubsub:do_subscribe_link:149]Deleting {mc_couch_events,<0.28650.34>} event handler: ok
[ns_server:debug,2012-09-12T17:06:06.283,ns_1@10.3.121.13:<0.28662.34>:ns_pubsub:do_subscribe_link:149]Deleting {ns_node_disco_events,<0.28650.34>} event handler: ok
[ns_server:debug,2012-09-12T17:06:06.283,ns_1@10.3.121.13:<0.29393.33>:single_bucket_sup:top_loop:28]Delegating exit {'EXIT',<0.28117.33>,shutdown} to child supervisor: <0.29394.33>
[ns_server:debug,2012-09-12T17:06:06.299,ns_1@10.3.121.13:<0.29428.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_stats_event,<0.29427.33>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:06.300,ns_1@10.3.121.13:<0.29426.33>:ns_pubsub:do_subscribe_link:134]Parent process of subscription {ns_tick_event,<0.29425.33>} exited with reason shutdown
[ns_server:debug,2012-09-12T17:06:06.305,ns_1@10.3.121.13:<0.29428.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_stats_event,<0.29427.33>} event handler: ok
[ns_server:debug,2012-09-12T17:06:06.307,ns_1@10.3.121.13:<0.29426.33>:ns_pubsub:do_subscribe_link:149]Deleting {ns_tick_event,<0.29425.33>} event handler: ok
[rebalance:info,2012-09-12T17:06:06.310,ns_1@10.3.121.13:<0.11978.466>:ebucketmigrator_srv:do_confirm_sent_messages:655]Got close ack!
[ns_server:info,2012-09-12T17:06:06.323,ns_1@10.3.121.13:ns_port_memcached:ns_port_server:log:169]memcached<0.28094.33>: Thu Sep 13 00:06:06.117323 3: TAP (Consumer) eq_tapq:anon_509 - disconnected
memcached<0.28094.33>: Thu Sep 13 00:06:06.123981 3: TAP (Consumer) eq_tapq:anon_1652 - disconnected
memcached<0.28094.33>: Thu Sep 13 00:06:06.144392 3: TAP (Consumer) eq_tapq:anon_480 - disconnected
memcached<0.28094.33>: Thu Sep 13 00:06:06.151955 3: TAP (Consumer) eq_tapq:anon_488 - disconnected
memcached<0.28094.33>: Thu Sep 13 00:06:06.203038 3: TAP (Consumer) eq_tapq:anon_504 - disconnected
memcached<0.28094.33>: Thu Sep 13 00:06:06.221683 3: TAP (Consumer) eq_tapq:anon_719 - disconnected
memcached<0.28094.33>: Thu Sep 13 00:06:06.227307 3: TAP (Consumer) eq_tapq:anon_499 - disconnected
memcached log from node 13 at the time it was killed
Thu Sep 13 00:06:07.606021 3: Schedule cleanup of "eq_tapq:anon_20"
Thu Sep 13 00:06:07.606058 3: Schedule cleanup of "eq_tapq:anon_34"
Thu Sep 13 00:06:07.606083 3: Schedule cleanup of "eq_tapq:anon_36"
Thu Sep 13 00:06:07.606115 3: Schedule cleanup of "eq_tapq:anon_40"
Thu Sep 13 00:06:07.606142 3: Schedule cleanup of "eq_tapq:anon_730"
Thu Sep 13 00:06:07.606165 3: Schedule cleanup of "eq_tapq:anon_2425"
Thu Sep 13 00:06:21.096648 3: Had to wait 13 s for shutdown
Thu Sep 13 00:06:31.465158 3: Shutting down tap connections!
Thu Sep 13 00:06:31.465199 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.15"
Thu Sep 13 00:06:31.465278 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.16"
Thu Sep 13 00:06:31.465357 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.22"
Thu Sep 13 00:06:31.465405 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.17"
Thu Sep 13 00:06:31.465414 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.15 - Clear the tap queues by force
Thu Sep 13 00:06:31.465441 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.20"
Thu Sep 13 00:06:31.465471 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.14"
Thu Sep 13 00:06:31.465492 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.25"
Thu Sep 13 00:06:31.465517 3: Schedule cleanup of "eq_tapq:replication_ns_1@10.3.121.24"
Thu Sep 13 00:06:31.465604 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.16 - Clear the tap queues by force
Thu Sep 13 00:06:31.467071 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.22 - Clear the tap queues by force
Thu Sep 13 00:06:31.467363 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.20 - Clear the tap queues by force
Thu Sep 13 00:06:31.467606 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.24 - Clear the tap queues by force
Thu Sep 13 00:06:31.467710 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.25 - Clear the tap queues by force
Thu Sep 13 00:06:31.467917 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.17 - Clear the tap queues by force
Thu Sep 13 00:06:31.468102 3: TAP (Producer) eq_tapq:replication_ns_1@10.3.121.14 - Clear the tap queues by force
Link to diags of all nodes https://s3.amazonaws.com/packages.couchbase/diag-logs/orange/201209/10nodes-1708-memcached-killed-after-reb-failed-20120912.tgz
Link to atop of node 13 https://s3.amazonaws.com/packages.couchbase/atop-files/orange/201209/atop-node13-20120912
-
- memcached.log.6.txt.gz
- 12/Sep/12 10:04 PM
- 217 kB
- Thuan Nguyen
-
- memcached.log.7.txt.gz
- 12/Sep/12 10:04 PM
- 27 kB
- Thuan Nguyen
Activity
- All
- Comments
- Work Log
- History
- Activity
- Gerrit Reviews
Show
Chiyoung Seo
added a comment - Alk, please see my comment and advise me why this can happen.
Hide
Karan Kumar
added a comment -
I was able to figure out some more stuff.
The beam.smp segfaulted around this time. Which caused all sorts of query/rebalance failures.
Sep 12 16:57:37 orange-3006 kernel: beam.smp[25844]: segfault at 10000000f ip 000000000043ef50 sp 00007f7139064d48 error 6 in beam.smp[400000+1d6000]
The beam.smp segfaulted around this time. Which caused all sorts of query/rebalance failures.
Sep 12 16:57:37 orange-3006 kernel: beam.smp[25844]: segfault at 10000000f ip 000000000043ef50 sp 00007f7139064d48 error 6 in beam.smp[400000+1d6000]
Show
Karan Kumar
added a comment - I was able to figure out some more stuff.
The beam.smp segfaulted around this time. Which caused all sorts of query/rebalance failures.
Sep 12 16:57:37 orange-3006 kernel: beam.smp[25844]: segfault at 10000000f ip 000000000043ef50 sp 00007f7139064d48 error 6 in beam.smp[400000+1d6000]
Hide
Karan Kumar
added a comment -
Unfortunately, I dont see any core file on the system.
Even though, cores are enabled on the machine.
Even though, cores are enabled on the machine.
Show
Karan Kumar
added a comment - Unfortunately, I dont see any core file on the system.
Even though, cores are enabled on the machine.
Show
Aleksey Kondratenko
added a comment - We'll need core dump here.
Hide
Karan Kumar
added a comment -
We are hitting this again.
But for some reason the cores are not getting generated.
beam.smp[16816]: segfault at 0 ip 00000030af08847b sp 00007f011e38fdd8 error 4 in libc-2.12.so[30af000000+186000]
But for some reason the cores are not getting generated.
beam.smp[16816]: segfault at 0 ip 00000030af08847b sp 00007f011e38fdd8 error 4 in libc-2.12.so[30af000000+186000]
Show
Karan Kumar
added a comment - We are hitting this again.
But for some reason the cores are not getting generated.
beam.smp[16816]: segfault at 0 ip 00000030af08847b sp 00007f011e38fdd8 error 4 in libc-2.12.so[30af000000+186000]
Show
Karan Kumar
added a comment - Increasing this to blocker
Hitting this more often than not
Hide
Karan Kumar
added a comment -
ulimit for all users have now been set to unlimited. and kernel.core_pattern = /data/core.%e.%p on all the nodes.
Show
Karan Kumar
added a comment - ulimit for all users have now been set to unlimited. and kernel.core_pattern = /data/core.%e.%p on all the nodes.
Hide
Karan Kumar
added a comment -
GNU gdb (GDB) Red Hat Enterprise Linux (7.2-50.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp...done.
[New Thread 28408]
[New Thread 28409]
[New Thread 28410]
[New Thread 28412]
[New Thread 28414]
[New Thread 28420]
[New Thread 28979]
[New Thread 28398]
[New Thread 28400]
[New Thread 28425]
[New Thread 28413]
[New Thread 28416]
[New Thread 28417]
[New Thread 28433]
[New Thread 28589]
[New Thread 29574]
[New Thread 28401]
[New Thread 28405]
[New Thread 28406]
[New Thread 28424]
[New Thread 28415]
[New Thread 28419]
[New Thread 29407]
[New Thread 29576]
[New Thread 28411]
[New Thread 28418]
[New Thread 28421]
[New Thread 28403]
[New Thread 28407]
[New Thread 28423]
[New Thread 28422]
Reading symbols from /lib64/libutil.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libutil.so.1
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libncurses.so.5...(no debugging symbols found)...done.
Loaded symbols for /lib64/libncurses.so.5
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libtinfo.so.5...(no debugging symbols found)...done.
Loaded symbols for /lib64/libtinfo.so.5
Reading symbols from /opt/couchbase/lib/erlang/lib/crypto-2.0.4/priv/lib/crypto.so...done.
Loaded symbols for /opt/couchbase/lib/erlang/lib/crypto-2.0.4/priv/lib/crypto.so
Reading symbols from /usr/lib64/libcrypto.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.6
Reading symbols from /usr/lib64/libssl.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl.so.6
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /opt/couchbase/lib/libv8.so...done.
Loaded symbols for /opt/couchbase/lib/libv8.so
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libfreebl3.so...(no debugging symbols found)...done.
Loaded symbols for /lib64/libfreebl3.so
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/ejson-0.1.0/priv/ejson.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/ejson-0.1.0/priv/ejson.so
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/snappy-1.0.4/priv/snappy_nif.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/snappy-1.0.4/priv/snappy_nif.so
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/mapreduce-1.0/priv/mapreduce_nif.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/mapreduce-1.0/priv/mapreduce_nif.so
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_icu_driver.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_icu_driver.so
Reading symbols from /opt/couchbase/lib/libicuuc.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicudata.so.44
Reading symbols from /opt/couchbase/lib/libicui18n.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicui18n.so.44
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_ejson_compare.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_ejson_compare.so
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/couch_view_parser-1.0/priv/couch_view_parser_nif.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/couch_view_parser-1.0/priv/couch_view_parser_nif.so
Core was generated by `/opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp -A 16 -sbt u -P 327680 -K tru'.
Program terminated with signal 11, Segmentation fault.
#0 0x000000000043ec6b in tree_delete (allctr=<value optimized out>, del=0x12c6a80, flags=<value optimized out>) at beam/erl_bestfit_alloc.c:435
435 beam/erl_bestfit_alloc.c: No such file or directory.
in beam/erl_bestfit_alloc.c
Missing separate debuginfos, use: debuginfo-install couchbase-server-2.0.0-1717.x86_64
(gdb) t a a bt
Thread 31 (Thread 0x7f4afea01700 (LWP 28422)):
#0 0x000000397260804d in pthread_join () from /lib64/libpthread.so.0
#1 0x00007f4af6371887 in v8::internal::ContextSwitcher::StopPreemption () at src/v8threads.cc:445
#2 0x00007f4af424b7a3 in destroyContext (ctx=0x7f4aac7b4f18) at mapreduce.cc:354
#3 0x00000000005138fb in nif_resource_dtor (bin=<value optimized out>) at beam/erl_nif.c:1217
#4 0x000000000048904b in erts_bin_free (offheap=<value optimized out>) at beam/erl_binary.h:303
#5 erts_cleanup_offheap (offheap=<value optimized out>) at beam/erl_message.c:174
#6 0x0000000000498a81 in delete_process (p=0x7f4aa286d568, pix_lock=<value optimized out>) at beam/erl_process.c:6621
#7 continue_exit_process (p=0x7f4aa286d568, pix_lock=<value optimized out>) at beam/erl_process.c:7663
#8 0x000000000051db94 in terminate_proc (c_p=0x7f4aa286d568, pc=<value optimized out>, reg=<value optimized out>, bf=<value optimized out>) at beam/beam_emu.c:5633
#9 handle_error (c_p=0x7f4aa286d568, pc=<value optimized out>, reg=<value optimized out>, bf=<value optimized out>) at beam/beam_emu.c:5508
#10 0x0000000000520cdf in process_main () at beam/beam_emu.c:3273
#11 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b000eafc0) at beam/erl_process.c:3782
#12 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200a0) at pthread/ethread.c:106
#13 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#14 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 30 (Thread 0x7f4afe000700 (LWP 28423)):
#0 0x0000000000522f94 in process_main () at x86_64-unknown-linux-gnu/opt/smp/beam_hot.h:919
#1 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b000f9080) at beam/erl_process.c:3782
#2 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200a0) at pthread/ethread.c:106
#3 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#4 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7f4afec23700 (LWP 28407)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8138) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8138) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8138) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7f4affe69700 (LWP 28403)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x0000000000462ad9 in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:697
#3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:472
#4 sys_msg_dispatcher_func (unused=<value optimized out>) at beam/erl_trace.c:3238
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200d0) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7f4afea47700 (LWP 28421)):
#0 0x000000397260f03d in waitpid () from /lib64/libpthread.so.0
#1 0x000000000053906f in child_waiter (unused=<value optimized out>) at sys/unix/sys.c:2821
#2 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620080) at pthread/ethread.c:106
#3 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#4 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7f4afeaad700 (LWP 28418)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8660) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8660) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8660) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7f4afeb9b700 (LWP 28411)):
---Type <return> to continue, or q <return> to quit---
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8318) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8318) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8318) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7f4aaed44700 (LWP 29576)):
#0 0x0000003971eab15d in nanosleep () from /lib64/libc.so.6
#1 0x0000003971edf124 in usleep () from /lib64/libc.so.6
#2 0x00007f4af6371848 in v8::internal::ContextSwitcher::Run (this=0x7f4af814d8d0) at src/v8threads.cc:457
#3 0x00007f4af62f1580 in v8::internal::ThreadEntry (arg=0x7f4af814d8d0) at src/platform-linux.cc:745
#4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f4a94844700 (LWP 29407)):
#0 0x0000003971eab15d in nanosleep () from /lib64/libc.so.6
#1 0x0000003971edf124 in usleep () from /lib64/libc.so.6
#2 0x00007f4af6371848 in v8::internal::ContextSwitcher::Run (this=0x7f4ab41b09d0) at src/v8threads.cc:457
#3 0x00007f4af62f1580 in v8::internal::ThreadEntry (arg=0x7f4ab41b09d0) at src/platform-linux.cc:745
#4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f4afea8b700 (LWP 28419)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d86d8) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d86d8) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d86d8) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f4afeb13700 (LWP 28415)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d84f8) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d84f8) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d84f8) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f4afd5ff700 (LWP 28424)):
#0 do_erts_alcu_free (type=<value optimized out>, unused=<value optimized out>, p=0x7f4aaf0b8608) at beam/erl_alloc_util.c:2894
#1 erts_alcu_free_thr_pref (type=<value optimized out>, unused=<value optimized out>, p=0x7f4aaf0b8608) at beam/erl_alloc_util.c:2949
#2 0x0000000000489022 in erts_free (offheap=<value optimized out>) at beam/erl_alloc.h:226
#3 erts_bin_free (offheap=<value optimized out>) at beam/erl_binary.h:307
#4 erts_cleanup_offheap (offheap=<value optimized out>) at beam/erl_message.c:174
#5 0x0000000000498a81 in delete_process (p=0x7f4aac311310, pix_lock=<value optimized out>) at beam/erl_process.c:6621
#6 continue_exit_process (p=0x7f4aac311310, pix_lock=<value optimized out>) at beam/erl_process.c:7663
#7 0x000000000051db94 in terminate_proc (c_p=0x7f4aac311310, pc=<value optimized out>, reg=<value optimized out>, bf=<value optimized out>) at beam/beam_emu.c:5633
#8 handle_error (c_p=0x7f4aac311310, pc=<value optimized out>, reg=<value optimized out>, bf=<value optimized out>) at beam/beam_emu.c:5508
#9 0x0000000000520cdf in process_main () at beam/beam_emu.c:3273
#10 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b00107140) at beam/erl_process.c:3782
#11 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200a0) at pthread/ethread.c:106
#12 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#13 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f4afec45700 (LWP 28406)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d80c0) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d80c0) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d80c0) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f4afec67700 (LWP 28405)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8048) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8048) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8048) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f4b0017c700 (LWP 28401)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x000000000046a1cf in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:697
#3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:472
#4 emergency_watchdog (unused=<value optimized out>) at beam/utils.c:3945
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200d0) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f4a93e43700 (LWP 29574)):
#0 0x0000003971eab15d in nanosleep () from /lib64/libc.so.6
#1 0x0000003971edf124 in usleep () from /lib64/libc.so.6
#2 0x00007f4af6371848 in v8::internal::ContextSwitcher::Run (this=0x7f4aa8091b00) at src/v8threads.cc:457
#3 0x00007f4af62f1580 in v8::internal::ThreadEntry (arg=0x7f4aa8091b00) at src/platform-linux.cc:745
#4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f4aaed55700 (LWP 28589)):
#0 0x000000397260d6c0 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f4af62f118d in v8::internal::LinuxSemaphore::Wait (this=0x7f4aa8009b10) at src/platform-linux.cc:869
#2 0x00007f4af630845c in v8::internal::RuntimeProfiler::WaitForSomeIsolateToEnterJS () at src/runtime-profiler.cc:386
#3 0x00007f4af62f2136 in v8::internal::SignalSender::Run (this=0x7f4aa8009470) at src/platform-linux.cc:1126
#4 0x00007f4af62f1580 in v8::internal::ThreadEntry (arg=0x7f4aa8009470) at src/platform-linux.cc:745
#5 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#6 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f4aaffff700 (LWP 28433)):
#0 0x0000003971eab15d in nanosleep () from /lib64/libc.so.6
#1 0x0000003971edf124 in usleep () from /lib64/libc.so.6
#2 0x00007f4af4249397 in terminatorLoop (args=<value optimized out>) at mapreduce_nif.cc:445
#3 0x0000000000584d89 in thr_wrapper (vtwd=0x7f4afea00ae0) at pthread/ethread.c:106
#4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f4afeacf700 (LWP 28417)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d85e8) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d85e8) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d85e8) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
Thread 12 (Thread 0x7f4afeaf1700 (LWP 28416)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8570) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8570) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8570) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f4afeb57700 (LWP 28413)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8408) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8408) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8408) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f4afcbfe700 (LWP 28425)):
#0 erts_smp_proc_raw_trylock__ (c_p=0x7f4aac2e15d0, ref=139958707146706) at beam/erl_process_lock.h:422
#1 erts_smp_proc_lock__ (c_p=0x7f4aac2e15d0, ref=139958707146706) at beam/erl_process_lock.h:455
#2 erts_smp_proc_lock (c_p=0x7f4aac2e15d0, ref=139958707146706) at beam/erl_process_lock.h:672
#3 demonitor (c_p=0x7f4aac2e15d0, ref=139958707146706) at beam/bif.c:377
#4 0x0000000000477c30 in demonitor_2 (A__p=0x7f4aac2e15d0, A_1=139958707146706, A_2=<value optimized out>) at beam/bif.c:505
#5 0x0000000000521927 in process_main () at beam/beam_emu.c:2542
#6 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b00115200) at beam/erl_process.c:3782
#7 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200a0) at pthread/ethread.c:106
#8 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#9 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f4b00b7d700 (LWP 28400)):
#0 0x000000397260e4ed in read () from /lib64/libpthread.so.0
#1 0x000000000053810e in signal_dispatcher_thread_func (unused=<value optimized out>) at sys/unix/sys.c:2906
#2 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff3161ffc0) at pthread/ethread.c:106
#3 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#4 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f4b00ea9700 (LWP 28398)):
#0 0x0000003971ede8b3 in select () from /lib64/libc.so.6
#1 0x0000000000537ba0 in erts_sys_main_thread () at sys/unix/sys.c:3049
#2 0x0000000000444761 in erl_start (argc=54, argv=<value optimized out>) at beam/erl_init.c:1441
#3 0x0000000000426219 in main (argc=<value optimized out>, argv=<value optimized out>) at sys/unix/erl_main.c:29
Thread 7 (Thread 0x7f4aae343700 (LWP 28979)):
#0 0x0000003971eab15d in nanosleep () from /lib64/libc.so.6
#1 0x0000003971edf124 in usleep () from /lib64/libc.so.6
#2 0x00007f4af6371848 in v8::internal::ContextSwitcher::Run (this=0x7f4aa4030140) at src/v8threads.cc:457
#3 0x00007f4af62f1580 in v8::internal::ThreadEntry (arg=0x7f4aa4030140) at src/platform-linux.cc:745
#4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f4afea69700 (LWP 28420)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8750) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8750) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8750) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
Thread 5 (Thread 0x7f4afeb35700 (LWP 28414)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8480) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8480) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8480) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f4afeb79700 (LWP 28412)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8390) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8390) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8390) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f4afebbd700 (LWP 28410)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d82a0) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d82a0) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d82a0) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f4afebdf700 (LWP 28409)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8228) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8228) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8228) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f4afec01700 (LWP 28408)):
#0 0x000000000043ec6b in tree_delete (allctr=<value optimized out>, del=0x12c6a80, flags=<value optimized out>) at beam/erl_bestfit_alloc.c:435
#1 0x0000000000437091 in mbc_free (allctr=0x12a86c0, p=<value optimized out>) at beam/erl_alloc_util.c:867
#2 0x000000000043833f in do_erts_alcu_free (type=<value optimized out>, unused=<value optimized out>, p=0x12c6bb8) at beam/erl_alloc_util.c:2898
#3 erts_alcu_free_thr_pref (type=<value optimized out>, unused=<value optimized out>, p=0x12c6bb8) at beam/erl_alloc_util.c:2949
#4 0x000000000054ca48 in file_async_ready (e=0x12c3230, data=0x12c6bb8) at drivers/common/efile_drv.c:2117
#5 0x0000000000480094 in async_ready (p=0x7f4afeca4e90, data=<value optimized out>) at beam/io.c:2445
#6 0x00000000004e906f in async_main (arg=0x7f4aff3d81b0) at beam/erl_async.c:296
#7 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#8 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#9 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp...done.
[New Thread 28408]
[New Thread 28409]
[New Thread 28410]
[New Thread 28412]
[New Thread 28414]
[New Thread 28420]
[New Thread 28979]
[New Thread 28398]
[New Thread 28400]
[New Thread 28425]
[New Thread 28413]
[New Thread 28416]
[New Thread 28417]
[New Thread 28433]
[New Thread 28589]
[New Thread 29574]
[New Thread 28401]
[New Thread 28405]
[New Thread 28406]
[New Thread 28424]
[New Thread 28415]
[New Thread 28419]
[New Thread 29407]
[New Thread 29576]
[New Thread 28411]
[New Thread 28418]
[New Thread 28421]
[New Thread 28403]
[New Thread 28407]
[New Thread 28423]
[New Thread 28422]
Reading symbols from /lib64/libutil.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libutil.so.1
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libncurses.so.5...(no debugging symbols found)...done.
Loaded symbols for /lib64/libncurses.so.5
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libtinfo.so.5...(no debugging symbols found)...done.
Loaded symbols for /lib64/libtinfo.so.5
Reading symbols from /opt/couchbase/lib/erlang/lib/crypto-2.0.4/priv/lib/crypto.so...done.
Loaded symbols for /opt/couchbase/lib/erlang/lib/crypto-2.0.4/priv/lib/crypto.so
Reading symbols from /usr/lib64/libcrypto.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.6
Reading symbols from /usr/lib64/libssl.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl.so.6
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /opt/couchbase/lib/libv8.so...done.
Loaded symbols for /opt/couchbase/lib/libv8.so
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libfreebl3.so...(no debugging symbols found)...done.
Loaded symbols for /lib64/libfreebl3.so
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/ejson-0.1.0/priv/ejson.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/ejson-0.1.0/priv/ejson.so
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/snappy-1.0.4/priv/snappy_nif.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/snappy-1.0.4/priv/snappy_nif.so
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/mapreduce-1.0/priv/mapreduce_nif.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/mapreduce-1.0/priv/mapreduce_nif.so
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_icu_driver.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_icu_driver.so
Reading symbols from /opt/couchbase/lib/libicuuc.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicudata.so.44
Reading symbols from /opt/couchbase/lib/libicui18n.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicui18n.so.44
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_ejson_compare.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_ejson_compare.so
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/couch_view_parser-1.0/priv/couch_view_parser_nif.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/couch_view_parser-1.0/priv/couch_view_parser_nif.so
Core was generated by `/opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp -A 16 -sbt u -P 327680 -K tru'.
Program terminated with signal 11, Segmentation fault.
#0 0x000000000043ec6b in tree_delete (allctr=<value optimized out>, del=0x12c6a80, flags=<value optimized out>) at beam/erl_bestfit_alloc.c:435
435 beam/erl_bestfit_alloc.c: No such file or directory.
in beam/erl_bestfit_alloc.c
Missing separate debuginfos, use: debuginfo-install couchbase-server-2.0.0-1717.x86_64
(gdb) t a a bt
Thread 31 (Thread 0x7f4afea01700 (LWP 28422)):
#0 0x000000397260804d in pthread_join () from /lib64/libpthread.so.0
#1 0x00007f4af6371887 in v8::internal::ContextSwitcher::StopPreemption () at src/v8threads.cc:445
#2 0x00007f4af424b7a3 in destroyContext (ctx=0x7f4aac7b4f18) at mapreduce.cc:354
#3 0x00000000005138fb in nif_resource_dtor (bin=<value optimized out>) at beam/erl_nif.c:1217
#4 0x000000000048904b in erts_bin_free (offheap=<value optimized out>) at beam/erl_binary.h:303
#5 erts_cleanup_offheap (offheap=<value optimized out>) at beam/erl_message.c:174
#6 0x0000000000498a81 in delete_process (p=0x7f4aa286d568, pix_lock=<value optimized out>) at beam/erl_process.c:6621
#7 continue_exit_process (p=0x7f4aa286d568, pix_lock=<value optimized out>) at beam/erl_process.c:7663
#8 0x000000000051db94 in terminate_proc (c_p=0x7f4aa286d568, pc=<value optimized out>, reg=<value optimized out>, bf=<value optimized out>) at beam/beam_emu.c:5633
#9 handle_error (c_p=0x7f4aa286d568, pc=<value optimized out>, reg=<value optimized out>, bf=<value optimized out>) at beam/beam_emu.c:5508
#10 0x0000000000520cdf in process_main () at beam/beam_emu.c:3273
#11 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b000eafc0) at beam/erl_process.c:3782
#12 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200a0) at pthread/ethread.c:106
#13 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#14 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 30 (Thread 0x7f4afe000700 (LWP 28423)):
#0 0x0000000000522f94 in process_main () at x86_64-unknown-linux-gnu/opt/smp/beam_hot.h:919
#1 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b000f9080) at beam/erl_process.c:3782
#2 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200a0) at pthread/ethread.c:106
#3 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#4 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7f4afec23700 (LWP 28407)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8138) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8138) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8138) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7f4affe69700 (LWP 28403)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x0000000000462ad9 in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:697
#3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:472
#4 sys_msg_dispatcher_func (unused=<value optimized out>) at beam/erl_trace.c:3238
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200d0) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7f4afea47700 (LWP 28421)):
#0 0x000000397260f03d in waitpid () from /lib64/libpthread.so.0
#1 0x000000000053906f in child_waiter (unused=<value optimized out>) at sys/unix/sys.c:2821
#2 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620080) at pthread/ethread.c:106
#3 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#4 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7f4afeaad700 (LWP 28418)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8660) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8660) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8660) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7f4afeb9b700 (LWP 28411)):
---Type <return> to continue, or q <return> to quit---
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8318) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8318) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8318) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7f4aaed44700 (LWP 29576)):
#0 0x0000003971eab15d in nanosleep () from /lib64/libc.so.6
#1 0x0000003971edf124 in usleep () from /lib64/libc.so.6
#2 0x00007f4af6371848 in v8::internal::ContextSwitcher::Run (this=0x7f4af814d8d0) at src/v8threads.cc:457
#3 0x00007f4af62f1580 in v8::internal::ThreadEntry (arg=0x7f4af814d8d0) at src/platform-linux.cc:745
#4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f4a94844700 (LWP 29407)):
#0 0x0000003971eab15d in nanosleep () from /lib64/libc.so.6
#1 0x0000003971edf124 in usleep () from /lib64/libc.so.6
#2 0x00007f4af6371848 in v8::internal::ContextSwitcher::Run (this=0x7f4ab41b09d0) at src/v8threads.cc:457
#3 0x00007f4af62f1580 in v8::internal::ThreadEntry (arg=0x7f4ab41b09d0) at src/platform-linux.cc:745
#4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f4afea8b700 (LWP 28419)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d86d8) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d86d8) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d86d8) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f4afeb13700 (LWP 28415)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d84f8) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d84f8) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d84f8) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f4afd5ff700 (LWP 28424)):
#0 do_erts_alcu_free (type=<value optimized out>, unused=<value optimized out>, p=0x7f4aaf0b8608) at beam/erl_alloc_util.c:2894
#1 erts_alcu_free_thr_pref (type=<value optimized out>, unused=<value optimized out>, p=0x7f4aaf0b8608) at beam/erl_alloc_util.c:2949
#2 0x0000000000489022 in erts_free (offheap=<value optimized out>) at beam/erl_alloc.h:226
#3 erts_bin_free (offheap=<value optimized out>) at beam/erl_binary.h:307
#4 erts_cleanup_offheap (offheap=<value optimized out>) at beam/erl_message.c:174
#5 0x0000000000498a81 in delete_process (p=0x7f4aac311310, pix_lock=<value optimized out>) at beam/erl_process.c:6621
#6 continue_exit_process (p=0x7f4aac311310, pix_lock=<value optimized out>) at beam/erl_process.c:7663
#7 0x000000000051db94 in terminate_proc (c_p=0x7f4aac311310, pc=<value optimized out>, reg=<value optimized out>, bf=<value optimized out>) at beam/beam_emu.c:5633
#8 handle_error (c_p=0x7f4aac311310, pc=<value optimized out>, reg=<value optimized out>, bf=<value optimized out>) at beam/beam_emu.c:5508
#9 0x0000000000520cdf in process_main () at beam/beam_emu.c:3273
#10 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b00107140) at beam/erl_process.c:3782
#11 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200a0) at pthread/ethread.c:106
#12 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#13 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f4afec45700 (LWP 28406)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d80c0) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d80c0) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d80c0) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f4afec67700 (LWP 28405)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8048) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8048) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8048) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f4b0017c700 (LWP 28401)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x000000000046a1cf in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:697
#3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:472
#4 emergency_watchdog (unused=<value optimized out>) at beam/utils.c:3945
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200d0) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f4a93e43700 (LWP 29574)):
#0 0x0000003971eab15d in nanosleep () from /lib64/libc.so.6
#1 0x0000003971edf124 in usleep () from /lib64/libc.so.6
#2 0x00007f4af6371848 in v8::internal::ContextSwitcher::Run (this=0x7f4aa8091b00) at src/v8threads.cc:457
#3 0x00007f4af62f1580 in v8::internal::ThreadEntry (arg=0x7f4aa8091b00) at src/platform-linux.cc:745
#4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f4aaed55700 (LWP 28589)):
#0 0x000000397260d6c0 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f4af62f118d in v8::internal::LinuxSemaphore::Wait (this=0x7f4aa8009b10) at src/platform-linux.cc:869
#2 0x00007f4af630845c in v8::internal::RuntimeProfiler::WaitForSomeIsolateToEnterJS () at src/runtime-profiler.cc:386
#3 0x00007f4af62f2136 in v8::internal::SignalSender::Run (this=0x7f4aa8009470) at src/platform-linux.cc:1126
#4 0x00007f4af62f1580 in v8::internal::ThreadEntry (arg=0x7f4aa8009470) at src/platform-linux.cc:745
#5 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#6 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f4aaffff700 (LWP 28433)):
#0 0x0000003971eab15d in nanosleep () from /lib64/libc.so.6
#1 0x0000003971edf124 in usleep () from /lib64/libc.so.6
#2 0x00007f4af4249397 in terminatorLoop (args=<value optimized out>) at mapreduce_nif.cc:445
#3 0x0000000000584d89 in thr_wrapper (vtwd=0x7f4afea00ae0) at pthread/ethread.c:106
#4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f4afeacf700 (LWP 28417)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d85e8) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d85e8) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d85e8) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
Thread 12 (Thread 0x7f4afeaf1700 (LWP 28416)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8570) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8570) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8570) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f4afeb57700 (LWP 28413)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8408) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8408) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8408) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f4afcbfe700 (LWP 28425)):
#0 erts_smp_proc_raw_trylock__ (c_p=0x7f4aac2e15d0, ref=139958707146706) at beam/erl_process_lock.h:422
#1 erts_smp_proc_lock__ (c_p=0x7f4aac2e15d0, ref=139958707146706) at beam/erl_process_lock.h:455
#2 erts_smp_proc_lock (c_p=0x7f4aac2e15d0, ref=139958707146706) at beam/erl_process_lock.h:672
#3 demonitor (c_p=0x7f4aac2e15d0, ref=139958707146706) at beam/bif.c:377
#4 0x0000000000477c30 in demonitor_2 (A__p=0x7f4aac2e15d0, A_1=139958707146706, A_2=<value optimized out>) at beam/bif.c:505
#5 0x0000000000521927 in process_main () at beam/beam_emu.c:2542
#6 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b00115200) at beam/erl_process.c:3782
#7 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200a0) at pthread/ethread.c:106
#8 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#9 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f4b00b7d700 (LWP 28400)):
#0 0x000000397260e4ed in read () from /lib64/libpthread.so.0
#1 0x000000000053810e in signal_dispatcher_thread_func (unused=<value optimized out>) at sys/unix/sys.c:2906
#2 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff3161ffc0) at pthread/ethread.c:106
#3 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#4 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f4b00ea9700 (LWP 28398)):
#0 0x0000003971ede8b3 in select () from /lib64/libc.so.6
#1 0x0000000000537ba0 in erts_sys_main_thread () at sys/unix/sys.c:3049
#2 0x0000000000444761 in erl_start (argc=54, argv=<value optimized out>) at beam/erl_init.c:1441
#3 0x0000000000426219 in main (argc=<value optimized out>, argv=<value optimized out>) at sys/unix/erl_main.c:29
Thread 7 (Thread 0x7f4aae343700 (LWP 28979)):
#0 0x0000003971eab15d in nanosleep () from /lib64/libc.so.6
#1 0x0000003971edf124 in usleep () from /lib64/libc.so.6
#2 0x00007f4af6371848 in v8::internal::ContextSwitcher::Run (this=0x7f4aa4030140) at src/v8threads.cc:457
#3 0x00007f4af62f1580 in v8::internal::ThreadEntry (arg=0x7f4aa4030140) at src/platform-linux.cc:745
#4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f4afea69700 (LWP 28420)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8750) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8750) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8750) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
Thread 5 (Thread 0x7f4afeb35700 (LWP 28414)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8480) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8480) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8480) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f4afeb79700 (LWP 28412)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8390) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8390) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8390) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f4afebbd700 (LWP 28410)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d82a0) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d82a0) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d82a0) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f4afebdf700 (LWP 28409)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8228) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8228) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8228) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f4afec01700 (LWP 28408)):
#0 0x000000000043ec6b in tree_delete (allctr=<value optimized out>, del=0x12c6a80, flags=<value optimized out>) at beam/erl_bestfit_alloc.c:435
#1 0x0000000000437091 in mbc_free (allctr=0x12a86c0, p=<value optimized out>) at beam/erl_alloc_util.c:867
#2 0x000000000043833f in do_erts_alcu_free (type=<value optimized out>, unused=<value optimized out>, p=0x12c6bb8) at beam/erl_alloc_util.c:2898
#3 erts_alcu_free_thr_pref (type=<value optimized out>, unused=<value optimized out>, p=0x12c6bb8) at beam/erl_alloc_util.c:2949
#4 0x000000000054ca48 in file_async_ready (e=0x12c3230, data=0x12c6bb8) at drivers/common/efile_drv.c:2117
#5 0x0000000000480094 in async_ready (p=0x7f4afeca4e90, data=<value optimized out>) at beam/io.c:2445
#6 0x00000000004e906f in async_main (arg=0x7f4aff3d81b0) at beam/erl_async.c:296
#7 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#8 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#9 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Show
Karan Kumar
added a comment - GNU gdb (GDB) Red Hat Enterprise Linux (7.2-50.el6)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later < http://gnu.org/licenses/gpl.html >
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
< http://www.gnu.org/software/gdb/bugs/ >...
Reading symbols from /opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp...done.
[New Thread 28408]
[New Thread 28409]
[New Thread 28410]
[New Thread 28412]
[New Thread 28414]
[New Thread 28420]
[New Thread 28979]
[New Thread 28398]
[New Thread 28400]
[New Thread 28425]
[New Thread 28413]
[New Thread 28416]
[New Thread 28417]
[New Thread 28433]
[New Thread 28589]
[New Thread 29574]
[New Thread 28401]
[New Thread 28405]
[New Thread 28406]
[New Thread 28424]
[New Thread 28415]
[New Thread 28419]
[New Thread 29407]
[New Thread 29576]
[New Thread 28411]
[New Thread 28418]
[New Thread 28421]
[New Thread 28403]
[New Thread 28407]
[New Thread 28423]
[New Thread 28422]
Reading symbols from /lib64/libutil.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libutil.so.1
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libncurses.so.5...(no debugging symbols found)...done.
Loaded symbols for /lib64/libncurses.so.5
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libtinfo.so.5...(no debugging symbols found)...done.
Loaded symbols for /lib64/libtinfo.so.5
Reading symbols from /opt/couchbase/lib/erlang/lib/crypto-2.0.4/priv/lib/crypto.so...done.
Loaded symbols for /opt/couchbase/lib/erlang/lib/crypto-2.0.4/priv/lib/crypto.so
Reading symbols from /usr/lib64/libcrypto.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libcrypto.so.6
Reading symbols from /usr/lib64/libssl.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libssl.so.6
Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libz.so.1
Reading symbols from /lib64/libgssapi_krb5.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgssapi_krb5.so.2
Reading symbols from /lib64/libkrb5.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5.so.3
Reading symbols from /lib64/libcom_err.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcom_err.so.2
Reading symbols from /lib64/libk5crypto.so.3...(no debugging symbols found)...done.
Loaded symbols for /lib64/libk5crypto.so.3
Reading symbols from /lib64/libkrb5support.so.0...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkrb5support.so.0
Reading symbols from /lib64/libkeyutils.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libkeyutils.so.1
Reading symbols from /lib64/libresolv.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libselinux.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libselinux.so.1
Reading symbols from /opt/couchbase/lib/libv8.so...done.
Loaded symbols for /opt/couchbase/lib/libv8.so
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /usr/lib64/libstdc++.so.6...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libstdc++.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/libfreebl3.so...(no debugging symbols found)...done.
Loaded symbols for /lib64/libfreebl3.so
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/ejson-0.1.0/priv/ejson.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/ejson-0.1.0/priv/ejson.so
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/snappy-1.0.4/priv/snappy_nif.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/snappy-1.0.4/priv/snappy_nif.so
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/mapreduce-1.0/priv/mapreduce_nif.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/mapreduce-1.0/priv/mapreduce_nif.so
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_icu_driver.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_icu_driver.so
Reading symbols from /opt/couchbase/lib/libicuuc.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicuuc.so.44
Reading symbols from /opt/couchbase/lib/libicudata.so.44...(no debugging symbols found)...done.
Loaded symbols for /opt/couchbase/lib/libicudata.so.44
Reading symbols from /opt/couchbase/lib/libicui18n.so.44...done.
Loaded symbols for /opt/couchbase/lib/libicui18n.so.44
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_ejson_compare.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/couch-1.2.0a-5282953-git/priv/lib/couch_ejson_compare.so
Reading symbols from /opt/couchbase/lib/couchdb/erlang/lib/couch_view_parser-1.0/priv/couch_view_parser_nif.so...done.
Loaded symbols for /opt/couchbase/lib/couchdb/erlang/lib/couch_view_parser-1.0/priv/couch_view_parser_nif.so
Core was generated by `/opt/couchbase/lib/erlang/erts-5.8.5/bin/beam.smp -A 16 -sbt u -P 327680 -K tru'.
Program terminated with signal 11, Segmentation fault.
#0 0x000000000043ec6b in tree_delete (allctr=<value optimized out>, del=0x12c6a80, flags=<value optimized out>) at beam/erl_bestfit_alloc.c:435
435 beam/erl_bestfit_alloc.c: No such file or directory.
in beam/erl_bestfit_alloc.c
Missing separate debuginfos, use: debuginfo-install couchbase-server-2.0.0-1717.x86_64
(gdb) t a a bt
Thread 31 (Thread 0x7f4afea01700 (LWP 28422)):
#0 0x000000397260804d in pthread_join () from /lib64/libpthread.so.0
#1 0x00007f4af6371887 in v8::internal::ContextSwitcher::StopPreemption () at src/v8threads.cc:445
#2 0x00007f4af424b7a3 in destroyContext (ctx=0x7f4aac7b4f18) at mapreduce.cc:354
#3 0x00000000005138fb in nif_resource_dtor (bin=<value optimized out>) at beam/erl_nif.c:1217
#4 0x000000000048904b in erts_bin_free (offheap=<value optimized out>) at beam/erl_binary.h:303
#5 erts_cleanup_offheap (offheap=<value optimized out>) at beam/erl_message.c:174
#6 0x0000000000498a81 in delete_process (p=0x7f4aa286d568, pix_lock=<value optimized out>) at beam/erl_process.c:6621
#7 continue_exit_process (p=0x7f4aa286d568, pix_lock=<value optimized out>) at beam/erl_process.c:7663
#8 0x000000000051db94 in terminate_proc (c_p=0x7f4aa286d568, pc=<value optimized out>, reg=<value optimized out>, bf=<value optimized out>) at beam/beam_emu.c:5633
#9 handle_error (c_p=0x7f4aa286d568, pc=<value optimized out>, reg=<value optimized out>, bf=<value optimized out>) at beam/beam_emu.c:5508
#10 0x0000000000520cdf in process_main () at beam/beam_emu.c:3273
#11 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b000eafc0) at beam/erl_process.c:3782
#12 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200a0) at pthread/ethread.c:106
#13 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#14 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 30 (Thread 0x7f4afe000700 (LWP 28423)):
#0 0x0000000000522f94 in process_main () at x86_64-unknown-linux-gnu/opt/smp/beam_hot.h:919
#1 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b000f9080) at beam/erl_process.c:3782
#2 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200a0) at pthread/ethread.c:106
#3 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#4 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 29 (Thread 0x7f4afec23700 (LWP 28407)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8138) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8138) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8138) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 28 (Thread 0x7f4affe69700 (LWP 28403)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x0000000000462ad9 in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:697
#3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:472
#4 sys_msg_dispatcher_func (unused=<value optimized out>) at beam/erl_trace.c:3238
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200d0) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7f4afea47700 (LWP 28421)):
#0 0x000000397260f03d in waitpid () from /lib64/libpthread.so.0
#1 0x000000000053906f in child_waiter (unused=<value optimized out>) at sys/unix/sys.c:2821
#2 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620080) at pthread/ethread.c:106
#3 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#4 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7f4afeaad700 (LWP 28418)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8660) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8660) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8660) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7f4afeb9b700 (LWP 28411)):
---Type <return> to continue, or q <return> to quit---
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8318) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8318) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8318) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7f4aaed44700 (LWP 29576)):
#0 0x0000003971eab15d in nanosleep () from /lib64/libc.so.6
#1 0x0000003971edf124 in usleep () from /lib64/libc.so.6
#2 0x00007f4af6371848 in v8::internal::ContextSwitcher::Run (this=0x7f4af814d8d0) at src/v8threads.cc:457
#3 0x00007f4af62f1580 in v8::internal::ThreadEntry (arg=0x7f4af814d8d0) at src/platform-linux.cc:745
#4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f4a94844700 (LWP 29407)):
#0 0x0000003971eab15d in nanosleep () from /lib64/libc.so.6
#1 0x0000003971edf124 in usleep () from /lib64/libc.so.6
#2 0x00007f4af6371848 in v8::internal::ContextSwitcher::Run (this=0x7f4ab41b09d0) at src/v8threads.cc:457
#3 0x00007f4af62f1580 in v8::internal::ThreadEntry (arg=0x7f4ab41b09d0) at src/platform-linux.cc:745
#4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f4afea8b700 (LWP 28419)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d86d8) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d86d8) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d86d8) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f4afeb13700 (LWP 28415)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d84f8) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d84f8) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d84f8) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f4afd5ff700 (LWP 28424)):
#0 do_erts_alcu_free (type=<value optimized out>, unused=<value optimized out>, p=0x7f4aaf0b8608) at beam/erl_alloc_util.c:2894
#1 erts_alcu_free_thr_pref (type=<value optimized out>, unused=<value optimized out>, p=0x7f4aaf0b8608) at beam/erl_alloc_util.c:2949
#2 0x0000000000489022 in erts_free (offheap=<value optimized out>) at beam/erl_alloc.h:226
#3 erts_bin_free (offheap=<value optimized out>) at beam/erl_binary.h:307
#4 erts_cleanup_offheap (offheap=<value optimized out>) at beam/erl_message.c:174
#5 0x0000000000498a81 in delete_process (p=0x7f4aac311310, pix_lock=<value optimized out>) at beam/erl_process.c:6621
#6 continue_exit_process (p=0x7f4aac311310, pix_lock=<value optimized out>) at beam/erl_process.c:7663
#7 0x000000000051db94 in terminate_proc (c_p=0x7f4aac311310, pc=<value optimized out>, reg=<value optimized out>, bf=<value optimized out>) at beam/beam_emu.c:5633
#8 handle_error (c_p=0x7f4aac311310, pc=<value optimized out>, reg=<value optimized out>, bf=<value optimized out>) at beam/beam_emu.c:5508
#9 0x0000000000520cdf in process_main () at beam/beam_emu.c:3273
#10 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b00107140) at beam/erl_process.c:3782
#11 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200a0) at pthread/ethread.c:106
#12 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#13 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f4afec45700 (LWP 28406)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d80c0) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d80c0) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d80c0) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f4afec67700 (LWP 28405)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8048) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8048) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8048) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f4b0017c700 (LWP 28401)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x000000000046a1cf in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:697
#3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:472
#4 emergency_watchdog (unused=<value optimized out>) at beam/utils.c:3945
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200d0) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f4a93e43700 (LWP 29574)):
#0 0x0000003971eab15d in nanosleep () from /lib64/libc.so.6
#1 0x0000003971edf124 in usleep () from /lib64/libc.so.6
#2 0x00007f4af6371848 in v8::internal::ContextSwitcher::Run (this=0x7f4aa8091b00) at src/v8threads.cc:457
#3 0x00007f4af62f1580 in v8::internal::ThreadEntry (arg=0x7f4aa8091b00) at src/platform-linux.cc:745
#4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f4aaed55700 (LWP 28589)):
#0 0x000000397260d6c0 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f4af62f118d in v8::internal::LinuxSemaphore::Wait (this=0x7f4aa8009b10) at src/platform-linux.cc:869
#2 0x00007f4af630845c in v8::internal::RuntimeProfiler::WaitForSomeIsolateToEnterJS () at src/runtime-profiler.cc:386
#3 0x00007f4af62f2136 in v8::internal::SignalSender::Run (this=0x7f4aa8009470) at src/platform-linux.cc:1126
#4 0x00007f4af62f1580 in v8::internal::ThreadEntry (arg=0x7f4aa8009470) at src/platform-linux.cc:745
#5 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#6 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f4aaffff700 (LWP 28433)):
#0 0x0000003971eab15d in nanosleep () from /lib64/libc.so.6
#1 0x0000003971edf124 in usleep () from /lib64/libc.so.6
#2 0x00007f4af4249397 in terminatorLoop (args=<value optimized out>) at mapreduce_nif.cc:445
#3 0x0000000000584d89 in thr_wrapper (vtwd=0x7f4afea00ae0) at pthread/ethread.c:106
#4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f4afeacf700 (LWP 28417)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d85e8) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d85e8) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d85e8) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
Thread 12 (Thread 0x7f4afeaf1700 (LWP 28416)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8570) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8570) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8570) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f4afeb57700 (LWP 28413)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8408) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8408) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8408) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f4afcbfe700 (LWP 28425)):
#0 erts_smp_proc_raw_trylock__ (c_p=0x7f4aac2e15d0, ref=139958707146706) at beam/erl_process_lock.h:422
#1 erts_smp_proc_lock__ (c_p=0x7f4aac2e15d0, ref=139958707146706) at beam/erl_process_lock.h:455
#2 erts_smp_proc_lock (c_p=0x7f4aac2e15d0, ref=139958707146706) at beam/erl_process_lock.h:672
#3 demonitor (c_p=0x7f4aac2e15d0, ref=139958707146706) at beam/bif.c:377
#4 0x0000000000477c30 in demonitor_2 (A__p=0x7f4aac2e15d0, A_1=139958707146706, A_2=<value optimized out>) at beam/bif.c:505
#5 0x0000000000521927 in process_main () at beam/beam_emu.c:2542
#6 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b00115200) at beam/erl_process.c:3782
#7 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff316200a0) at pthread/ethread.c:106
#8 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#9 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f4b00b7d700 (LWP 28400)):
#0 0x000000397260e4ed in read () from /lib64/libpthread.so.0
#1 0x000000000053810e in signal_dispatcher_thread_func (unused=<value optimized out>) at sys/unix/sys.c:2906
#2 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff3161ffc0) at pthread/ethread.c:106
#3 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#4 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f4b00ea9700 (LWP 28398)):
#0 0x0000003971ede8b3 in select () from /lib64/libc.so.6
#1 0x0000000000537ba0 in erts_sys_main_thread () at sys/unix/sys.c:3049
#2 0x0000000000444761 in erl_start (argc=54, argv=<value optimized out>) at beam/erl_init.c:1441
#3 0x0000000000426219 in main (argc=<value optimized out>, argv=<value optimized out>) at sys/unix/erl_main.c:29
Thread 7 (Thread 0x7f4aae343700 (LWP 28979)):
#0 0x0000003971eab15d in nanosleep () from /lib64/libc.so.6
#1 0x0000003971edf124 in usleep () from /lib64/libc.so.6
#2 0x00007f4af6371848 in v8::internal::ContextSwitcher::Run (this=0x7f4aa4030140) at src/v8threads.cc:457
#3 0x00007f4af62f1580 in v8::internal::ThreadEntry (arg=0x7f4aa4030140) at src/platform-linux.cc:745
#4 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f4afea69700 (LWP 28420)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8750) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8750) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8750) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
---Type <return> to continue, or q <return> to quit---
Thread 5 (Thread 0x7f4afeb35700 (LWP 28414)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8480) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8480) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8480) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f4afeb79700 (LWP 28412)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8390) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8390) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8390) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f4afebbd700 (LWP 28410)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d82a0) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d82a0) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d82a0) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f4afebdf700 (LWP 28409)):
#0 0x000000397260b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4aff3d8228) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4aff3d8228) at beam/erl_async.c:207
#4 async_main (arg=0x7f4aff3d8228) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#6 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f4afec01700 (LWP 28408)):
#0 0x000000000043ec6b in tree_delete (allctr=<value optimized out>, del=0x12c6a80, flags=<value optimized out>) at beam/erl_bestfit_alloc.c:435
#1 0x0000000000437091 in mbc_free (allctr=0x12a86c0, p=<value optimized out>) at beam/erl_alloc_util.c:867
#2 0x000000000043833f in do_erts_alcu_free (type=<value optimized out>, unused=<value optimized out>, p=0x12c6bb8) at beam/erl_alloc_util.c:2898
#3 erts_alcu_free_thr_pref (type=<value optimized out>, unused=<value optimized out>, p=0x12c6bb8) at beam/erl_alloc_util.c:2949
#4 0x000000000054ca48 in file_async_ready (e=0x12c3230, data=0x12c6bb8) at drivers/common/efile_drv.c:2117
#5 0x0000000000480094 in async_ready (p=0x7f4afeca4e90, data=<value optimized out>) at beam/io.c:2445
#6 0x00000000004e906f in async_main (arg=0x7f4aff3d81b0) at beam/erl_async.c:296
#7 0x0000000000584d89 in thr_wrapper (vtwd=0x7fff31620060) at pthread/ethread.c:106
#8 0x00000039726077f1 in start_thread () from /lib64/libpthread.so.0
#9 0x0000003971ee5ccd in clone () from /lib64/libc.so.6
Show
Karan Kumar
added a comment - https://friendpaste.com/6ctSNIR61xOtG8HSieJ4qA
Hide
Aleksey Kondratenko
added a comment -
Another beam crash in some memory management code in beam. Perhaps duplicate of some other crash we have
Show
Aleksey Kondratenko
added a comment - Another beam crash in some memory management code in beam. Perhaps duplicate of some other crash we have
Hide
Thuan Nguyen
added a comment -
Raw erlang core dump is here https://s3.amazonaws.com/packages.couchbase/core-dump/orange/201209/core.beam.smp.28398.gz
Show
Thuan Nguyen
added a comment - Raw erlang core dump is here https://s3.amazonaws.com/packages.couchbase/core-dump/orange/201209/core.beam.smp.28398.gz
Show
Aleksey Kondratenko
added a comment - Karan, what you want me to do with that?
Show
Aleksey Kondratenko
added a comment - I've heard you were looking for core dump?
Hide
Karan Kumar
added a comment -
Ahh.... If we have duplicate of this bug.. somewhere we can close it..
But the segfault's of any component IMHO are quite severe
But the segfault's of any component IMHO are quite severe
Show
Karan Kumar
added a comment - Ahh.... If we have duplicate of this bug.. somewhere we can close it..
But the segfault's of any component IMHO are quite severe
Hide
Karan Kumar
added a comment -
Another beam.smp segfault:-
#0 right_rotate (allctr=<value optimized out>, del=<value optimized out>, flags=<value optimized out>) at beam/erl_bestfit_alloc.c:252
252 beam/erl_bestfit_alloc.c: No such file or directory.
in beam/erl_bestfit_alloc.c
Missing separate debuginfos, use: debuginfo-install couchbase-server-2.0.0-1728.x86_64
(gdb) t a a bt
Thread 28 (Thread 0x7f4b54ff3700 (LWP 6632)):
#0 0x000000344ee0ebad in fsync () from /lib64/libpthread.so.0
#1 0x000000000053b8db in efile_fsync (errInfo=0xa9744c, fd=<value optimized out>) at drivers/unix/unix_efile.c:806
#2 0x0000000000550e18 in invoke_fsync (data=0xa97420) at drivers/common/efile_drv.c:940
#3 0x00000000004e8f55 in async_main (arg=0x7f4b557a8138) at beam/erl_async.c:281
#4 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#5 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#6 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7f4b54dd1700 (LWP 6647)):
#0 0x000000344ee0dff4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000344ee09328 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x000000344ee091f7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x000000000043811f in ethr_mutex_lock (type=<value optimized out>, extra=<value optimized out>, size=280) at ../include/internal/ethr_mutex.h:589
#4 erts_mtx_lock (type=<value optimized out>, extra=<value optimized out>, size=280) at beam/erl_threads.h:622
#5 erts_alcu_alloc_thr_pref (type=<value optimized out>, extra=<value optimized out>, size=280) at beam/erl_alloc_util.c:2864
#6 0x000000000054fa5b in file_outputv (e=0xa8fb90, ev=0x7f4b54dd0c10) at drivers/common/efile_drv.c:2563
#7 0x0000000000483c78 in erts_write_to_port (caller_id=<value optimized out>, p=0x7f4b54dd0a10, list=139960099274753) at beam/io.c:1159
#8 0x00000000004f5301 in do_port_command (A__p=0x7f4b029c3060, A_1=638167, A_2=139961559834186, A_3=<value optimized out>, flags=0) at beam/erl_bif_port.c:181
#9 0x0000000000521927 in process_main () at beam/beam_emu.c:2542
#10 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b564bafc0) at beam/erl_process.c:3782
#11 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5fc0) at pthread/ethread.c:106
#12 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#13 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7f4b539cf700 (LWP 6649)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000585463 in wait__ (e=0xa19c60) at pthread/ethr_event.c:185
#2 ethr_event_wait (e=0xa19c60) at pthread/ethr_event.c:218
#3 0x000000000049821d in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f4b564d7140, rq=0x7f4b564ba7c0) at beam/erl_threads.h:1809
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4b564d7140, rq=0x7f4b564ba7c0) at beam/erl_process.c:1076
#5 0x000000000049a248 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:5421
#6 0x000000000051e65f in process_main () at beam/beam_emu.c:1225
#7 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b564d7140) at beam/erl_process.c:3782
#8 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5fc0) at pthread/ethread.c:106
#9 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#10 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7f4b543d0700 (LWP 6648)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000585463 in wait__ (e=0xa19ba0) at pthread/ethr_event.c:185
#2 ethr_event_wait (e=0xa19ba0) at pthread/ethr_event.c:218
#3 0x000000000049821d in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f4b564c9080, rq=0x7f4b564ba5c0) at beam/erl_threads.h:1809
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4b564c9080, rq=0x7f4b564ba5c0) at beam/erl_process.c:1076
#5 0x000000000049a248 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:5421
#6 0x000000000051e65f in process_main () at beam/beam_emu.c:1225
#7 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b564c9080) at beam/erl_process.c:3782
#8 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5fc0) at pthread/ethread.c:106
#9 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#10 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7f4b52fce700 (LWP 6650)):
#0 0x000000344e6e62c3 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000053de47 in check_fd_events (ps=0x7f4b570012c8, pr=0x7f4b52fcd360, len=0x7f4b52fcdb7c, utvp=0x7f4b52fcdb60) at sys/common/erl_poll.c:1918
#2 erts_poll_wait_kp (ps=0x7f4b570012c8, pr=0x7f4b52fcd360, len=0x7f4b52fcdb7c, utvp=0x7f4b52fcdb60) at sys/common/erl_poll.c:2053
#3 0x0000000000540efe in erts_check_io_kp (do_wait=<value optimized out>) at sys/common/erl_check_io.c:1161
#4 0x0000000000498551 in scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4b564e5200, rq=0x7f4b564ba9c0) at beam/erl_process.c:1221
#5 0x000000000049a248 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:5421
#6 0x000000000051e65f in process_main () at beam/beam_emu.c:1225
#7 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b564e5200) at beam/erl_process.c:3782
#8 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5fc0) at pthread/ethread.c:106
#9 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#10 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f4afdee0700 (LWP 7076)):
#0 0x000000344ee0d6c0 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f4b50a6718d in v8::internal::LinuxSemaphore::Wait (this=0x7f4b08009b10) at src/platform-linux.cc:869
#2 0x00007f4b50a7e45c in v8::internal::RuntimeProfiler::WaitForSomeIsolateToEnterJS () at src/runtime-profiler.cc:386
#3 0x00007f4b50a68136 in v8::internal::SignalSender::Run (this=0x7f4af80019c0) at src/platform-linux.cc:1126
#4 0x00007f4b50a67580 in v8::internal::ThreadEntry (arg=0x7f4af80019c0) at src/platform-linux.cc:745
#5 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#6 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f4b54f27700 (LWP 6638)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8408) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8408) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8408) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f4b5654c700 (LWP 6628)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x000000000046a1cf in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:697
#3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:472
#4 emergency_watchdog (unused=<value optimized out>) at beam/utils.c:3945
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5ff0) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f4b54e9f700 (LWP 6642)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a85e8) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a85e8) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a85e8) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f4b54faf700 (LWP 6634)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8228) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8228) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8228) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f4b54ee3700 (LWP 6640)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a84f8) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a84f8) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a84f8) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f4b56239700 (LWP 6629)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x0000000000462ad9 in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:697
#3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:472
#4 sys_msg_dispatcher_func (unused=<value optimized out>) at beam/erl_trace.c:3238
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5ff0) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f4b54f8d700 (LWP 6635)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a82a0) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a82a0) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a82a0) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f4b0062c700 (LWP 6660)):
#0 0x000000344e6ab15d in nanosleep () from /lib64/libc.so.6
#1 0x000000344e6df124 in usleep () from /lib64/libc.so.6
#2 0x00007f4b00631397 in terminatorLoop (args=<value optimized out>) at mapreduce_nif.cc:445
#3 0x0000000000584d89 in thr_wrapper (vtwd=0x7f4b54dd0ae0) at pthread/ethread.c:106
#4 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f4b54f05700 (LWP 6639)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8480) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8480) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8480) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f4b55015700 (LWP 6631)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a80c0) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a80c0) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a80c0) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f4af3171700 (LWP 7145)):
#0 0x000000344e6ab15d in nanosleep () from /lib64/libc.so.6
#1 0x000000344e6df124 in usleep () from /lib64/libc.so.6
#2 0x00007f4b50ae7848 in v8::internal::ContextSwitcher::Run (this=0x7f4b0414ace0) at src/v8threads.cc:457
#3 0x00007f4b50a67580 in v8::internal::ThreadEntry (arg=0x7f4b0414ace0) at src/platform-linux.cc:745
#4 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f4b56f4d700 (LWP 6627)):
#0 0x000000344ee0e4ed in read () from /lib64/libpthread.so.0
#1 0x000000000053810e in signal_dispatcher_thread_func (unused=<value optimized out>) at sys/unix/sys.c:2906
#2 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5ee0) at pthread/ethread.c:106
#3 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#4 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f4b54ec1700 (LWP 6641)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8570) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8570) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8570) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f4b54f49700 (LWP 6637)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8390) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8390) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8390) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f4b57279700 (LWP 6626)):
#0 0x000000344e6de8b3 in select () from /lib64/libc.so.6
#1 0x0000000000537ba0 in erts_sys_main_thread () at sys/unix/sys.c:3049
#2 0x0000000000444761 in erl_start (argc=54, argv=<value optimized out>) at beam/erl_init.c:1441
#3 0x0000000000426219 in main (argc=<value optimized out>, argv=<value optimized out>) at sys/unix/erl_main.c:29
Thread 7 (Thread 0x7f4b54f6b700 (LWP 6636)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8318) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8318) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8318) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f4b54fd1700 (LWP 6633)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a81b0) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a81b0) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a81b0) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f4b54e17700 (LWP 6646)):
#0 0x000000344ee0f03d in waitpid () from /lib64/libpthread.so.0
#1 0x000000000053906f in child_waiter (unused=<value optimized out>) at sys/unix/sys.c:2821
#2 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5fa0) at pthread/ethread.c:106
#3 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#4 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f4b55037700 (LWP 6630)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8048) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8048) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8048) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f4b54e5b700 (LWP 6644)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a86d8) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a86d8) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a86d8) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f4b54e39700 (LWP 6645)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8750) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8750) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8750) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f4b54e7d700 (LWP 6643)):
#0 right_rotate (allctr=<value optimized out>, del=<value optimized out>, flags=<value optimized out>) at beam/erl_bestfit_alloc.c:252
#1 tree_delete (allctr=<value optimized out>, del=<value optimized out>, flags=<value optimized out>) at beam/erl_bestfit_alloc.c:496
#2 0x00000000004370db in mbc_free (allctr=0xa7c2c0, p=<value optimized out>) at beam/erl_alloc_util.c:883
#3 0x000000000043833f in do_erts_alcu_free (type=<value optimized out>, unused=<value optimized out>, p=0xa87a38) at beam/erl_alloc_util.c:2898
#4 erts_alcu_free_thr_pref (type=<value optimized out>, unused=<value optimized out>, p=0xa87a38) at beam/erl_alloc_util.c:2949
#5 0x000000000054cac0 in file_async_ready (e=0xa84238, data=0xa87a38) at drivers/common/efile_drv.c:2091
#6 0x0000000000480094 in async_ready (p=0x7f4b553521e0, data=<value optimized out>) at beam/io.c:2445
#7 0x00000000004e906f in async_main (arg=0x7f4b557a8660) at beam/erl_async.c:296
#8 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#9 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#10 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
#0 right_rotate (allctr=<value optimized out>, del=<value optimized out>, flags=<value optimized out>) at beam/erl_bestfit_alloc.c:252
252 beam/erl_bestfit_alloc.c: No such file or directory.
in beam/erl_bestfit_alloc.c
Missing separate debuginfos, use: debuginfo-install couchbase-server-2.0.0-1728.x86_64
(gdb) t a a bt
Thread 28 (Thread 0x7f4b54ff3700 (LWP 6632)):
#0 0x000000344ee0ebad in fsync () from /lib64/libpthread.so.0
#1 0x000000000053b8db in efile_fsync (errInfo=0xa9744c, fd=<value optimized out>) at drivers/unix/unix_efile.c:806
#2 0x0000000000550e18 in invoke_fsync (data=0xa97420) at drivers/common/efile_drv.c:940
#3 0x00000000004e8f55 in async_main (arg=0x7f4b557a8138) at beam/erl_async.c:281
#4 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#5 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#6 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7f4b54dd1700 (LWP 6647)):
#0 0x000000344ee0dff4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000344ee09328 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x000000344ee091f7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x000000000043811f in ethr_mutex_lock (type=<value optimized out>, extra=<value optimized out>, size=280) at ../include/internal/ethr_mutex.h:589
#4 erts_mtx_lock (type=<value optimized out>, extra=<value optimized out>, size=280) at beam/erl_threads.h:622
#5 erts_alcu_alloc_thr_pref (type=<value optimized out>, extra=<value optimized out>, size=280) at beam/erl_alloc_util.c:2864
#6 0x000000000054fa5b in file_outputv (e=0xa8fb90, ev=0x7f4b54dd0c10) at drivers/common/efile_drv.c:2563
#7 0x0000000000483c78 in erts_write_to_port (caller_id=<value optimized out>, p=0x7f4b54dd0a10, list=139960099274753) at beam/io.c:1159
#8 0x00000000004f5301 in do_port_command (A__p=0x7f4b029c3060, A_1=638167, A_2=139961559834186, A_3=<value optimized out>, flags=0) at beam/erl_bif_port.c:181
#9 0x0000000000521927 in process_main () at beam/beam_emu.c:2542
#10 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b564bafc0) at beam/erl_process.c:3782
#11 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5fc0) at pthread/ethread.c:106
#12 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#13 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7f4b539cf700 (LWP 6649)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000585463 in wait__ (e=0xa19c60) at pthread/ethr_event.c:185
#2 ethr_event_wait (e=0xa19c60) at pthread/ethr_event.c:218
#3 0x000000000049821d in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f4b564d7140, rq=0x7f4b564ba7c0) at beam/erl_threads.h:1809
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4b564d7140, rq=0x7f4b564ba7c0) at beam/erl_process.c:1076
#5 0x000000000049a248 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:5421
#6 0x000000000051e65f in process_main () at beam/beam_emu.c:1225
#7 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b564d7140) at beam/erl_process.c:3782
#8 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5fc0) at pthread/ethread.c:106
#9 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#10 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7f4b543d0700 (LWP 6648)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000585463 in wait__ (e=0xa19ba0) at pthread/ethr_event.c:185
#2 ethr_event_wait (e=0xa19ba0) at pthread/ethr_event.c:218
#3 0x000000000049821d in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f4b564c9080, rq=0x7f4b564ba5c0) at beam/erl_threads.h:1809
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4b564c9080, rq=0x7f4b564ba5c0) at beam/erl_process.c:1076
#5 0x000000000049a248 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:5421
#6 0x000000000051e65f in process_main () at beam/beam_emu.c:1225
#7 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b564c9080) at beam/erl_process.c:3782
#8 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5fc0) at pthread/ethread.c:106
#9 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#10 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7f4b52fce700 (LWP 6650)):
#0 0x000000344e6e62c3 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000053de47 in check_fd_events (ps=0x7f4b570012c8, pr=0x7f4b52fcd360, len=0x7f4b52fcdb7c, utvp=0x7f4b52fcdb60) at sys/common/erl_poll.c:1918
#2 erts_poll_wait_kp (ps=0x7f4b570012c8, pr=0x7f4b52fcd360, len=0x7f4b52fcdb7c, utvp=0x7f4b52fcdb60) at sys/common/erl_poll.c:2053
#3 0x0000000000540efe in erts_check_io_kp (do_wait=<value optimized out>) at sys/common/erl_check_io.c:1161
#4 0x0000000000498551 in scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4b564e5200, rq=0x7f4b564ba9c0) at beam/erl_process.c:1221
#5 0x000000000049a248 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:5421
#6 0x000000000051e65f in process_main () at beam/beam_emu.c:1225
#7 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b564e5200) at beam/erl_process.c:3782
#8 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5fc0) at pthread/ethread.c:106
#9 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#10 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f4afdee0700 (LWP 7076)):
#0 0x000000344ee0d6c0 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f4b50a6718d in v8::internal::LinuxSemaphore::Wait (this=0x7f4b08009b10) at src/platform-linux.cc:869
#2 0x00007f4b50a7e45c in v8::internal::RuntimeProfiler::WaitForSomeIsolateToEnterJS () at src/runtime-profiler.cc:386
#3 0x00007f4b50a68136 in v8::internal::SignalSender::Run (this=0x7f4af80019c0) at src/platform-linux.cc:1126
#4 0x00007f4b50a67580 in v8::internal::ThreadEntry (arg=0x7f4af80019c0) at src/platform-linux.cc:745
#5 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#6 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f4b54f27700 (LWP 6638)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8408) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8408) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8408) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f4b5654c700 (LWP 6628)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x000000000046a1cf in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:697
#3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:472
#4 emergency_watchdog (unused=<value optimized out>) at beam/utils.c:3945
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5ff0) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f4b54e9f700 (LWP 6642)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a85e8) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a85e8) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a85e8) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f4b54faf700 (LWP 6634)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8228) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8228) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8228) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f4b54ee3700 (LWP 6640)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a84f8) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a84f8) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a84f8) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f4b56239700 (LWP 6629)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x0000000000462ad9 in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:697
#3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:472
#4 sys_msg_dispatcher_func (unused=<value optimized out>) at beam/erl_trace.c:3238
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5ff0) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f4b54f8d700 (LWP 6635)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a82a0) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a82a0) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a82a0) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f4b0062c700 (LWP 6660)):
#0 0x000000344e6ab15d in nanosleep () from /lib64/libc.so.6
#1 0x000000344e6df124 in usleep () from /lib64/libc.so.6
#2 0x00007f4b00631397 in terminatorLoop (args=<value optimized out>) at mapreduce_nif.cc:445
#3 0x0000000000584d89 in thr_wrapper (vtwd=0x7f4b54dd0ae0) at pthread/ethread.c:106
#4 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f4b54f05700 (LWP 6639)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8480) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8480) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8480) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f4b55015700 (LWP 6631)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a80c0) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a80c0) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a80c0) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f4af3171700 (LWP 7145)):
#0 0x000000344e6ab15d in nanosleep () from /lib64/libc.so.6
#1 0x000000344e6df124 in usleep () from /lib64/libc.so.6
#2 0x00007f4b50ae7848 in v8::internal::ContextSwitcher::Run (this=0x7f4b0414ace0) at src/v8threads.cc:457
#3 0x00007f4b50a67580 in v8::internal::ThreadEntry (arg=0x7f4b0414ace0) at src/platform-linux.cc:745
#4 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f4b56f4d700 (LWP 6627)):
#0 0x000000344ee0e4ed in read () from /lib64/libpthread.so.0
#1 0x000000000053810e in signal_dispatcher_thread_func (unused=<value optimized out>) at sys/unix/sys.c:2906
#2 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5ee0) at pthread/ethread.c:106
#3 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#4 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f4b54ec1700 (LWP 6641)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8570) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8570) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8570) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f4b54f49700 (LWP 6637)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8390) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8390) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8390) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f4b57279700 (LWP 6626)):
#0 0x000000344e6de8b3 in select () from /lib64/libc.so.6
#1 0x0000000000537ba0 in erts_sys_main_thread () at sys/unix/sys.c:3049
#2 0x0000000000444761 in erl_start (argc=54, argv=<value optimized out>) at beam/erl_init.c:1441
#3 0x0000000000426219 in main (argc=<value optimized out>, argv=<value optimized out>) at sys/unix/erl_main.c:29
Thread 7 (Thread 0x7f4b54f6b700 (LWP 6636)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8318) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8318) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8318) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f4b54fd1700 (LWP 6633)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a81b0) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a81b0) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a81b0) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f4b54e17700 (LWP 6646)):
#0 0x000000344ee0f03d in waitpid () from /lib64/libpthread.so.0
#1 0x000000000053906f in child_waiter (unused=<value optimized out>) at sys/unix/sys.c:2821
#2 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5fa0) at pthread/ethread.c:106
#3 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#4 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f4b55037700 (LWP 6630)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8048) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8048) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8048) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f4b54e5b700 (LWP 6644)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a86d8) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a86d8) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a86d8) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f4b54e39700 (LWP 6645)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8750) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8750) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8750) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f4b54e7d700 (LWP 6643)):
#0 right_rotate (allctr=<value optimized out>, del=<value optimized out>, flags=<value optimized out>) at beam/erl_bestfit_alloc.c:252
#1 tree_delete (allctr=<value optimized out>, del=<value optimized out>, flags=<value optimized out>) at beam/erl_bestfit_alloc.c:496
#2 0x00000000004370db in mbc_free (allctr=0xa7c2c0, p=<value optimized out>) at beam/erl_alloc_util.c:883
#3 0x000000000043833f in do_erts_alcu_free (type=<value optimized out>, unused=<value optimized out>, p=0xa87a38) at beam/erl_alloc_util.c:2898
#4 erts_alcu_free_thr_pref (type=<value optimized out>, unused=<value optimized out>, p=0xa87a38) at beam/erl_alloc_util.c:2949
#5 0x000000000054cac0 in file_async_ready (e=0xa84238, data=0xa87a38) at drivers/common/efile_drv.c:2091
#6 0x0000000000480094 in async_ready (p=0x7f4b553521e0, data=<value optimized out>) at beam/io.c:2445
#7 0x00000000004e906f in async_main (arg=0x7f4b557a8660) at beam/erl_async.c:296
#8 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#9 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#10 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Show
Karan Kumar
added a comment - Another beam.smp segfault:-
#0 right_rotate (allctr=<value optimized out>, del=<value optimized out>, flags=<value optimized out>) at beam/erl_bestfit_alloc.c:252
252 beam/erl_bestfit_alloc.c: No such file or directory.
in beam/erl_bestfit_alloc.c
Missing separate debuginfos, use: debuginfo-install couchbase-server-2.0.0-1728.x86_64
(gdb) t a a bt
Thread 28 (Thread 0x7f4b54ff3700 (LWP 6632)):
#0 0x000000344ee0ebad in fsync () from /lib64/libpthread.so.0
#1 0x000000000053b8db in efile_fsync (errInfo=0xa9744c, fd=<value optimized out>) at drivers/unix/unix_efile.c:806
#2 0x0000000000550e18 in invoke_fsync (data=0xa97420) at drivers/common/efile_drv.c:940
#3 0x00000000004e8f55 in async_main (arg=0x7f4b557a8138) at beam/erl_async.c:281
#4 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#5 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#6 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 27 (Thread 0x7f4b54dd1700 (LWP 6647)):
#0 0x000000344ee0dff4 in __lll_lock_wait () from /lib64/libpthread.so.0
#1 0x000000344ee09328 in _L_lock_854 () from /lib64/libpthread.so.0
#2 0x000000344ee091f7 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3 0x000000000043811f in ethr_mutex_lock (type=<value optimized out>, extra=<value optimized out>, size=280) at ../include/internal/ethr_mutex.h:589
#4 erts_mtx_lock (type=<value optimized out>, extra=<value optimized out>, size=280) at beam/erl_threads.h:622
#5 erts_alcu_alloc_thr_pref (type=<value optimized out>, extra=<value optimized out>, size=280) at beam/erl_alloc_util.c:2864
#6 0x000000000054fa5b in file_outputv (e=0xa8fb90, ev=0x7f4b54dd0c10) at drivers/common/efile_drv.c:2563
#7 0x0000000000483c78 in erts_write_to_port (caller_id=<value optimized out>, p=0x7f4b54dd0a10, list=139960099274753) at beam/io.c:1159
#8 0x00000000004f5301 in do_port_command (A__p=0x7f4b029c3060, A_1=638167, A_2=139961559834186, A_3=<value optimized out>, flags=0) at beam/erl_bif_port.c:181
#9 0x0000000000521927 in process_main () at beam/beam_emu.c:2542
#10 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b564bafc0) at beam/erl_process.c:3782
#11 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5fc0) at pthread/ethread.c:106
#12 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#13 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 26 (Thread 0x7f4b539cf700 (LWP 6649)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000585463 in wait__ (e=0xa19c60) at pthread/ethr_event.c:185
#2 ethr_event_wait (e=0xa19c60) at pthread/ethr_event.c:218
#3 0x000000000049821d in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f4b564d7140, rq=0x7f4b564ba7c0) at beam/erl_threads.h:1809
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4b564d7140, rq=0x7f4b564ba7c0) at beam/erl_process.c:1076
#5 0x000000000049a248 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:5421
#6 0x000000000051e65f in process_main () at beam/beam_emu.c:1225
#7 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b564d7140) at beam/erl_process.c:3782
#8 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5fc0) at pthread/ethread.c:106
#9 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#10 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 25 (Thread 0x7f4b543d0700 (LWP 6648)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000585463 in wait__ (e=0xa19ba0) at pthread/ethr_event.c:185
#2 ethr_event_wait (e=0xa19ba0) at pthread/ethr_event.c:218
#3 0x000000000049821d in erts_tse_wait (fcalls=<value optimized out>, esdp=0x7f4b564c9080, rq=0x7f4b564ba5c0) at beam/erl_threads.h:1809
#4 scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4b564c9080, rq=0x7f4b564ba5c0) at beam/erl_process.c:1076
#5 0x000000000049a248 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:5421
#6 0x000000000051e65f in process_main () at beam/beam_emu.c:1225
#7 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b564c9080) at beam/erl_process.c:3782
#8 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5fc0) at pthread/ethread.c:106
#9 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#10 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 24 (Thread 0x7f4b52fce700 (LWP 6650)):
#0 0x000000344e6e62c3 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000053de47 in check_fd_events (ps=0x7f4b570012c8, pr=0x7f4b52fcd360, len=0x7f4b52fcdb7c, utvp=0x7f4b52fcdb60) at sys/common/erl_poll.c:1918
#2 erts_poll_wait_kp (ps=0x7f4b570012c8, pr=0x7f4b52fcd360, len=0x7f4b52fcdb7c, utvp=0x7f4b52fcdb60) at sys/common/erl_poll.c:2053
#3 0x0000000000540efe in erts_check_io_kp (do_wait=<value optimized out>) at sys/common/erl_check_io.c:1161
#4 0x0000000000498551 in scheduler_wait (fcalls=<value optimized out>, esdp=0x7f4b564e5200, rq=0x7f4b564ba9c0) at beam/erl_process.c:1221
#5 0x000000000049a248 in schedule (p=<value optimized out>, calls=<value optimized out>) at beam/erl_process.c:5421
#6 0x000000000051e65f in process_main () at beam/beam_emu.c:1225
#7 0x000000000048d496 in sched_thread_func (vesdp=0x7f4b564e5200) at beam/erl_process.c:3782
#8 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5fc0) at pthread/ethread.c:106
#9 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#10 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 23 (Thread 0x7f4afdee0700 (LWP 7076)):
#0 0x000000344ee0d6c0 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007f4b50a6718d in v8::internal::LinuxSemaphore::Wait (this=0x7f4b08009b10) at src/platform-linux.cc:869
#2 0x00007f4b50a7e45c in v8::internal::RuntimeProfiler::WaitForSomeIsolateToEnterJS () at src/runtime-profiler.cc:386
#3 0x00007f4b50a68136 in v8::internal::SignalSender::Run (this=0x7f4af80019c0) at src/platform-linux.cc:1126
#4 0x00007f4b50a67580 in v8::internal::ThreadEntry (arg=0x7f4af80019c0) at src/platform-linux.cc:745
#5 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#6 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 22 (Thread 0x7f4b54f27700 (LWP 6638)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8408) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8408) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8408) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 21 (Thread 0x7f4b5654c700 (LWP 6628)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x000000000046a1cf in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:697
#3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:472
#4 emergency_watchdog (unused=<value optimized out>) at beam/utils.c:3945
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5ff0) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 20 (Thread 0x7f4b54e9f700 (LWP 6642)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a85e8) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a85e8) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a85e8) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 19 (Thread 0x7f4b54faf700 (LWP 6634)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8228) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8228) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8228) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 18 (Thread 0x7f4b54ee3700 (LWP 6640)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a84f8) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a84f8) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a84f8) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 17 (Thread 0x7f4b56239700 (LWP 6629)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x0000000000462ad9 in erts_cnd_wait (unused=<value optimized out>) at beam/erl_threads.h:697
#3 erts_smp_cnd_wait (unused=<value optimized out>) at beam/erl_smp.h:472
#4 sys_msg_dispatcher_func (unused=<value optimized out>) at beam/erl_trace.c:3238
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5ff0) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 16 (Thread 0x7f4b54f8d700 (LWP 6635)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a82a0) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a82a0) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a82a0) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7f4b0062c700 (LWP 6660)):
#0 0x000000344e6ab15d in nanosleep () from /lib64/libc.so.6
#1 0x000000344e6df124 in usleep () from /lib64/libc.so.6
#2 0x00007f4b00631397 in terminatorLoop (args=<value optimized out>) at mapreduce_nif.cc:445
#3 0x0000000000584d89 in thr_wrapper (vtwd=0x7f4b54dd0ae0) at pthread/ethread.c:106
#4 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7f4b54f05700 (LWP 6639)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8480) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8480) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8480) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7f4b55015700 (LWP 6631)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a80c0) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a80c0) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a80c0) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7f4af3171700 (LWP 7145)):
#0 0x000000344e6ab15d in nanosleep () from /lib64/libc.so.6
#1 0x000000344e6df124 in usleep () from /lib64/libc.so.6
#2 0x00007f4b50ae7848 in v8::internal::ContextSwitcher::Run (this=0x7f4b0414ace0) at src/v8threads.cc:457
#3 0x00007f4b50a67580 in v8::internal::ThreadEntry (arg=0x7f4b0414ace0) at src/platform-linux.cc:745
#4 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#5 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7f4b56f4d700 (LWP 6627)):
#0 0x000000344ee0e4ed in read () from /lib64/libpthread.so.0
#1 0x000000000053810e in signal_dispatcher_thread_func (unused=<value optimized out>) at sys/unix/sys.c:2906
#2 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5ee0) at pthread/ethread.c:106
#3 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#4 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7f4b54ec1700 (LWP 6641)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8570) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8570) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8570) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7f4b54f49700 (LWP 6637)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8390) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8390) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8390) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7f4b57279700 (LWP 6626)):
#0 0x000000344e6de8b3 in select () from /lib64/libc.so.6
#1 0x0000000000537ba0 in erts_sys_main_thread () at sys/unix/sys.c:3049
#2 0x0000000000444761 in erl_start (argc=54, argv=<value optimized out>) at beam/erl_init.c:1441
#3 0x0000000000426219 in main (argc=<value optimized out>, argv=<value optimized out>) at sys/unix/erl_main.c:29
Thread 7 (Thread 0x7f4b54f6b700 (LWP 6636)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8318) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8318) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8318) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7f4b54fd1700 (LWP 6633)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a81b0) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a81b0) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a81b0) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7f4b54e17700 (LWP 6646)):
#0 0x000000344ee0f03d in waitpid () from /lib64/libpthread.so.0
#1 0x000000000053906f in child_waiter (unused=<value optimized out>) at sys/unix/sys.c:2821
#2 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5fa0) at pthread/ethread.c:106
#3 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#4 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7f4b55037700 (LWP 6630)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8048) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8048) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8048) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 3 (Thread 0x7f4b54e5b700 (LWP 6644)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
---Type <return> to continue, or q <return> to quit---
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a86d8) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a86d8) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a86d8) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7f4b54e39700 (LWP 6645)):
#0 0x000000344ee0b3dc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x0000000000582e29 in ethr_cond_wait (cnd=<value optimized out>, mtx=<value optimized out>) at common/ethr_mutex.c:1350
#2 0x00000000004e8efb in erts_cnd_wait (arg=0x7f4b557a8750) at beam/erl_threads.h:697
#3 async_get (arg=0x7f4b557a8750) at beam/erl_async.c:207
#4 async_main (arg=0x7f4b557a8750) at beam/erl_async.c:274
#5 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#6 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#7 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Thread 1 (Thread 0x7f4b54e7d700 (LWP 6643)):
#0 right_rotate (allctr=<value optimized out>, del=<value optimized out>, flags=<value optimized out>) at beam/erl_bestfit_alloc.c:252
#1 tree_delete (allctr=<value optimized out>, del=<value optimized out>, flags=<value optimized out>) at beam/erl_bestfit_alloc.c:496
#2 0x00000000004370db in mbc_free (allctr=0xa7c2c0, p=<value optimized out>) at beam/erl_alloc_util.c:883
#3 0x000000000043833f in do_erts_alcu_free (type=<value optimized out>, unused=<value optimized out>, p=0xa87a38) at beam/erl_alloc_util.c:2898
#4 erts_alcu_free_thr_pref (type=<value optimized out>, unused=<value optimized out>, p=0xa87a38) at beam/erl_alloc_util.c:2949
#5 0x000000000054cac0 in file_async_ready (e=0xa84238, data=0xa87a38) at drivers/common/efile_drv.c:2091
#6 0x0000000000480094 in async_ready (p=0x7f4b553521e0, data=<value optimized out>) at beam/io.c:2445
#7 0x00000000004e906f in async_main (arg=0x7f4b557a8660) at beam/erl_async.c:296
#8 0x0000000000584d89 in thr_wrapper (vtwd=0x7fffa31f5f80) at pthread/ethread.c:106
#9 0x000000344ee077f1 in start_thread () from /lib64/libpthread.so.0
#10 0x000000344e6e5ccd in clone () from /lib64/libc.so.6
Hide
Thuan Nguyen
added a comment -
Raw erlang core file is here https://s3.amazonaws.com/packages.couchbase/core-dump/orange/201209/core.beam.smp.6626.gz
Show
Thuan Nguyen
added a comment - Raw erlang core file is here https://s3.amazonaws.com/packages.couchbase/core-dump/orange/201209/core.beam.smp.6626.gz
Hide
Karan Kumar
added a comment -
Another crash.
We are seeing it most deterministically.
https://friendpaste.com/55MrWtpTIufnEMRrxXbLsI
We are seeing it most deterministically.
https://friendpaste.com/55MrWtpTIufnEMRrxXbLsI
Show
Karan Kumar
added a comment - Another crash.
We are seeing it most deterministically.
https://friendpaste.com/55MrWtpTIufnEMRrxXbLsI
Hide
Thuan Nguyen
added a comment -
Got another erlang segfault during swap rebalance in new system test setup with all SSD drive and consistent view enable
Erlang crashed on node added to cluster
Link to stack trace of erlang crash node https://friendpaste.com/58uGMaPS8xo2tcLsDSTTTM
Erlang crashed on node added to cluster
Link to stack trace of erlang crash node https://friendpaste.com/58uGMaPS8xo2tcLsDSTTTM
Show
Thuan Nguyen
added a comment - Got another erlang segfault during swap rebalance in new system test setup with all SSD drive and consistent view enable
Erlang crashed on node added to cluster
Link to stack trace of erlang crash node https://friendpaste.com/58uGMaPS8xo2tcLsDSTTTM
Hide
Thuan Nguyen
added a comment -
Stack trace erlang crash on build 2.0.0-1832 on system test
https://friendpaste.com/59jNoNCWLSrPOyTCQK76Rq
https://friendpaste.com/59jNoNCWLSrPOyTCQK76Rq
Show
Thuan Nguyen
added a comment - Stack trace erlang crash on build 2.0.0-1832 on system test
https://friendpaste.com/59jNoNCWLSrPOyTCQK76Rq
Hide
Filipe Manana
added a comment -
Tony, do you think you can save all the files (database, indexes, etc) from the moment the crash happens?
I think it's more helpful rather than pasting a stack trace everytime it happens.
I think it's more helpful rather than pasting a stack trace everytime it happens.
Show
Filipe Manana
added a comment - Tony, do you think you can save all the files (database, indexes, etc) from the moment the crash happens?
I think it's more helpful rather than pasting a stack trace everytime it happens.
Hide
Filipe Manana
added a comment -
Tony, does this always happen as well on the same machine (physical or virtual) ?
Show
Filipe Manana
added a comment - Tony, does this always happen as well on the same machine (physical or virtual) ?
Hide
Thuan Nguyen
added a comment -
Crash did not happen on the same vm. It happened not only in our vms but also in ec2 too as I did test last time
Show
Thuan Nguyen
added a comment - Crash did not happen on the same vm. It happened not only in our vms but also in ec2 too as I did test last time
Hide
Hit erlang crash on toybuild 10112 on system test.
Link to stack trace on node 39 https://friendpaste.com/59jNoNCWLSrPOyTCQKWoCO
Core dump is on /data on node 39.
Cluster information:
- 8 centos 6.2 64bit server with 4 cores CPU
- Each server has 32 GB RAM and 400 GB SSD disk.
- 24.8 GB RAM for couchbase server at each node
- SSD disk format ext4 on /data
- Each server has its own SSD drive, no disk sharing with other server.
- Create cluster with 6 nodes installed couchbase server 2.0.0-10112 (toy build)
- Cluster has 2 buckets, default (12GB) and saslbucket (12GB).
- Each bucket has one doc and 2 views for each doc (default d1 and saslbucket d11)
- Disable consistent view on cluster.
10.6.2.37
10.6.2.38
10.6.2.39
10.6.2.40
10.6.2.44
10.6.2.45
10.6.2.42
10.6.2.43
* Load 14 million items to both bucket. Each key has size from 512 bytes to 1024 bytes
* Queries all 4 views from 2 docs
* Mutate 14 million items with key size from 1500 to 1024 bytes
* Do swap rebalance 2 nodes at the same time.
* Cluster is left as in crashed state to investigate
* Link to collect info of all nodes https://s3.amazonaws.com/packages.couchbase/collect_info/orange/2_0_0/201210/8nodes-col-toybuild-10112-beam-segfault-20121012-003212.tgz
Link to stack trace on node 39 https://friendpaste.com/59jNoNCWLSrPOyTCQKWoCO
Core dump is on /data on node 39.
Cluster information:
- 8 centos 6.2 64bit server with 4 cores CPU
- Each server has 32 GB RAM and 400 GB SSD disk.
- 24.8 GB RAM for couchbase server at each node
- SSD disk format ext4 on /data
- Each server has its own SSD drive, no disk sharing with other server.
- Create cluster with 6 nodes installed couchbase server 2.0.0-10112 (toy build)
- Cluster has 2 buckets, default (12GB) and saslbucket (12GB).
- Each bucket has one doc and 2 views for each doc (default d1 and saslbucket d11)
- Disable consistent view on cluster.
10.6.2.37
10.6.2.38
10.6.2.39
10.6.2.40
10.6.2.44
10.6.2.45
10.6.2.42
10.6.2.43
* Load 14 million items to both bucket. Each key has size from 512 bytes to 1024 bytes
* Queries all 4 views from 2 docs
* Mutate 14 million items with key size from 1500 to 1024 bytes
* Do swap rebalance 2 nodes at the same time.
* Cluster is left as in crashed state to investigate
* Link to collect info of all nodes https://s3.amazonaws.com/packages.couchbase/collect_info/orange/2_0_0/201210/8nodes-col-toybuild-10112-beam-segfault-20121012-003212.tgz
Show
Thuan Nguyen
added a comment - - edited Hit erlang crash on toybuild 10112 on system test.
Link to stack trace on node 39 https://friendpaste.com/59jNoNCWLSrPOyTCQKWoCO
Core dump is on /data on node 39.
Cluster information:
- 8 centos 6.2 64bit server with 4 cores CPU
- Each server has 32 GB RAM and 400 GB SSD disk.
- 24.8 GB RAM for couchbase server at each node
- SSD disk format ext4 on /data
- Each server has its own SSD drive, no disk sharing with other server.
- Create cluster with 6 nodes installed couchbase server 2.0.0-10112 (toy build)
- Cluster has 2 buckets, default (12GB) and saslbucket (12GB).
- Each bucket has one doc and 2 views for each doc (default d1 and saslbucket d11)
- Disable consistent view on cluster.
10.6.2.37
10.6.2.38
10.6.2.39
10.6.2.40
10.6.2.44
10.6.2.45
10.6.2.42
10.6.2.43
* Load 14 million items to both bucket. Each key has size from 512 bytes to 1024 bytes
* Queries all 4 views from 2 docs
* Mutate 14 million items with key size from 1500 to 1024 bytes
* Do swap rebalance 2 nodes at the same time.
* Cluster is left as in crashed state to investigate
* Link to collect info of all nodes https://s3.amazonaws.com/packages.couchbase/collect_info/orange/2_0_0/201210/8nodes-col-toybuild-10112-beam-segfault-20121012-003212.tgz
Hide
Damien Katz
added a comment -
Right now we think this might be related to another bug spotted on Erlang VMs. When started with async threads and using "raw" file descriptors, if the process that opened the file is shutdown or crashes abnormally, the the file descriptor is leaked.
one possible fix is to change the erlang startup parameters to turn off async file io, so this:
> erl +A 16 +sbt u +P 327680 +K true
becomes:
> erl +sbt u +P 327680 +K true
one possible fix is to change the erlang startup parameters to turn off async file io, so this:
> erl +A 16 +sbt u +P 327680 +K true
becomes:
> erl +sbt u +P 327680 +K true
Show
Damien Katz
added a comment - Right now we think this might be related to another bug spotted on Erlang VMs. When started with async threads and using "raw" file descriptors, if the process that opened the file is shutdown or crashes abnormally, the the file descriptor is leaked.
one possible fix is to change the erlang startup parameters to turn off async file io, so this:
> erl +A 16 +sbt u +P 327680 +K true
becomes:
> erl +sbt u +P 327680 +K true
Hide
Aleksey Kondratenko
added a comment -
Just keep in mind that without async io we'll have _massive_ timeouts all over the place. We specifically _had exactly this problem_ in early days of 1.6.0.
Show
Aleksey Kondratenko
added a comment - Just keep in mind that without async io we'll have _massive_ timeouts all over the place. We specifically _had exactly this problem_ in early days of 1.6.0.
Hide
Farshid Ghods
added a comment -
according to karan he has not seen any crash or rebalance timeouts on cluster where there are lot of rebalancing
1- 2.0 cluster doing views , 2 buckets , 30M items , 2 ddocs , 2 views , 8 nodes under 30k ops/sec and 300 queries per second
2- 20+ node cluster running 1.8.x key value use cases
performance team is also running more performance tests to verify that +s12 works ( without +a) option
we have not yet run xdcr system tests with these settings.
1- 2.0 cluster doing views , 2 buckets , 30M items , 2 ddocs , 2 views , 8 nodes under 30k ops/sec and 300 queries per second
2- 20+ node cluster running 1.8.x key value use cases
performance team is also running more performance tests to verify that +s12 works ( without +a) option
we have not yet run xdcr system tests with these settings.
Show
Farshid Ghods
added a comment - according to karan he has not seen any crash or rebalance timeouts on cluster where there are lot of rebalancing
1- 2.0 cluster doing views , 2 buckets , 30M items , 2 ddocs , 2 views , 8 nodes under 30k ops/sec and 300 queries per second
2- 20+ node cluster running 1.8.x key value use cases
performance team is also running more performance tests to verify that +s12 works ( without +a) option
we have not yet run xdcr system tests with these settings.
Hide
Ketaki Gangal
added a comment -
Tested xdcr-functional tests with the setting +S 120:120, it works fine.
Show
Ketaki Gangal
added a comment - Tested xdcr-functional tests with the setting +S 120:120, it works fine.
Hide
Steve Yen
added a comment -
Notes from mtg with damien, alk, farshid, sriram, sharon, aaron, yaseen, aaron
priority: key-value use-case cannot regress from 1.8.1
plan A - more testing with no-async (farshid).
* +S 120:120 (not 12:12)
* QE run tests on real disk & durable configuration setting.
* look for timeouts in the U/I.
* windows.
* use physical disk boxes from perf team.
plan B - track down root cause.
* QE will run system tests against toy-build from siri, on VMs ok.
* instrument memory calls, look for culprit - toy build (siri).
* idea - hammer NIF erlang unit tests repeatedly overnight (alk).
* idea - try "A1" (try this if plan A doesn't work).
+ frees up scheduler threads.
+ and single async-i/o thread might mitigate crashes.
- this will serialize I/O even if we have multiple disks.
plan C - separate processes (if plan A doesn't work, and alk's team has time).
+ should fix the problem.
- costly.
- some tight coupling between some modules.
plan D - run async for key-value use case, sync for view use case.
-----------
by the way, 1.8.1 used async threads.
async threads mode can leak file descriptors.
* this was some different issue.
priority: key-value use-case cannot regress from 1.8.1
plan A - more testing with no-async (farshid).
* +S 120:120 (not 12:12)
* QE run tests on real disk & durable configuration setting.
* look for timeouts in the U/I.
* windows.
* use physical disk boxes from perf team.
plan B - track down root cause.
* QE will run system tests against toy-build from siri, on VMs ok.
* instrument memory calls, look for culprit - toy build (siri).
* idea - hammer NIF erlang unit tests repeatedly overnight (alk).
* idea - try "A1" (try this if plan A doesn't work).
+ frees up scheduler threads.
+ and single async-i/o thread might mitigate crashes.
- this will serialize I/O even if we have multiple disks.
plan C - separate processes (if plan A doesn't work, and alk's team has time).
+ should fix the problem.
- costly.
- some tight coupling between some modules.
plan D - run async for key-value use case, sync for view use case.
-----------
by the way, 1.8.1 used async threads.
async threads mode can leak file descriptors.
* this was some different issue.
Show
Steve Yen
added a comment - Notes from mtg with damien, alk, farshid, sriram, sharon, aaron, yaseen, aaron
priority: key-value use-case cannot regress from 1.8.1
plan A - more testing with no-async (farshid).
* +S 120:120 (not 12:12)
* QE run tests on real disk & durable configuration setting.
* look for timeouts in the U/I.
* windows.
* use physical disk boxes from perf team.
plan B - track down root cause.
* QE will run system tests against toy-build from siri, on VMs ok.
* instrument memory calls, look for culprit - toy build (siri).
* idea - hammer NIF erlang unit tests repeatedly overnight (alk).
* idea - try "A1" (try this if plan A doesn't work).
+ frees up scheduler threads.
+ and single async-i/o thread might mitigate crashes.
- this will serialize I/O even if we have multiple disks.
plan C - separate processes (if plan A doesn't work, and alk's team has time).
+ should fix the problem.
- costly.
- some tight coupling between some modules.
plan D - run async for key-value use case, sync for view use case.
-----------
by the way, 1.8.1 used async threads.
async threads mode can leak file descriptors.
* this was some different issue.
Hide
Aleksey Kondratenko
added a comment -
Just verified perf team's RHEL 5.8 boxes. Indeed RHEL 5.8 even on ext3 has barriers ON by default. That's good for me.
I tested with my tool and I'm seeing typical performance from 7.2k RPMs disk.
I tested with my tool and I'm seeing typical performance from 7.2k RPMs disk.
Show
Aleksey Kondratenko
added a comment - Just verified perf team's RHEL 5.8 boxes. Indeed RHEL 5.8 even on ext3 has barriers ON by default. That's good for me.
I tested with my tool and I'm seeing typical performance from 7.2k RPMs disk.
Show
Ronnie Sun
added a comment - btw, the test we were running used +S 16:16 instead of 12:12
Hide
Aleksey Kondratenko
added a comment -
Tested on /data2 partition that's on lvm. And found that barriers actually don't work.
So system tests physical box is NOT ok. Looks like older (or maybe even modern) LVM does not provide durable disk configuration.
So system tests physical box is NOT ok. Looks like older (or maybe even modern) LVM does not provide durable disk configuration.
Show
Aleksey Kondratenko
added a comment - Tested on /data2 partition that's on lvm. And found that barriers actually don't work.
So system tests physical box is NOT ok. Looks like older (or maybe even modern) LVM does not provide durable disk configuration.
Hide
Damien Katz
added a comment -
Mounting evidence suggests this bug is caused by the +A erlang startup settings, which turns on async IO for port drivers by using a pool of threads to perform the IO. We still don't understand the root cause, but it appears there is a race condition/cache coherency problem with how port drivers are freed in the VM.
Reassigning to Alk, as he will check in the change to disable the async threads and bump up the # of schedulers to mitigate problems with timeouts due to blocking IO.
Reassigning to Alk, as he will check in the change to disable the async threads and bump up the # of schedulers to mitigate problems with timeouts due to blocking IO.
Show
Damien Katz
added a comment - Mounting evidence suggests this bug is caused by the +A erlang startup settings, which turns on async IO for port drivers by using a pool of threads to perform the IO. We still don't understand the root cause, but it appears there is a race condition/cache coherency problem with how port drivers are freed in the VM.
Reassigning to Alk, as he will check in the change to disable the async threads and bump up the # of schedulers to mitigate problems with timeouts due to blocking IO.
Show
Aleksey Kondratenko
added a comment - We merged "async off" commit.
Hide
Karen Zeller
added a comment -
Added to RN : Couchbase Server had intermittently crashed
during rebalance due to Erlang virtual machine
issues; we now
disable asynchronous threads and perform garbage collection
more often to avoid timeouts and process crashes.
during rebalance due to Erlang virtual machine
issues; we now
disable asynchronous threads and perform garbage collection
more often to avoid timeouts and process crashes.
Show
Karen Zeller
added a comment - Added to RN : Couchbase Server had intermittently crashed
during rebalance due to Erlang virtual machine
issues; we now
disable asynchronous threads and perform garbage collection
more often to avoid timeouts and process crashes.
Hide
Thuan Nguyen
added a comment -
Integrated in github-couchdb-preview #544 (See [http://qa.hq.northscale.net/job/github-couchdb-preview/544/])
MB-6638: use fork of file_sorter that doesn't use compression (Revision 6bbe1cf89b2f6b5c9cf098b81c5ea60d339f8f0a)
Result = SUCCESS
Aliaksey Kandratsenka :
Files :
* src/couch_set_view/src/couch_set_view_updater.erl
* src/couchdb/couch_db_updater.erl
* src/couchdb/Makefile.am
* Makefile.am
* src/couchdb/file_sorter_2.erl
Result = SUCCESS
Aliaksey Kandratsenka :
Files :
* src/couch_set_view/src/couch_set_view_updater.erl
* src/couchdb/couch_db_updater.erl
* src/couchdb/Makefile.am
* Makefile.am
* src/couchdb/file_sorter_2.erl
Show
Thuan Nguyen
added a comment - Integrated in github-couchdb-preview #544 (See [ http://qa.hq.northscale.net/job/github-couchdb-preview/544/ ])
MB-6638 : use fork of file_sorter that doesn't use compression (Revision 6bbe1cf89b2f6b5c9cf098b81c5ea60d339f8f0a)
Result = SUCCESS
Aliaksey Kandratsenka :
Files :
* src/couch_set_view/src/couch_set_view_updater.erl
* src/couchdb/couch_db_updater.erl
* src/couchdb/Makefile.am
* Makefile.am
* src/couchdb/file_sorter_2.erl
[views:info,2012-09-12T16:57:37.570,ns_1@10.3.121.23:'capi_set_view_manager-saslbucket':capi_set_view_manager:apply_index_states:464]
Calling couch_set_view:add_replica_partitions([<<"saslbucket">>,
<<"_design/d11">>,
[18,19,32,33,34,35,44,45,46,
105,106,107,114,120,152,153,
158,159,160,181,182,198,199,
216,217,224,225,226,227,228,
229,230,242,245,253,276,277,
278,279,280,281,282,283,300,
301,302,303,324,356,357,358,
359,360,361,381,382,383,384,
385,443,444,445,451,452,453,
454,455,463,464,465,466,467,
468,481,482,483,506,507,508,
520,521,522,523,524,527,552,
553,554,565,574,575,576,577,
578,579,591,592,594,595,596,
597,602,603,604,619,620,621,
627,628,629,632,633,634,635,
636,637,656,657,729,730,731,
732,749,750,761,808,809,810,
811,812,836,842,848,849,850,
851,873,874,875,907]])
[ns_server:info,2012-09-12T17:38:36.992,nonode@nohost:<0.60.0>:ns_server:init_logging:198]Started & configured logging
[ns_server:info,2012-09-12T17:38:37.003,nonode@nohost:<0.60.0>:ns_server:log_pending:30]Static config terms: