IP address seems to have changed
Tue, 01/29/2013 - 13:27
Hi,
I've just installed couchbase 2 on four VMware vm's, each vm is running rhel5 and has 4gb ram, 2 CPUs and 10gb disk space. All nodes are on the same network.
Per node I have allocated 3gb for couchbase, I've created a default couchbase bucket of 100mb, a memcached bucket of 100mb and a 2gb couchbase bucket.
My cluster has just been sitting there idle for 2 days but today I got this alert "IP address seems to have changed. Unable to listen on 'ns_1@10.28.80.55'."
It is not possible for the ip to change on any of the nodes, why could I get such an alert?
Regards,
Marvin.
I've looked at the debug logs on host that said it could not listen and found a few things that look interesting, I've no idea what they mean but I'll paste them here in case they are useful.
[ns_server:debug,2013-01-29T17:58:11.549,ns_1@10.28.80.55:compaction_daemon<0.1177.0>:compaction_daemon:handle_info:269]Starting compaction for the following buckets:
[<<"ResultCache">>,<<"default">>]
[ns_server:info,2013-01-29T17:58:11.553,ns_1@10.28.80.55:<0.28389.96>:compaction_daemon:try_to_cleanup_indexes:439]Cleaning up indexes for bucket `ResultCache`
[ns_server:info,2013-01-29T17:58:11.553,ns_1@10.28.80.55:<0.28389.96>:compaction_daemon:spawn_bucket_compactor:404]Compacting bucket ResultCache with config:
[{database_fragmentation_threshold,{30,undefined}},
{view_fragmentation_threshold,{30,undefined}}]
[ns_server:debug,2013-01-29T17:58:11.557,ns_1@10.28.80.55:<0.28392.96>:compaction_daemon:file_needs_compaction:831]Estimated size for `ResultCache`: data 48222, file 4226094
[ns_server:debug,2013-01-29T17:58:11.557,ns_1@10.28.80.55:<0.28392.96>:compaction_daemon:file_needs_compaction:836]Estimated file size for `ResultCache` is less than min_file_size 67239936; skipping
[ns_server:info,2013-01-29T17:58:11.561,ns_1@10.28.80.55:<0.28393.96>:compaction_daemon:try_to_cleanup_indexes:439]Cleaning up indexes for bucket `default`
[ns_server:info,2013-01-29T17:58:11.561,ns_1@10.28.80.55:<0.28393.96>:compaction_daemon:spawn_bucket_compactor:404]Compacting bucket default with config:
[{database_fragmentation_threshold,{30,undefined}},
{view_fragmentation_threshold,{30,undefined}}]
[ns_server:debug,2013-01-29T17:58:11.565,ns_1@10.28.80.55:<0.28396.96>:compaction_daemon:file_needs_compaction:831]Estimated size for `default`: data 47709, file 8428590
[ns_server:debug,2013-01-29T17:58:11.565,ns_1@10.28.80.55:<0.28396.96>:compaction_daemon:file_needs_compaction:836]Estimated file size for `default` is less than min_file_size 67239936; skipping
[ns_server:debug,2013-01-29T17:58:11.565,ns_1@10.28.80.55:compaction_daemon<0.1177.0>:compaction_daemon:handle_info:309]Finished compaction iteration.
[ns_server:debug,2013-01-29T17:58:11.566,ns_1@10.28.80.55:compaction_daemon<0.1177.0>:compaction_daemon:schedule_next_compaction:1200]Finished compaction too soon. Next run will be in 30s
[error_logger:error,2013-01-29T17:58:27.405,ns_1@10.28.80.55:error_logger<0.5.0>:ale_error_logger_handler:log_report:72]
=========================SUPERVISOR REPORT=========================
Supervisor: {local,menelaus_sup}
Context: child_terminated
Reason: {timeout,{gen_server,call,[disksup,get_disk_data,5000]}}
Offender: [{pid,<0.20626.96>},
{name,menelaus_web_alerts_srv},
{mfargs,{menelaus_web_alerts_srv,start_link,[]}},
{restart_type,permanent},
{shutdown,5000},
{child_type,worker}]
[error_logger:info,2013-01-29T17:58:27.405,ns_1@10.28.80.55:error_logger<0.5.0>:ale_error_logger_handler:log_report:72]
=========================PROGRESS REPORT=========================
supervisor: {local,menelaus_sup}
started: [{pid,<0.28472.96>},
{name,menelaus_web_alerts_srv},
{mfargs,{menelaus_web_alerts_srv,start_link,[]}},
{restart_type,permanent},
{shutdown,5000},
{child_type,worker}]
[ns_server:info,2013-01-29T17:58:31.246,ns_1@10.28.80.55:ns_config_rep<0.1075.0>:ns_config_rep:do_pull:341]Pulling config from: 'ns_1@10.28.80.53'
[error_logger:error,2013-01-29T17:58:35.409,ns_1@10.28.80.55:error_logger<0.5.0>:ale_error_logger_handler:log_report:72]
=========================SUPERVISOR REPORT=========================
Supervisor: {local,menelaus_sup}
Context: child_terminated
Reason: {timeout,{gen_server,call,[disksup,get_disk_data,5000]}}
Offender: [{pid,<0.28472.96>},
{name,menelaus_web_alerts_srv},
{mfargs,{menelaus_web_alerts_srv,start_link,[]}},
{restart_type,permanent},
{shutdown,5000},
{child_type,worker}]
[error_logger:info,2013-01-29T17:58:35.409,ns_1@10.28.80.55:error_logger<0.5.0>:ale_error_logger_handler:log_report:72]
=========================PROGRESS REPORT=========================
supervisor: {local,menelaus_sup}
started: [{pid,<0.28501.96>},
{name,menelaus_web_alerts_srv},
{mfargs,{menelaus_web_alerts_srv,start_link,[]}},
{restart_type,permanent},
{shutdown,5000},
{child_type,worker}]
[ns_server:debug,2013-01-29T17:58:38.062,ns_1@10.28.80.55:<0.20634.96>:ns_pubsub:do_subscribe_link:132]Parent process of subscription {buckets_events,<0.20633.96>} exited with reason {timeout,
{gen_server,
call,
[disksup,
get_disk_data,
5000]}}
[error_logger:error,2013-01-29T17:58:38.062,ns_1@10.28.80.55:error_logger<0.5.0>:ale_error_logger_handler:log_msg:76]** Generic server ns_heart terminating
** Last message in was beat
** When Server state == {state,undefined,<0.20634.96>}
** Reason for termination ==
** {timeout,{gen_server,call,[disksup,get_disk_data,5000]}}
[ns_server:error,2013-01-29T17:58:48.867,ns_1@10.28.80.55:<0.28512.96>:menelaus_web_alerts_srv:can_listen:345]Cannot listen due to nxdomain from inet:getaddr
[error_logger:error,2013-01-29T17:58:48.867,ns_1@10.28.80.55:error_logger<0.5.0>:ale_error_logger_handler:log_report:72]
=========================CRASH REPORT=========================
crasher:
initial call: ns_heart:init/1
pid: <0.20633.96>
registered_name: ns_heart
exception exit: {timeout,{gen_server,call,[disksup,get_disk_data,5000]}}
in function gen_server:terminate/6
ancestors: [ns_server_sup,ns_server_cluster_sup,<0.59.0>]
messages: [beat,beat,beat]
links: [<0.1062.0>,<0.20634.96>,<0.56.0>]
dictionary: []
trap_exit: true
status: running
heap_size: 17711
stack_size: 24
reductions: 2258553
neighbours:
[error_logger:error,2013-01-29T17:58:48.867,ns_1@10.28.80.55:error_logger<0.5.0>:ale_error_logger_handler:log_report:72]
=========================SUPERVISOR REPORT=========================
Supervisor: {local,ns_server_sup}
Context: child_terminated
Reason: {timeout,{gen_server,call,[disksup,get_disk_data,5000]}}
Offender: [{pid,<0.20633.96>},
{name,ns_heart},
{mfargs,{ns_heart,start_link,[]}},
{restart_type,permanent},
{shutdown,1000},
{child_type,worker}]
[error_logger:info,2013-01-29T17:58:48.868,ns_1@10.28.80.55:error_logger<0.5.0>:ale_error_logger_handler:log_report:72]
=========================PROGRESS REPORT=========================
supervisor: {local,ns_server_sup}
started: [{pid,<0.28509.96>},
{name,ns_heart},
{mfargs,{ns_heart,start_link,[]}},
{restart_type,permanent},
{shutdown,1000},
{child_type,worker}]
[ns_server:info,2013-01-29T17:58:48.869,ns_1@10.28.80.55:mb_master<0.1109.0>:mb_master:candidate:365]Changing master from 'ns_1@10.28.80.53' to 'ns_1@10.28.80.54'
[user:info,2013-01-29T17:58:48.869,ns_1@10.28.80.55:<0.28512.96>:menelaus_web_alerts_srv:global_alert:87]IP address seems to have changed. Unable to listen on 'ns_1@10.28.80.55'.
[ns_server:warn,2013-01-29T17:58:48.879,ns_1@10.28.80.55:mb_master<0.1109.0>:mb_master:handle_info:204]Skipped 3 heartbeats
[ns_server:debug,2013-01-29T17:58:48.891,ns_1@10.28.80.55:compaction_daemon<0.1177.0>:compaction_daemon:handle_info:269]Starting compaction for the following buckets:
[<<"ResultCache">>,<<"default">>]
[ns_server:info,2013-01-29T17:58:48.895,ns_1@10.28.80.55:<0.28520.96>:compaction_daemon:try_to_cleanup_indexes:439]Cleaning up indexes for bucket `ResultCache`
[ns_server:info,2013-01-29T17:58:48.897,ns_1@10.28.80.55:ns_config_rep<0.1075.0>:ns_config_rep:do_pull:341]Pulling config from: 'ns_1@10.28.80.53'
[ns_server:error,2013-01-29T17:58:48.897,ns_1@10.28.80.55:ns_memcached-ResultCache<0.13213.0>:ns_memcached:handle_info:630]handle_info(ensure_bucket,..) took too long: 29769320 us
[ns_server:error,2013-01-29T17:58:48.898,ns_1@10.28.80.55:<0.13217.0>:ns_memcached:verify_report_long_call:297]call {stats,<<>>} took too long: 30653083 us
[ns_server:error,2013-01-29T17:58:48.898,ns_1@10.28.80.55:ns_memcached-default<0.1298.0>:ns_memcached:handle_info:630]handle_info(ensure_bucket,..) took too long: 25436374 us
[ns_server:error,2013-01-29T17:58:48.899,ns_1@10.28.80.55:<0.1303.0>:ns_memcached:verify_report_long_call:297]call {stats,<<>>} took too long: 30654044 us
[ns_server:info,2013-01-29T17:58:48.928,ns_1@10.28.80.55:<0.28520.96>:compaction_daemon:spawn_bucket_compactor:404]Compacting bucket ResultCache with config:
[{database_fragmentation_threshold,{30,undefined}},
{view_fragmentation_threshold,{30,undefined}}]
[stats:warn,2013-01-29T17:58:48.938,ns_1@10.28.80.55:<0.13224.0>:stats_collector:latest_tick:223]Dropped 12 ticks
[stats:warn,2013-01-29T17:58:48.946,ns_1@10.28.80.55:<0.1317.0>:stats_collector:latest_tick:223]Dropped 12 ticks
[ns_server:error,2013-01-29T17:58:49.093,ns_1@10.28.80.55:ns_doctor<0.1095.0>:ns_doctor:update_status:205]The following buckets became not ready on node 'ns_1@10.28.80.55': ["ResultCache"], those of them are active ["ResultCache"]
[ns_server:debug,2013-01-29T17:58:49.110,ns_1@10.28.80.55:<0.28545.96>:compaction_daemon:file_needs_compaction:831]Estimated size for `ResultCache`: data 47869, file 4226094
[ns_server:debug,2013-01-29T17:58:49.110,ns_1@10.28.80.55:<0.28545.96>:compaction_daemon:file_needs_compaction:836]Estimated file size for `ResultCache` is less than min_file_size 67239936; skipping
[ns_server:info,2013-01-29T17:58:49.116,ns_1@10.28.80.55:<0.28561.96>:compaction_daemon:try_to_cleanup_indexes:439]Cleaning up indexes for bucket `default`
[ns_server:info,2013-01-29T17:58:49.129,ns_1@10.28.80.55:ns_doctor<0.1095.0>:ns_doctor:update_status:211]The following buckets became ready on node 'ns_1@10.28.80.55': ["ResultCache"]
[ns_server:info,2013-01-29T17:58:49.131,ns_1@10.28.80.55:<0.28561.96>:compaction_daemon:spawn_bucket_compactor:404]Compacting bucket default with config:
[{database_fragmentation_threshold,{30,undefined}},
{view_fragmentation_threshold,{30,undefined}}]
[ns_server:debug,2013-01-29T17:58:49.135,ns_1@10.28.80.55:<0.28569.96>:compaction_daemon:file_needs_compaction:831]Estimated size for `default`: data 47709, file 8428590
[ns_server:debug,2013-01-29T17:58:49.135,ns_1@10.28.80.55:<0.28569.96>:compaction_daemon:file_needs_compaction:836]Estimated file size for `default` is less than min_file_size 67239936; skipping
[ns_server:debug,2013-01-29T17:58:49.135,ns_1@10.28.80.55:compaction_daemon<0.1177.0>:compaction_daemon:handle_info:309]Finished compaction iteration.
[ns_server:debug,2013-01-29T17:58:49.136,ns_1@10.28.80.55:compaction_daemon<0.1177.0>:compaction_daemon:schedule_next_compaction:1200]Finished compaction too soon. Next run will be in 29s
[stats:warn,2013-01-29T17:58:51.501,ns_1@10.28.80.55:system_stats_collector<0.1168.0>:system_stats_collector:handle_info:133]lost 1 ticks
[stats:warn,2013-01-29T17:58:51.502,ns_1@10.28.80.55:<0.1317.0>:stats_collector:latest_tick:223]Dropped 20 ticks
[stats:warn,2013-01-29T17:58:51.504,ns_1@10.28.80.55:<0.13224.0>:stats_collector:latest_tick:223]Dropped 20 ticks
[stats:warn,2013-01-29T17:58:51.505,ns_1@10.28.80.55:<0.16567.0>:stats_collector:latest_tick:223]Dropped 19 ticks