Details
Description
there are many steps to reproduce a set of such errors but the simplest is:
on the same cluster create bucket replication
for example, in attached logs replication was from 'standard_bucket0' to "bucket0"
Server error during processing: ["web request failed",
{path,"/pools/default"},
{type,exit},
{what,
{timeout,
{gen_server,call,
[ns_cookie_manager,cookie_get]}}},
{trace,
[{gen_server,call,2},
{menelaus_web,build_nodes_info_fun,3},
{menelaus_web,build_pool_info,4},
{menelaus_web,handle_pool_info,2},
{menelaus_web,loop,3},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]}] (repeated 77 times) menelaus_web019 ns_1@10.3.121.63 02:23:08 - Wed Nov 28, 2012
Server error during processing: ["web request failed",
{path,"/pools/default"},
{type,exit},
{what,
{timeout,
{gen_server,call,
[ns_cookie_manager,cookie_get]}}},
{trace,
[{gen_server,call,2},
{menelaus_web,build_nodes_info_fun,3},
{menelaus_web,build_pool_info,4},
{menelaus_web,handle_pool_info_wait,5},
{menelaus_web,loop,3},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]}] menelaus_web019 ns_1@10.3.121.63 02:22:40 - Wed Nov 28, 2012
Haven't heard from a higher priority node or a master, so I'm taking over. mb_master000 ns_1@10.3.121.64 02:22:38 - Wed Nov 28, 2012
Server error during processing: ["web request failed",
{path,"/pools/default/buckets"},
{type,exit},
{what,
{timeout,
{gen_server,call,
[ns_cookie_manager,cookie_get]}}},
{trace,
[{gen_server,call,2},
{menelaus_web,build_nodes_info_fun,3},
{menelaus_web_buckets,
build_bucket_node_infos,5},
{menelaus_web_buckets,build_bucket_info,5},
{menelaus_web_buckets,
'-handle_bucket_list/2-lc$^0/1-0-',3},
{menelaus_web_buckets,
'-handle_bucket_list/2-lc$^0/1-0-',3},
{menelaus_web_buckets,handle_bucket_list,2},
{menelaus_web,loop,3}]}] menelaus_web019 ns_1@10.3.121.63 02:22:37 - Wed Nov 28, 2012
Server error during processing: ["web request failed",
{path,"/pools/default"},
{type,exit},
{what,
{timeout,
{gen_server,call,
[ns_cookie_manager,cookie_get]}}},
{trace,
[{gen_server,call,2},
{menelaus_web,build_nodes_info_fun,3},
{menelaus_web,build_pool_info,4},
{menelaus_web,handle_pool_info,2},
{menelaus_web,loop,3},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]}] menelaus_web019 ns_1@10.3.121.63 02:22:37 - Wed Nov 28, 2012
on the same cluster create bucket replication
for example, in attached logs replication was from 'standard_bucket0' to "bucket0"
Server error during processing: ["web request failed",
{path,"/pools/default"},
{type,exit},
{what,
{timeout,
{gen_server,call,
[ns_cookie_manager,cookie_get]}}},
{trace,
[{gen_server,call,2},
{menelaus_web,build_nodes_info_fun,3},
{menelaus_web,build_pool_info,4},
{menelaus_web,handle_pool_info,2},
{menelaus_web,loop,3},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]}] (repeated 77 times) menelaus_web019 ns_1@10.3.121.63 02:23:08 - Wed Nov 28, 2012
Server error during processing: ["web request failed",
{path,"/pools/default"},
{type,exit},
{what,
{timeout,
{gen_server,call,
[ns_cookie_manager,cookie_get]}}},
{trace,
[{gen_server,call,2},
{menelaus_web,build_nodes_info_fun,3},
{menelaus_web,build_pool_info,4},
{menelaus_web,handle_pool_info_wait,5},
{menelaus_web,loop,3},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]}] menelaus_web019 ns_1@10.3.121.63 02:22:40 - Wed Nov 28, 2012
Haven't heard from a higher priority node or a master, so I'm taking over. mb_master000 ns_1@10.3.121.64 02:22:38 - Wed Nov 28, 2012
Server error during processing: ["web request failed",
{path,"/pools/default/buckets"},
{type,exit},
{what,
{timeout,
{gen_server,call,
[ns_cookie_manager,cookie_get]}}},
{trace,
[{gen_server,call,2},
{menelaus_web,build_nodes_info_fun,3},
{menelaus_web_buckets,
build_bucket_node_infos,5},
{menelaus_web_buckets,build_bucket_info,5},
{menelaus_web_buckets,
'-handle_bucket_list/2-lc$^0/1-0-',3},
{menelaus_web_buckets,
'-handle_bucket_list/2-lc$^0/1-0-',3},
{menelaus_web_buckets,handle_bucket_list,2},
{menelaus_web,loop,3}]}] menelaus_web019 ns_1@10.3.121.63 02:22:37 - Wed Nov 28, 2012
Server error during processing: ["web request failed",
{path,"/pools/default"},
{type,exit},
{what,
{timeout,
{gen_server,call,
[ns_cookie_manager,cookie_get]}}},
{trace,
[{gen_server,call,2},
{menelaus_web,build_nodes_info_fun,3},
{menelaus_web,build_pool_info,4},
{menelaus_web,handle_pool_info,2},
{menelaus_web,loop,3},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]}] menelaus_web019 ns_1@10.3.121.63 02:22:37 - Wed Nov 28, 2012
is the cluster still around ? junyi can log in and take a look