[system test] "Failed to grab remote bucket info from any of known nodes">> when create remote xdcr setup

Junyi Xie added a comment - 30/Jan/13 9:02 PM Aliaskey, why shall we see the badmatch error from within remote_cluster_info when the buckets on both sides are empty (the replication is created)? Thanks.

Farshid Ghods added a comment - 30/Jan/13 11:00 PM Tony, did you wait long enough after bucket creation to create xdcr replication stream ?

Thuan Nguyen added a comment - 31/Jan/13 12:01 AM I think I wait more than 5 mins to setup xdcr

Ketaki Gangal added a comment - 31/Jan/13 11:32 AM Yes, waited / recreated this issue multiple times on same setup.

Aliaksey Artamonau added a comment - 31/Jan/13 1:24 PM Here's what I see on the source cluster: [ns_server:error,2013-01-30T17:57:12.396, ns_1@cen-2501.hq.couchbase.com :<0.21948.1>:remote_clusters_info:mk_json_get:987]Request to http://Administrator:****@10.3.3.7:8091/pools/default?uuid=732af0336139bab90d477ff8367efa93 failed: {error,rest_error,        <<"Timeout connecting to \"10.3.3.7\" on port 8091. This could be due to an incorrect host/port combination or a firewall in place between the servers.">>,        {error,timeout}} [ns_server:info,2013-01-30T17:57:12.396, ns_1@cen-2501.hq.couchbase.com :<0.21948.1>:remote_clusters_info:remote_bucket_from_nodes:727]Failed to get bucket info from node {remote_node,"10.3.3.7",8091}: {error,rest_error,        <<"Timeout connecting to \"10.3.3.7\" on port 8091. This could be due to an incorrect host/port combination or a firewall in place between the servers.">>,        {error,timeout}} [ns_server:error,2013-01-30T17:57:13.455, ns_1@cen-2501.hq.couchbase.com :<0.21932.1>:remote_clusters_info:mk_json_get:987]Request to http://Administrator:****@10.3.3.9:8091/pools/default?uuid=732af0336139bab90d477ff8367efa93 failed: {error,rest_error,        <<"Timeout connecting to \"10.3.3.9\" on port 8091. This could be due to an incorrect host/port combination or a firewall in place between the servers.">>,        {error,timeout}} [ns_server:info,2013-01-30T17:57:13.455, ns_1@cen-2501.hq.couchbase.com :<0.21932.1>:remote_clusters_info:remote_bucket_from_nodes:727]Failed to get bucket info from node {remote_node,"10.3.3.9",8091}: {error,rest_error,        <<"Timeout connecting to \"10.3.3.9\" on port 8091. This could be due to an incorrect host/port combination or a firewall in place between the servers.">>,        {error,timeout}} So it really cannot reach any of the destination nodes. Let me try to figure out what happens there.

Aliaksey Artamonau added a comment - 31/Jan/13 5:45 PM On the destination cluster I see a bunch of timeouts about the same time when replication was created: [menelaus:warn,2013-01-30T17:37:04.312, ns_1@10.3.3.9 :<0.21026.229>:menelaus_web:loop:430]Server error during processing: ["web request failed",                                  {path,"/pools/default"},                                  {type,exit},                                  {what,                                   {timeout,                                    {gen_server,call,                                     [ns_node_disco,nodes_wanted]}}},                                  {trace,                                   [{gen_server,call,2},                                    {ns_cluster_membership,active_nodes,1},                                    {ns_storage_conf,cluster_storage_info,0},                                    {menelaus_web,build_pool_info,4},                                    {menelaus_web,handle_pool_info,2},                                    {menelaus_web,loop,3},                                    {mochiweb_http,headers,5},                                    {proc_lib,init_p_do_apply,3}]}] [menelaus:warn,2013-01-30T17:37:04.296, ns_1@10.3.3.9 :<0.21086.229>:menelaus_web:loop:430]Server error during processing: ["web request failed",                                  {path,"/pools/default"},                                  {type,exit},                                  {what,                                   {timeout,                                    {gen_server,call,                                     [ns_node_disco,nodes_wanted]}}},                                  {trace,                                   [{gen_server,call,2},                                    {menelaus_web,build_pool_info,4},                                    {menelaus_web,handle_pool_info,2},                                    {menelaus_web,loop,3},                                    {mochiweb_http,headers,5},                                    {proc_lib,init_p_do_apply,3}]}] [menelaus:warn,2013-01-30T17:37:04.313, ns_1@10.3.3.9 :<0.21012.229>:menelaus_web:loop:430]Server error during processing: ["web request failed",                                  {path,"/pools/default"},                                  {type,exit},                                  {what,                                   {timeout,                                    {gen_server,call,                                     [ns_node_disco,nodes_wanted]}}},                                  {trace,                                   [{gen_server,call,2},                                    {ns_cluster_membership,active_nodes,1},                                    {ns_storage_conf,cluster_storage_info,0},                                    {menelaus_web,build_pool_info,4},                                    {menelaus_web,handle_pool_info,2},                                    {menelaus_web,loop,3},                                    {mochiweb_http,headers,5},                                    {proc_lib,init_p_do_apply,3}]}] And this is linux box where we have not seen this issue recently. Can we test if this is reproducible? And please collect cpu delays info while reproducing.

Ketaki Gangal added a comment - 06/Feb/13 11:39 AM Hi Tony, On the next run, can you repro this? If yes, can you add information on cpu delays info while reproducing. -Ketaki

Aliaksey Artamonau added a comment - 06/Feb/13 7:01 PM MB-7697

Ketaki Gangal added a comment - 06/Feb/13 7:10 PM Hi TOny, Are these failing on 2.0 or 2.0.1? -Ketaki

Thuan Nguyen added a comment - 06/Feb/13 7:42 PM On 2.0.0