XDCR - adding remote cluster fails

Hello Elfar,

It sounds like you added the EIP after the cluster was established and tried just using that when setting up XDCR. Unfortunately, the EIP's must be configured for all nodes, and the cluster itself must be configured to use them.

The reason it fails when you don't do that is that the bucket map contains the wrong addreses (internal only ones). The only way it can work is if you configure the cluster (during initial set up) with the DNS names of the Elastic IPs. This way they resolve correctly to both the internal and external addresses respectively.

This section of the manual has some additional information about using Couchbase XDCR in cloud environments like EC2:

http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-admin-tasks...

Also, this blog I wrote goes through the necessary steps:

http://blog.couchbase.com/cross-data-center-replication-step-step-guide-...

The blog entry doesn't use Elastic IPs, but the concept is the same. You must use a DNS name that correctly resolves to the internal IP within the region, and to a public IP anywhere else.

Thanks for the input,

I thought I removed all relevant files and went through the setup process via the GUI again after I added the EIP (only one instance in this cluster). Had not seen this blog link you sent, will carefully try to follow that and see how that works out.

One thing though, what I wrote last in my post, the test I did with a strictly local ec2 cluster (with two instances) using a local instance. That cluster is not using EIP nor DNS (it was an independent test to exclude EIP/DNS issues), just normal ec2 and testing from another instance in same account and region (firewall open for all ports), should this not have given me some reply besides an empty []?

curl -v -u Administrator:********** http://private_ip_from_a_instance_in_same_region:8091/pools/default/remo...

Also on a side note, if I have cluster with two instances Private_IP_1 and Private_IP_2 does it matter what IP I use, and is there some smart way to make sure it uses IP_2 if IP_1 fails or should I simply setup some load balancing stuff to make sure I can always reach the cluster, after failover, if either goes down?

Appreciate your thoughts.

Elfar

Hi again,

here is an update on my efforts. As a test and to exclude any DNS issues etc. I am working with EC2 instances on the same account and region.

Under XDCR tab I insert:

Cluster Name: Bookingtest
IP/hostname: PRIVATE_EC2_IP:8091

and the correct user/pass for that 1-node cluster I am trying to add to a 2-node cluster.

I get this error still:

Server error during processing: ["web request failed",
{path,"/pools/default/remoteClusters"},
{type,error},
{what,
{case_clause,
{error,
{case_clause,false},
[{ns_config,'-update_key/2-fun-0-',3},
{ns_config,handle_call,3},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]}}},
{trace,
[{menelaus_web_remote_clusters,
cas_remote_clusters,2},
{menelaus_web_remote_clusters,
do_handle_remote_clusters_post,4},
{menelaus_web,loop,3},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]}]

Command line I did some testing suggested in your docs, from the the same 2-node cluster to the same 1-node cluster, like:

curl -v -u Administrator:********** http://PRIVATE_EC2_IP:8091/pools/default/remoteClusters and I get an empty []

* About to connect() to THE_IP port 8091 (#0)
* Trying THE_IP... connected
* Server auth using Basic with user 'Administrator'
> GET /pools/default/remoteClusters HTTP/1.1
> Authorization: Basic QWRtaW5pc3RyYXRvcjpvbmU0dGhlM29hZA==
> User-Agent: curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: THE_IP:8091
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: Couchbase Server 2.0.0-1976-rel-enterprise
< Pragma: no-cache
< Date: Fri, 01 Mar 2013 08:58:05 GMT
< Content-Type: application/json
< Content-Length: 2
< Cache-Control: no-cache
<
* Connection #0 to host THE_IP left intact
* Closing connection #0
[]

Running:

curl -v -u Administrator:********** http://PRIVATE_EC2_IP:8091/pools/default/buckets

gives me nice json with lot of data so nothing wrong with the curl command per se.

Am I going about this the wrong way or? Just installed latest 2.0.0 enterprise on ubuntu 12.04, no changes in the setup and only setup the default bucket, no other buckets.

Regards,

Elfar

Elfar,

On the XDCR screen you do NOT need to point to port 8091, just use the internal IP address of 1 of the nodes in the remote / destination cluster directly, Couchbase takes care of the rest. If you are in the same zone, you don't have to worry about public IPs, just use the private IPs. it will work fine. I've tried it several times.

Cluster Name: Bookingtest
IP/hostname: 10.34.12.XXX

Here's a screenshot just in case.
https://www.evernote.com/shard/s161/sh/9a431355-68ee-4871-9c24-f8fffe79d...

Thanks,

nut dropping the 8091 changed nothing, still the same error with using private IP only

Server error during processing: ["web request failed",
{path,"/pools/default/remoteClusters"},
{type,error},
{what,
{case_clause,
{error,
{case_clause,false},
[{ns_config,'-update_key/2-fun-0-',3},
{ns_config,handle_call,3},
{gen_server,handle_msg,5},
{proc_lib,init_p_do_apply,3}]}}},
{trace,
[{menelaus_web_remote_clusters,
cas_remote_clusters,2},
{menelaus_web_remote_clusters,
do_handle_remote_clusters_post,4},
{menelaus_web,loop,3},
{mochiweb_http,headers,5},
{proc_lib,init_p_do_apply,3}]}]

and note that the path is /pools/default/remoteClusters in that error call, and the fact that I get an empty json when trying that path command line.

Both CB work fine and I can access them and store data, just cannot create a cluster reference between those 2 small test clusters of 2 and 1 nodes, with only default bucket.

Your screenshot was gone when I finally got to this, which Linux and CB were you running?

Regards,

Elfar

OK, interesting, just tried it the other way, from 1-node cluster to the 2-node cluster, that worked! Is it possible I need at least 2 nodes in a cluster for this to work or?

-elfar

In general, XDCR is a a more intensive operation than intra-cluster replication. Minimum recommended setup is to use two nodes -> two nodes.

That said, the default number of streams connected in parallel is 32. But this can be tuned per your hardware. If you are simply trying it on large EC2 instances you will likely have to reduce this number.

XDCR is extremely scalable and the more nodes you have the better the performance. We have seen XDCR throughput at 50-100K ops / sec on 9-15 node EC2 clusters.

take a look at the tunables here.

http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-admin-resta...

hope this helps.

OK, thanks for the tunables link. I was using a 2-node large instance cluster and seeing if my medium instance 1 node cluster for the developers could receive the live data from the 2-nodes so they could use that for testing with live data (the traffic is not heavy on this cluster). Guess 2-large -> 1-medium became too much to handle. Thanks for your replies, appreciate it.

-elfar