couchbase 1.8 in ec2 with external hostname work and custom hostnames fails
we are evaluating couchbase in ec2 with intent of deploying a large couchbase cluster in ec2. So far, if we edited the /opt/couchbase/bin/couchbase-server and add ec2 external hostname, then start the cluster. Everything works fine. we are able to have a 4 node cluster working.
But seeing that ec2 external hostname does not persist, we want to use custom hostname such as couchbaseprodnode-*. But instead of using a DNS service, we have the /etc/hosts. So that we can just update the /etc/hosts file when the cluster topology changes. /etc/hosts is as follow:
127.0.0.1 localhost.localdomain localhost
10.92.243.56 couchbasenode-2
10.90.251.180 couchbasenode-4
10.124.139.211 couchbasenode-1
10.123.45.244 couchbasenode-3
But this does not work. our client side moxi got a down-stream conn error even with a 1 node cluster. and if we tried to add nodes to the cluster, the rebalance fails. And the cluster goes into a busy state and not accessible thru the web console. and thru couchbase-cli, we got inconsistence data depending on what node we run couchbase-cli.
any help is appreciated. Is there a configuration step we missed. Let me know if more info would help debug this issue.
Is there no way to config couchbase to use /etc/hosts? then we would not need a dynamic DNS or EIPs. also, how can I contact you if I want to discuss this issue further?
thanks.
hi paulroo,
I'm so sorry I didn't get back to you on this. A colleague just pointed it out to me.
The challenge here is that Couchbase Server will want a consistent way for the hosts to be seen across multiple nodes. Whether you use /etc/hosts or DNS or NIS or something else it doesn't really matter. What does matter is that the systems have a consistent way of advertising themselves to each other. In some cases, you may need to change the way couchbase starts up to tell it to tell other servers it's a particular hostname or IP, but we built it such that in most cases if you have consistent IP addresses and hostnames, it'll automatically do the right thing.
Feel free to contact me (matt at couchbase) if stuck-- I'll keep a better eye on the forum here. Sorry for the long delay.
Matt
Given that IPs can change unless they're elastic IPs, we'd usually recommend setting up a dynamic DNS service or using EIPs and associated hostnames. Personally, I prefer some kind of dynamic DNS to EIP, since Amazon's recommendations say use EIPs for things you generally loadbalance across.
I know there are a number of possible configurations-- I'm happy to talk through this further if this answer didn't quite hit what you were looking for.