couchbase 1.8 in ec2 with external hostname work and custom hostnames fails
we are evaluating couchbase in ec2 with intent of deploying a large couchbase cluster in ec2. So far, if we edited the /opt/couchbase/bin/couchbase-server and add ec2 external hostname, then start the cluster. Everything works fine. we are able to have a 4 node cluster working.
But seeing that ec2 external hostname does not persist, we want to use custom hostname such as couchbaseprodnode-*. But instead of using a DNS service, we have the /etc/hosts. So that we can just update the /etc/hosts file when the cluster topology changes. /etc/hosts is as follow:
127.0.0.1 localhost.localdomain localhost
But this does not work. our client side moxi got a down-stream conn error even with a 1 node cluster. and if we tried to add nodes to the cluster, the rebalance fails. And the cluster goes into a busy state and not accessible thru the web console. and thru couchbase-cli, we got inconsistence data depending on what node we run couchbase-cli.
any help is appreciated. Is there a configuration step we missed. Let me know if more info would help debug this issue.