Cloud - Inter availability zone couchbase cluster
Hello,
Please note that following setup WORKS with Couchbase 1.8.0, but doesn't NOT working 1.8.1 and 2.0.0
I'll start from beginning, we have couchbase in OpenStack cloud.
Our provider gives us two availability zones, we have a cluster of just 2 nodes, one in each zone.
First problem is that nodes can communicate only via floating IPs between zones and not the local ones.
I found advice on couchbase site somewhere on how to switch couchbase to use hostname and not IPs, which I implemented but that was still no go. I was getting errors while trying to join second host saying that it's unable to hear on . This seemed logical as hostname I was using was resolving to floating IP which is not on any of the interfaces on those hosts. Then I added hostname to 127.0.0.1 mapping to /etc/hosts and that allowed me to build a cluster ok. Than in version 1.8.0 from then it was all good, I could re-balance, failover etc, all was good. Problem is that in 1.8.1 and 2.0.0 that setup doesn't work. While I can join / create cluster of 2 nodes with above setup, I cannot re-balance, this always fails. So question is what changed in re-balancing between versions 1.8.0 and 1.8.1 / 2 ?
In 2.0 I am always ending up with this error:
Rebalance exited with reason {unexpected_exit,
{'EXIT',<0.15964.0>,
{{badmatch,
[{'EXIT',
{timeout,
{gen_server,call,
[<15350.1961.0>,had_backfill,30000]}}}]},
[{ns_single_vbucket_mover,
'-wait_backfill_determination/1-fun-1-',
1}]}}}
I've looked around but didn't find similar issue.
Any help / advice greatly appreciated.
I found the problem, port 11209 was not open, fact that it worked with 1.8.0 was a bit confusing here.