Adding server node breaks cluster
We are running a two-node cluster and upon trying to add a third node through the user interface the cluster goes into an unconfigured state. The logs show
Conflicting configuration changes to field nodes_wanted:
[{'_vclock',[{'ns_1@10.0.1.41',{1,63503398676}}]},'ns_1@10.0.1.41'] and
[{'_vclock',[{'ns_1@10.0.2.41',{3,63503398676}},
{'ns_1@10.0.3.41',{3,63498365113}}]},
'ns_1@10.0.1.41','ns_1@10.0.2.41','ns_1@10.0.3.41'], choosing the former.
I tried this on two new machines with Couchbase server 1.8 and the same thing happened- rather than seeing a new node successfully join the cluster, I am redirected to the cluster configuration screen. From those logs we have
Conflicting configuration changes to field nodes_wanted:
[{'_vclock',[{'ns_1@10.0.2.43',{1,63503642427}}]},'ns_1@10.0.2.43'] and
[{'_vclock',[{'ns_1@10.0.3.43',{2,63503642427}}]},
'ns_1@10.0.2.43','ns_1@10.0.3.43'], choosing the former.
These machines were all started from the same AWS image and no additional configuration had been performed on the two nodes that already exist, except for the fact that they were in a cluster. It seems very undesirable for one node to hijack the cluster configuration without any prompting.
This is likely because they are all from the same AWS image and the configuration had been left there. As a result, the configuration is getting confused and it's trying to automatically resolve the config.
When cloning an OS configuration with Couchbase Server installed, you'll need to remove, at a minimum, the config and data directories from the installation. On linux this is typically under /opt/couchbase/var/lib/couchbase