Issues adding a new node to existing cluster
Our current cluster has 2 nodes with 8GB ram and 3.1ghz processors, both with 1TB hdds and 6GB ram allocated to Couchbase per node. Both nodes are running Couchbase 2.0.0.
I've installed Couchbase 2.0.1 on a new node with the same specifications but with 500GB hdd and it hasn't gone as planned.
We have various web servers connecting to Couchbase for session data and cached data and they've each been putting one of the following errors into the /var/log/apache2/error.log file throughout the time I was attempting to add this new node in;
[Thu May 16 15:42:26 2013] [notice] child pid 5412 exit signal Aborted (6)
apache2: src/instance.c:542: relocate_packets: Assertion `ringbuffer_read(&src->pending_cookies, &ct, sizeof(ct)) == sizeof(ct)' failed.
[Thu May 16 15:41:20 2013] [notice] child pid 22333 exit signal Segmentation fault (11)
We were getting "Empty Response" messages in the browser for the second error, the first one didn't make any noticeable problems but did stop when I stopped adding the node in.
I added the node to the cluster and started to rebalance which had mixed results as it was either incredibly slow or was hanging. I was able to stop rebalancing and start again which enabled me to get another 10-20MB transferred over but if I left it hanging it would do 1MB every 5mins or so which for 12GB in the cluster would take a very long time.
Because of the issues we then had on the web servers I stopped the rebalance, removed the node from the cluster and performed another rebalance which took about 10 minutes to complete to get the node removed.
Everything is back to normal and stable now but as far as I can tell this isn't supposed to happen? I thought it was supposed to be capable of adding and removing nodes without any problems whatsoever? Irrespective of the issues on the web servers, surely it isn't supposed to take so long to add a new node into the cluster?
Finally, we have the nodes in the cluster on their external IPs; would it be possible to get them to use their local IPs as we have all of our servers on a virtual LAN and would probably make transferring data around much faster? Preferably without incurring any downtime if we can!
Appreciate any help and advice.