This morning one of our EC2 instances was beginning to have issues. I set up a new node and initiated a swap rebalance from the CLI (add new server + remove issue server). During the run of that operation I got
“(500) Internal Server Error
[“Unexpected server error, request logged.”]”
However looking in the GUI I could see that the new server had been added and was pending rebalance. Figuring that it hadn’t worked I clicked remove on the issue server and triggered a rebalance to start the swap rebalance that way. I again got an server error, this time in the GUI. Except this time the new node was added to the cluster and rebalancing became impossible to start.
At this point I tried to failover the issue server, as the new server was healthy and the old server was showing as “pending”. The GUI then greyed out the rebalance button and a progress bar marked “0% complete” showed up on each server node line.
Now when I can no longer manipulate servers/rebalancing in the GUI and when attempting to complete the failover from the CLI I simply get
ERROR: unable to failover ns_1@nodename.server.com (500) Internal Server Error
[“Unexpected server error, request logged.”]