Couchbase logging 100's of warnings in eventlog and websites are down....CouchbaseServer: Restarted erlang machine.
I've got two nodes running, and have been running for some time. Today I needed to create three additional buckets. The first two went in fine however after adding the third, couchbase hit a tree, fell over and burst into flames. The control panel indicated one node was pending, i refreshed the console then both servers pending. I restarted one server process and that node displayed as down. Restarting the other did nothing - it showed as pending. I then could not access the control panel. The websites that relied on couchbase went down (still trying to figure that one out!). restarting the physical servers did nothing to resolve the problem.
Examining the event log on both of the node servers shows the following that was logged every few seconds..
CouchbaseServer: Restarted erlang machine.
CouchbaseServer: Erlang service restarted
Over and over these messages are logged and no indication of what the problem is.
Any help is appreciated.