Membase Cluster malfunction without any warnning
Tue, 02/08/2011 - 21:31
I'm using Membase Server 1.6.5 on Windows Web Server 2008 R2 (x64). I have two Membase server that doing HA and running ok.
I was rebooting the Server 1 without doing any Fail Over today. When this Server 1 rebooted, this node looks like add back to the Membase Cluster without any problem. The Membase Console didn't notice me I have to to Rebalacne or showning any warnning.
The problem I met is that my web frontend was writing cached data into Membase Cluster without error but read nothing from the Membase Cluster. When I was doing a Fail Over on my Server 1, the problem was just solved. Then I add Server 1 back to cluster agagin and doing a Rebalance is ok too.
Is this a bug or something needs to improvement?
Will, I don't think there's any actual problem here...more of a process I think. We can certainly do a better job of alerting as well.
When a server goes down, the client should be receiving failures when trying to write to that server (other nodes in the cluster will continue to function just fine). When you fail it over, the replicas elsewhere in the cluster become active and that data should begin to flow. I think the first issue here is why your client didn't receive any errors even though it should have. We're quite familiar with the Enyim client and I know that it will return errors in this case...is it possible your code wasn't picking up the error correctly?
Our UI (and the REST interface) will actually mark a server as down (or "unhealthy") in this scenario so if you have an external monitoring system, you'll want to set it up to watch for this.
The "Cluster Overview" page on the Membase UI should also have shown that the server was down...please let me know if this isn't the case.
You certainly did the right thing in failing over the server to return access to your data. As I mentioned in the other thread, if you have an expected even where you want to remove the server, it is best to initiate a remove followed by a rebalance.
Let me know if you have any other questions.
Perry
Forum support is great for free but sometimes you need a guaranteed response time and dedicated resources for your questions or issues.
Consider purchasing enterprise-level support from Couchbase: http://www.couchbase.com/products-and-services/overview
Call or email "sales -at- couchbase-dot- com" today!