Any good example of java code that handles node fault?

-Does the exception go away if you fail over the server?
-Which server's URI are you pointing the client at? Does it make a difference if you point it at the "other" one and/or both of them?

Perry

note, cross posted here: http://stackoverflow.com/questions/6360006/membase-node-fault-handeling-...

Hi Perry,

>>Does the exception go away if you fail over the server?
If I manually fail over the server it throws some exceptions, but afterwards if switches to another server and keep operations. Maybe some data is lost, but I can adjust my app to handle temporal connection loss (in from of my app is rabbitMQ), so it is ok. In the log file you will see that at first I tried to fail over one server, then I brought it back and rebalanced and after I did the same for another server.

Here is a link to my exceptions log: https://rapidshare.com/files/1933999309/log.txt

>>Which server's URI are you pointing the client at? Does it make a difference if you point it at the "other" one and/or both of them?
I have to servers in my test cluster. One is 192.168.1.10 and another one 192.168.1.9.

In my test java code I connect to them using this code:

MemcachedClient c = new MemcachedClient(new BinaryConnectionFactory(), AddrUtil.getAddresses("192.168.1.9:11211 192.168.1.10:11211"));

after that I just do get and set in a loop with few million iterations.

Please, provide me with more details because for production everything must be super stable.

Perry, thank you advance!
Max

Hi Max,

As you can imagine, there are a few things at play here, depending on the state the cluster is in. There are a few different failure modes, so the client behavior will depend on what is happening at the server side.

To describe the scenario it looks like you were working on, before a failure, all data generally flows just fine to the servers of course. Even in this state though, the client can give failures back to your requests if, for instance, a network interruption bothered one of the operations and it didn't happen within the timeout period.

When a server fails, we either see the failure immediately, or we see operations start timing out. That's the nature of TCP. After a certain threshold of timeouts, we consider the server to have failed and attempt to reconnect.

During this time, you will see operation failures for things destined for the down server.

After the node is failed over (either through clicking failover in the console or through auto-failover, depending on the version you are deploying, etc.), the client will know what the new location should be for any given item. Any new requests will then go to the new location for that given item.

Since this change isn't atomic, the client can receive, for a period of time, responses from a server indicating that it's not responsible for a given item. This is the source of the "not my vbucket" log messages. When the client receives these from a given server, it will automatically retry the operation as long as it has not passed the timeout threshold. Once it's hit the timeout, your request of the client would see the failure.

Fundamentally, when using the client, you'll either see the operation succeed or you'll see it fail if the given node responsible for that item is unavailable. Once you initiate a failover, within a second or less, all requests should flow just fine.

Does that help?

Thanks,

Matt

Hi ingenthr!

Thank you very much for your reply! It is very interesting to know some deep details about membase!

Unfortunately your answer didn't answer my question 100%. It is great to know a theory behind membase. But my question was about why build in "Failover" function works as expected while manual shutdown of a random membase node crushes client application. See my post above for deeper details.

Moreover, you guys suggest to use spymemcached as a standard client library, but guys from spymemcached also didn't give an answer on the same question: http://code.google.com/p/spymemcached/issues/detail?id=181&colspec=stars...

So, once again. My question is very determined: How I can properly handle node failure in my application? Do you have any documentation about it or best practice approach?

Of course, it is clear that every application is very specific but there must be a single robust way to handle a very general case of a random node failure.

This is what I am asking for.

Anyway, thank you very much for the time you spent answering my question. You theoretical explanation gave me some ideas... So I will try something our, but good manual still missing. And it is missing not only by me but I bet by many other developers which are searching among many databases available on the web now. If you would provide something like I ask - i believe it will be very good point to consider membase as a basis datastore for many-many developers over the world.

Hi,

You're mostly asking questions about server behavior rather than client behavior, actually. The way Membase works, if a given node in the cluster fails, then you can't perform any operations against it. I'm not certain I understand what you're saying though when you say the "client crushes". Do you mean crashes? You will receive exceptions from the client on operations bound for the down node, but other operations will go through just fine until you fail over the node.

As far as "you guys", you may notice if you look closely at the spymemcached project, it's the same guys. ;) Will update your bug there in a moment.

As far as how to handle node failure in your application, as you mention that would depend on the context of the application. In some cases, you can probably queue operations for retry at the application level or go to a different datasource. In other cases, you may just have to propagate the error up to the user. As a system, Membase trades off partial availability in the case of node failures until you hit the failover button. Have a search for the "CAP theorem" (if you're not familiar with it already) and you'll see the different tradeoffs with distributed systems.

Generally speaking, we're moving Membase toward automatic failover, based on rules. We believe consistency is important from a programmer's perspective, but always work to make sure the system is fast to respond, even in the case of failures. You may want to read http://www.couchbase.org/wiki/display/membase/Failover+with+Membase if you've not already.

I hope that helps. I may be missing some aspect of your questions. If so, please try to rephrase and I'll try to answer.

Thanks,

Matt

Ah!

Now I see :-) Really great guys do many great things! ;-)

I absolutely agree that speed is a key. For me it is ok to handle failures on the client, I just need clean and proper way to do that.

So, I think sinse most of the time membase runs like a charm it is really makes no sense to spend server resources on handling failures in some very complicated way. But in my case it is something strange, spymemcache library just failing down... And doesn't recover... This is why I started this thread.

Thank you again guys for great software!
Look forward for robust solution!

Max

Extra text:

As far as how to handle node failure in your application, as you mention that would depend on the context of the application. In some cases, you can probably queue operations for retry at the application level or go to a different datasource. In other cases, you may just have to propagate the error up to the user. As a system, Membase trades off partial availability in the case of node failures until you hit the failover button. Have a search for the "CAP theorem" (if you're not familiar with it already) and you'll see the different tradeoffs with distributed systems.

This is what I do already in my system. Everything work fine when I click "failover" button in Membase UI. But when I go and pull power switch or kill membase process on one of the nodes via terminal something goes wrong and application fails...

So, at this point I don't know where exactly the issue. It could be that membase itself cannot identify failed node, so it doesn't tell to spymemcache driver to switch to another node or it could be something wrong with spymemcache driver...

But here the question arise: If membase server works fine, why "failover" from UI works while from terminal doesn't?

When a node goes down initially, nothing actually happens. Spymemcached will start timing out requests to that node, this is something that your app should handle gracefully. Requests to other nodes will continue to work.

Once a failover is initiated (and it is the same thing from UI or terminal), the replica copies will be activated and the client should begin receiving responses for those requests again.

We're working on the first release of our auto-failover functionality, look for it in 1.7.1. In reality, all this will be doing is performing the same failover action that you can do manually today.

Hope that answers your questions, I'd need more data on what difference you're seeing when doing a failover via the UI or terminal, it's really going through the same code path (via the REST API)

Perry