Home | Forums | Couchbase | Couchbase Server 1.8.x

Auto fail over seems to be node dependant?

6 replies [Last post]
Tue, 10/30/2012 - 04:28
geosword
Offline
Joined: 10/09/2012
Groups: None

Hi All,
Im seeing some inconsistent behavior with auto failover, and wondered if anyone can explain whats going on. Here's my setup:

Ive got (in a testing environment) 3 nodes running ubuntu precise & couchbase 1.8.1 configured 1 bucket and 1 replica. I Also have a webserver running moxi, which is running a couple of small test php scripts (using the memcache library). One script sets 2000 keys to known values. Key1 =1, Key2 = 2 and so on, the other script gets those values, and checks that the result it gets back is correct. Im testing autofail over, which is enabled and set at 30 seconds.

Moxi has the address of all three nodes in its moxi-cluster.cfg.

The method im using for "downing" any node is to stop the couchbase service using the init.d script, or service couchbase-server stop if you're on redhat.

So here's what I'm seeing:
-------------------------
If I stop the service on node1, then no data is returned by the get script, until after (around) 30 seconds, when the auto fail over is performed. After this point, all data is accessible. Which is what I would expect.

If I stop the service on node3, then no data is return by the get script, EVEN AFTER the auto fail over has occurred.

Anyone any ideas as to why this might be happening?

Thanks in advance!

Geosword

Top
Tue, 10/30/2012 - 05:21
fercho
Offline
Joined: 07/27/2012
Groups: None

Hi Geosword.
I get the same results of you when stop node1 on my tests but using no moxy on webservers and using pecl/memcache extension instead couchbase extension (php).
I did another test replacing pecl/memcache with couchbase and no moxy on webserver (couchbase php extension is smart client so I don't install moxy) and everything go almost perfect (I say "almost" because I get one segmentation fault on the script at moment I shutdown the server... later executions of the script work fine).
I didn't test failovers on different nodes, I only check with random one.
do you clean autofailver notification after shutdown node1?

Top
Tue, 10/30/2012 - 05:33
geosword
Offline
Joined: 10/09/2012
Groups: None

Hi Fercho,
Thanks for the reply. Yes I reset the auto failover notification. Which version of the couchbase php library are you using? We're still using the memcache libraries as I mentioned, but moving over to couchbase libraries is looking more and more attractive. Especially if it makes auto fail over a bit more reliable. With the couchbase library you are using, was the data available after you shut down the node but before the auto fail over happened?
If so, this is a really strong reason to move over to the couchbase libraries.

Thanks
Geo

Top
Tue, 10/30/2012 - 05:57
fercho
Offline
Joined: 07/27/2012
Groups: None

Geo
Yesterday morningn I put a new thread showing this behavior (http://www.couchbase.com/forums/thread/strange-behavior-when-one-node-down).
During the day I was doing tests, changing memcache calls with couchbase calls.
I work on CentOS 5, so first install C Client Library libcouchbase1-1.0.4-1.i386.rpm from this place (http://www.couchbase.com/develop/c/current). Download and install rpm accord to you OS.
Then, I install php client library (http://www.couchbase.com/develop/php/current). Install rpm, modify php.ini to load couchbase.so and verify with phpinfo():

couchbase
couchbase support	enabled
version	1.0.5
json support	yes
fastlz support	no
zlib support	yes
(...)

Finaly, change references from memcache functions to couchbase functions (warning: there's no close option for couchbase):

Connection:

$hosts = array('HOST_A:8091','HOST_B:8091','HOST_C:8091');
$cb = new Couchbase($hosts,'','','p_bucket');

Set:
for ($i = 1; $i <= 10; $i++) {
    $cb->set('key_'.$i, 'value...', 3600) or die ("Failed to save data at the server");
    echo "Storing key_".$i." in couchbase...<br/>\n";
};

Get:
      $cb_r = new Couchbase($hosts,'','','p_bucket');
      for ($i = 1; $i <= 10; $i++){
 
          $get_result = $cb_r->get('key_'.$i);
          echo "<br/>\nData from the cache:";
          var_dump($get_result);
      }

Top
Thu, 11/22/2012 - 13:27
Neo-matrix
Offline
Joined: 10/15/2012
Groups: None

Hi,
In this case i dont think the issue is with the Client or Moxi. Because the reads and writes are fine when the nodes are up and running, When you trying an auto-failover you are facing this issue.

Say a auto-failover triggered after 30seconds automatically/manually on a node, then the replica copies will be moved to active status from replica status and will start serving the requests, eventhough the node is failed. This is what really happens. So if you have replicas, then it should work fine.

To check this, i would recommend to grep the active items before and after failover.

Thanks
Neo

Top
Fri, 11/23/2012 - 08:12
fercho
Offline
Joined: 07/27/2012
Groups: None

Hi Neo.

Quote:

To check this, i would recommend to grep the active items before and after failover.

Sorry, but what do you mean when you say "grep the active items before and after failover"?
Thanks!
Fercho

Top
Fri, 11/23/2012 - 10:17
Neo-matrix
Offline
Joined: 10/15/2012
Groups: None

Hi,

In other words, could you please check this "vb_active_num - Number of active vBuckets" before and after the fail over of node 3.

Because we know that there should be 1024 vbuckets in a cluster. so if the node 3 fails and the replica data correspoding to that node should be moved to the active status, so the vbucket will be reconstructed in such a way that all the datas are served and it must be 1024.

If need any clarifications, please let me know.

Thanks,
Neo

Top