[#CCBC-64] PHP Client reports Time Out errors on simple SET and GET operations when Rebalance is in Progress - Couchbase

Hari Subramaniam made changes - 09/May/12 9:26 AM

Field	Original Value	New Value
Description	This bug is reported on behalf of a brand new paid-customer (SocialPoint - Spain). Whenever they issue a rebalance(after removing a node from the cluster or add a new node), client operations(PHP - sample code attached) time out with the error below PHP Warning: Couchbase::set(): Failed to store a value to server: Operation timed out in /root/test_worker.php on line <n> This issue is consistent for both the customer as well as our in-house reproduction in Support. There are no errors on the server side. Rebalance does complete. And once rebalance is complete, the same script runs fine without any errors. IMPACT: Customer considers this a BLOCKER for rolling out their new game. About 28k revenue is in jeopardy pending a resolution to this issue. Support Case# 1156 (http://support.couchbase.com/tickets/1156) EXPECTATION: A workaround or configuration change to alleviate the cluster un-responsiveness OR FIX to the PHP code.	This bug is reported by a customer. Whenever they issue a rebalance(after removing a node from the cluster or add a new node), client operations(PHP - sample code attached) time out with the error below PHP Warning: Couchbase::set(): Failed to store a value to server: Operation timed out in /root/test_worker.php on line <n> This issue is consistent for both the customer as well as our in-house reproduction in Support. There are no errors on the server side. Rebalance does complete. And once rebalance is complete, the same script runs fine without any errors. IMPACT: Customer considers this a BLOCKER for rolling out their new game. About 28k revenue is in jeopardy pending a resolution to this issue. Support Case# 1156 (http://support.couchbase.com/tickets/1156) EXPECTATION: A workaround or configuration change to alleviate the cluster un-responsiveness OR FIX to the PHP code.

Hide

Permalink

Matt Ingenthron added a comment - 09/May/12 11:54 AM - edited

PHP Client Library Version installed?
libcouchbase version installed?

CentOS Version?

Also needed is a description of what kind of rebalance is happening here. A "failover" click followed by a "rebalance" is different than a "remove node" followed by a "rebalance". Please outline the specific steps you'd used to reproduce.

Show

Matt Ingenthron added a comment - 09/May/12 11:54 AM - edited PHP Client Library Version installed? libcouchbase version installed? CentOS Version? Also needed is a description of what kind of rebalance is happening here. A "failover" click followed by a "rebalance" is different than a "remove node" followed by a "rebalance". Please outline the specific steps you'd used to reproduce.

Matt Ingenthron made changes - 09/May/12 11:54 AM

Assignee

Jan Lehnardt [ jan ]

Hari Subramaniam [ hari ]

Matt Ingenthron made changes - 09/May/12 11:56 AM

Description

This bug is reported by a customer. Whenever they issue a rebalance(after removing a node from the cluster or add a new node), client operations(PHP - sample code attached) time out with the error below

PHP Warning: Couchbase::set(): Failed to store a value to server: Operation timed out in /root/test_worker.php on line <n>

This issue is consistent for both the customer as well as our in-house reproduction in Support. There are no errors on the server side. Rebalance does complete. And once rebalance is complete, the same script runs fine without any errors.

IMPACT:
Customer considers this a BLOCKER for rolling out their new game. About 28k revenue is in jeopardy pending a resolution to this issue.
Support Case# 1156 (http://support.couchbase.com/tickets/1156)

EXPECTATION:
A workaround or configuration change to alleviate the cluster un-responsiveness OR FIX to the PHP code.

This bug is reported by a customer. Whenever they issue a rebalance(after removing a node from the cluster or add a new node), client operations(PHP - sample code attached) time out with the error below

PHP Warning: Couchbase::set(): Failed to store a value to server: Operation timed out in /root/test_worker.php on line <n>

This issue is consistent for both the customer as well as our in-house reproduction in Support. There are no errors on the server side. Rebalance does complete. And once rebalance is complete, the same script runs fine without any errors.

Hari Subramaniam made changes - 09/May/12 12:11 PM

Assignee

Hari Subramaniam [ hari ]

Matt Ingenthron [ ingenthr ]

Hide

Permalink

Matt Ingenthron added a comment - 09/May/12 12:15 PM

I do see the PHP client version info, sorry I'd missed that. I do still need the libcouchbase version info though.

Initial investigation shows this is likely an issue in libcouchbase, but we need more about the type of rebalance being done. Please comment.

Show

Matt Ingenthron added a comment - 09/May/12 12:15 PM I do see the PHP client version info, sorry I'd missed that. I do still need the libcouchbase version info though. Initial investigation shows this is likely an issue in libcouchbase, but we need more about the type of rebalance being done. Please comment.

Hide

Permalink

Hari Subramaniam added a comment - 09/May/12 12:32 PM

updated Environment w/OS versions + libcouchbase versions.

Show

Hari Subramaniam added a comment - 09/May/12 12:32 PM updated Environment w/OS versions + libcouchbase versions.

Hari Subramaniam made changes - 09/May/12 12:32 PM

Environment

Centos running Couchbase 1.8.0. 3 Node Cluster, plenty of available RAM and DISK. PHP 5.3

Centos5x OR Ubuntu 11x running Couchbase 1.8.0. 3 Node Cluster, plenty of available RAM and DISK. PHP 5.3. libcouchbase version 1.0.3-1

Matt Ingenthron made changes - 09/May/12 12:52 PM

Assignee

Matt Ingenthron [ ingenthr ]

Sergey Avseyev [ avsej ]

Hide

Permalink

Matt Ingenthron added a comment - 09/May/12 2:32 PM

After some investigation with Sergey, it seems that during the 2.5 second period we give to try and re-try an operation if we receive a not-my-vbucket response, we're retrying constantly but the operation times out and thus we send this response up to the PHP client.

To verify this, Sergey set the timeout to 10 seconds, and verified that the issue could not be seen. This is abnormally high though and would indicate that there is something wrong at the server side. There is no reason we should see more than 2.5 seconds for a vbucket to transfer from one node to another during rebalance.

For another level of verification, we'll check to be sure the client is trying both the place the config states the vbucket is active and the place that the ffwd map states the vbucket is going. If this shows libcouchbase is behaving correctly, then the problem is at the server side, not the client side. It's possible there's a bug in the server here that moxi would mask with it's less sophisticated retry algorithm.

It's equally possible that there is a libcouchbase configuration update problem. This next level of verification should tell us for sure.

Note, PHP does not currently give the user the ability to raise the timeout to 10 seconds, but I don't think that's the right solution here. If a single vbucket transfer is taking longer than that, it needs to be addressed at the server side.

Show

Matt Ingenthron added a comment - 09/May/12 2:32 PM After some investigation with Sergey, it seems that during the 2.5 second period we give to try and re-try an operation if we receive a not-my-vbucket response, we're retrying constantly but the operation times out and thus we send this response up to the PHP client. To verify this, Sergey set the timeout to 10 seconds, and verified that the issue could not be seen. This is abnormally high though and would indicate that there is something wrong at the server side. There is no reason we should see more than 2.5 seconds for a vbucket to transfer from one node to another during rebalance. For another level of verification, we'll check to be sure the client is trying both the place the config states the vbucket is active and the place that the ffwd map states the vbucket is going. If this shows libcouchbase is behaving correctly, then the problem is at the server side, not the client side. It's possible there's a bug in the server here that moxi would mask with it's less sophisticated retry algorithm. It's equally possible that there is a libcouchbase configuration update problem. This next level of verification should tell us for sure. Note, PHP does not currently give the user the ability to raise the timeout to 10 seconds, but I don't think that's the right solution here. If a single vbucket transfer is taking longer than that, it needs to be addressed at the server side.

Hide

Permalink

Sergey Avseyev added a comment - 10/May/12 10:43 AM

I've analyzed the packets sent to the server and found that libcouchbase doesn't send out in time to network the corrected packet after NOT_MY_VBUCKET error. It is copying it into the internal ouput buffer, but it doesn't hit network. Although if increase timeout it working. So the issue in libcouchbase layer and it isn't php specific. Complete rebalance log could be found here http://files.avsej.net/add-node-rebalance.dump (~200M)

Show

Sergey Avseyev added a comment - 10/May/12 10:43 AM I've analyzed the packets sent to the server and found that libcouchbase doesn't send out in time to network the corrected packet after NOT_MY_VBUCKET error. It is copying it into the internal ouput buffer, but it doesn't hit network. Although if increase timeout it working. So the issue in libcouchbase layer and it isn't php specific. Complete rebalance log could be found here http://files.avsej.net/add-node-rebalance.dump (~200M)

Sergey Avseyev made changes - 10/May/12 10:43 AM

Attachment

report.pdf [ 13105 ]

Sergey Avseyev made changes - 10/May/12 10:44 AM

Project	Couchbase PHP client library [ 10049 ]	Couchbase C client library libcouchbase [ 10070 ]
Key	PCBC-64	~~CCBC-64~~
Affects Version/s		1.0.2 [ 10284 ]
Affects Version/s	1.0.2 [ 10310 ]
Component/s		library [ 10110 ]
Component/s	library [ 10134 ]

Hide

Permalink

Matt Ingenthron added a comment - 10/May/12 8:22 PM

A fix has been produced, reviewed. http://review.couchbase.org/#change,15882

Will send a verification of the fix to support for either support or the customer to verify the fix, then will include it in the next patch update.

Show

Matt Ingenthron added a comment - 10/May/12 8:22 PM A fix has been produced, reviewed. http://review.couchbase.org/#change,15882 Will send a verification of the fix to support for either support or the customer to verify the fix, then will include it in the next patch update.

Hide

Permalink

Sergey Avseyev added a comment - 15/May/12 4:23 PM

The patch was merged

Show

Sergey Avseyev added a comment - 15/May/12 4:23 PM The patch was merged

Sergey Avseyev made changes - 15/May/12 4:23 PM

Assignee

Sergey Avseyev [ avsej ]

Jan Lehnardt [ jan ]

Hide

Permalink

Hari Subramaniam added a comment - 21/May/12 10:34 AM

Fix has been verified by the customer. Considering this bug as addressed.

Show

Hari Subramaniam added a comment - 21/May/12 10:34 AM Fix has been verified by the customer. Considering this bug as addressed.

Sergey Avseyev made changes - 07/Jun/12 6:10 AM

Status	Open [ 1 ]	Resolved [ 5 ]
Fix Version/s		1.0.4 [ 10327 ]
Resolution		Fixed [ 1 ]

Sergey Avseyev made changes - 13/Nov/12 4:21 AM

Status

Resolved [ 5 ]

Closed [ 6 ]

PHP Client reports Time Out errors on simple SET and GET operations when Rebalance is in Progress

Details

Description

Attachments

Activity

People

Dates

Agile