Couchbase C client library libcouchbase

timeouts seen after failover, rebalance and add back

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Blocker Blocker
  • Resolution: Fixed
  • Affects Version/s: 1.0.4
  • Fix Version/s: 1.0.5
  • Component/s: None
  • Security Level: Public
  • Labels:
    None
  • Environment:
    PHP 5.3.3 (cli) (built: Jun 27 2012 12:25:48)
    libcouchbase1-1.0.4-1
    CentOS release 5.8 (Final), x86_64
    Couchbase Server 1.8.1 Enterprise

Description

scenario:
1. Start PHP client in a loop setting and getting against a 2 node cluster
2. Click failover to kick a node out, click rebalance to make it unassociated
3. Walk through the setup wizard on that node, re-add it to the cluster
4. After adding, click rebalance

Expected behavior:
During rebalance in step 4, which is an add node scenario, no timeouts are expected.

Observed behavior:
During rebalance in step 4, I see timeouts from PHP, and they continue even after the rebalance has completed.

Activity

Ascending order - Click to sort in descending order
Matt Ingenthron made changes -
Field Original Value New Value
Assignee Matt Ingenthron [ ingenthr ] Sergey Avseyev [ avsej ]
Matt Ingenthron made changes -
Priority Major [ 3 ] Blocker [ 1 ]
Hide
Permalink
Matt Ingenthron added a comment -
A packet capture of this same issue, with the client on MacOS X and CentOS 5.8 servers with Couchbase Server 1.8.1 enterprise edition may be found at http://dl.dropbox.com/u/1537838/failover-maybe-issue
Show
Matt Ingenthron added a comment - A packet capture of this same issue, with the client on MacOS X and CentOS 5.8 servers with Couchbase Server 1.8.1 enterprise edition may be found at http://dl.dropbox.com/u/1537838/failover-maybe-issue
Matt Ingenthron made changes -
Environment PHP 5.3.3 (cli) (built: Jun 27 2012 12:25:48)
libcouchbase1-1.0.4-1
CentOS release 5.8 (Final), x86_64 		PHP 5.3.3 (cli) (built: Jun 27 2012 12:25:48)
libcouchbase1-1.0.4-1
CentOS release 5.8 (Final), x86_64
Couchbase Server 1.8.1 Enterprise
Hide
Permalink
Sergey Avseyev added a comment -
http://review.couchbase.org/19563
Show
Sergey Avseyev added a comment - http://review.couchbase.org/19563
Hide
Permalink
Matt Ingenthron added a comment -
Note from discussion, this is a possible fix, not sure.
Show
Matt Ingenthron added a comment - Note from discussion, this is a possible fix, not sure.
Matt Ingenthron made changes -
Project Couchbase PHP client library [ 10049 ] Couchbase C client library libcouchbase [ 10070 ]
Key PCBC-98 CCBC-91
Affects Version/s 1.0.4 [ 10327 ]
Affects Version/s 1.0.5 [ 10339 ]
Fix Version/s 1.0.5 [ 10331 ]
Fix Version/s 1.0.5 [ 10339 ]
Hide
Permalink
Matt Ingenthron added a comment -
Sergey and I reproduced the issue, and it's related to the series of steps outlined above. The underlying libcouchbase is not receiving the updated configuration for some reason, and thus is sending items to the wrong node, and then they're timing out.

Sergey will do more work on finding the specific cause.
Show
Matt Ingenthron added a comment - Sergey and I reproduced the issue, and it's related to the series of steps outlined above. The underlying libcouchbase is not receiving the updated configuration for some reason, and thus is sending items to the wrong node, and then they're timing out. Sergey will do more work on finding the specific cause.
Hide
Permalink
Sergey Avseyev added a comment -
The patch http://review.couchbase.org/19599 and aforementioned http://review.couchbase.org/19563 solves the issue.

To reproduce it for sure you should failover the node is currently used by client to listen config changes. (Usually it is the first successfull node from initial node list)
Show
Sergey Avseyev added a comment - The patch http://review.couchbase.org/19599 and aforementioned http://review.couchbase.org/19563 solves the issue. To reproduce it for sure you should failover the node is currently used by client to listen config changes. (Usually it is the first successfull node from initial node list)
Sergey Avseyev made changes -
Status Open [ 1 ] Resolved [ 5 ]
Resolution Fixed [ 1 ]
Sergey Avseyev made changes -
Status Resolved [ 5 ] Closed [ 6 ]

People

  • Assignee:
    Sergey Avseyev
    Reporter:
    Matt Ingenthron
Vote (0)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved: