Couchbase C client library libcouchbase

timeouts seen after failover, rebalance and add back

Details

  • Type: Bug Bug
  • Status: Closed Closed
  • Priority: Blocker Blocker
  • Resolution: Fixed
  • Affects Version/s: 1.0.4
  • Fix Version/s: 1.0.5
  • Component/s: None
  • Security Level: Public
  • Labels:
    None
  • Environment:
    PHP 5.3.3 (cli) (built: Jun 27 2012 12:25:48)
    libcouchbase1-1.0.4-1
    CentOS release 5.8 (Final), x86_64
    Couchbase Server 1.8.1 Enterprise

Description

scenario:
1. Start PHP client in a loop setting and getting against a 2 node cluster
2. Click failover to kick a node out, click rebalance to make it unassociated
3. Walk through the setup wizard on that node, re-add it to the cluster
4. After adding, click rebalance

Expected behavior:
During rebalance in step 4, which is an add node scenario, no timeouts are expected.

Observed behavior:
During rebalance in step 4, I see timeouts from PHP, and they continue even after the rebalance has completed.

Activity

Descending order - Click to sort in ascending order
Hide
Permalink
Sergey Avseyev added a comment -
The patch http://review.couchbase.org/19599 and aforementioned http://review.couchbase.org/19563 solves the issue.

To reproduce it for sure you should failover the node is currently used by client to listen config changes. (Usually it is the first successfull node from initial node list)
Show
Sergey Avseyev added a comment - The patch http://review.couchbase.org/19599 and aforementioned http://review.couchbase.org/19563 solves the issue. To reproduce it for sure you should failover the node is currently used by client to listen config changes. (Usually it is the first successfull node from initial node list)
Hide
Permalink
Matt Ingenthron added a comment -
Sergey and I reproduced the issue, and it's related to the series of steps outlined above. The underlying libcouchbase is not receiving the updated configuration for some reason, and thus is sending items to the wrong node, and then they're timing out.

Sergey will do more work on finding the specific cause.
Show
Matt Ingenthron added a comment - Sergey and I reproduced the issue, and it's related to the series of steps outlined above. The underlying libcouchbase is not receiving the updated configuration for some reason, and thus is sending items to the wrong node, and then they're timing out. Sergey will do more work on finding the specific cause.
Hide
Permalink
Matt Ingenthron added a comment -
Note from discussion, this is a possible fix, not sure.
Show
Matt Ingenthron added a comment - Note from discussion, this is a possible fix, not sure.
Hide
Permalink
Sergey Avseyev added a comment -
http://review.couchbase.org/19563
Show
Sergey Avseyev added a comment - http://review.couchbase.org/19563
Hide
Permalink
Matt Ingenthron added a comment -
A packet capture of this same issue, with the client on MacOS X and CentOS 5.8 servers with Couchbase Server 1.8.1 enterprise edition may be found at http://dl.dropbox.com/u/1537838/failover-maybe-issue
Show
Matt Ingenthron added a comment - A packet capture of this same issue, with the client on MacOS X and CentOS 5.8 servers with Couchbase Server 1.8.1 enterprise edition may be found at http://dl.dropbox.com/u/1537838/failover-maybe-issue

People

  • Assignee:
    Sergey Avseyev
    Reporter:
    Matt Ingenthron
Vote (0)
Watch (1)

Dates

  • Created:
    Updated:
    Resolved: