Couchbase Server

SYNC command needs coarse grain server side timeout

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Major Major
  • Resolution: Fixed
  • Affects Version/s: 1.7 alpha 2
  • Fix Version/s: 2.0-beta
  • Component/s: couchbase-bucket
  • Security Level: Public
  • Labels:
    None
  • Flagged:
    Impediment

Description

We need some fixed coarse grain server side timeout for the SYNC command to prevent SYNC requests from using a connection forever.

We need to balance the amount with the fact that SYNC ops could take some while to happen for long lists of keys.

Proposed value 30 sec [Matt, Dustin, Jan, does that make sense?]

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Matt Ingenthron added a comment -
30 sec seems awfully long.

I'll recommend 2.5s. That's enough time for a TCP retransmit, if needed, plus a little bit of slop. Sadly, it's also enough time to easily overwhelm the downstreams, so may need a separate pool for things that can block in most clients.

It should be noted that this can still mess up applications, since their logic may rely on sync and can quickly block threads at that level. This should have flags all over it in the SDK.
Show
Matt Ingenthron added a comment - 30 sec seems awfully long. I'll recommend 2.5s. That's enough time for a TCP retransmit, if needed, plus a little bit of slop. Sadly, it's also enough time to easily overwhelm the downstreams, so may need a separate pool for things that can block in most clients. It should be noted that this can still mess up applications, since their logic may rely on sync and can quickly block threads at that level. This should have flags all over it in the SDK.
Hide
Permalink
Steve Yen added a comment -
Possibly a separate bug: we also need more error response codes. Why wait for a timeout if we know SYNC just won't work -- for example, because too many TAP clients are down.

Related, there's a hole where "third party" TAP clients (such as the recently demo'ed sqoop TAP integration) can mess up the 'successful replica' counting.

Show
Steve Yen added a comment - Possibly a separate bug: we also need more error response codes. Why wait for a timeout if we know SYNC just won't work -- for example, because too many TAP clients are down. Related, there's a hole where "third party" TAP clients (such as the recently demo'ed sqoop TAP integration) can mess up the 'successful replica' counting.
Hide
Permalink
Matt Ingenthron added a comment -
Is that separate hole worthy of a bug?
Show
Matt Ingenthron added a comment - Is that separate hole worthy of a bug?
Hide
Permalink
Filipe Manana added a comment -
Pushed a change to gerrit for review, http://review.membase.org/#change,6557
Show
Filipe Manana added a comment - Pushed a change to gerrit for review, http://review.membase.org/#change,6557
Hide
Permalink
Matt Ingenthron added a comment -
Via email, you'd asked about how to make this configurable. I would recommend you look at flushctl for an example, but the basic idea is that there is a configuration command, for which the CLI tool is currently the poorly-named flushctl. The parameter is validated and applied to the running engine here:
http://src.norbye.org/source/xref/membase-1.7/ep-engine/ep_engine.cc#222

Show
Matt Ingenthron added a comment - Via email, you'd asked about how to make this configurable. I would recommend you look at flushctl for an example, but the basic idea is that there is a configuration command, for which the CLI tool is currently the poorly-named flushctl. The parameter is validated and applied to the running engine here: http://src.norbye.org/source/xref/membase-1.7/ep-engine/ep_engine.cc#222
Hide
Permalink
Matt Ingenthron added a comment -
This was apparently merged some time back.
Show
Matt Ingenthron added a comment - This was apparently merged some time back.

People

  • Assignee:
    Filipe Manana
    Reporter:
    Frank Weigel
Vote (0)
Watch (4)

Dates

  • Created:
    Updated:
    Resolved: