overstressed memcached keeps conns in close_wait even if the client has already closed its conns

Farshid Ghods added a comment - 20/Mar/12 6:48 PM moxi closes the conneciton but memcached keeps it open as close_wait so next connection will be rejected this causes further issues during the rebalancing

Steve Yen added a comment - 26/Mar/12 1:14 PM One workaround, which we used for a customer with a large cluster, was to increase their client-side moxi timeouts to larger values (larger than the longest backfill request) Two longer term solution ideas include... 1) Unified dispatcher in memcached should help fix this (allows more than one R/O dispatcher for a single bucket). 2) Allow adaptive/dynamic timeout backoffs in moxi.

Steve Yen added a comment - 26/Mar/12 7:33 PM 3) Change moxi default timeout from 5 secs to ?

Steve Yen added a comment - 26/Mar/12 7:38 PM 4) When moxi has a timeout, moxi shouldn't close a downstream connection?

Trond Norbye added a comment - 20/Jun/12 1:48 AM Do we have any "tap stats" from this node? in our tap implementation we're "reserving" the connection until we believe it's safe to close it from the ep-engine side (and keeping the connections in a "pending close" state. (a reconnect to the same tap struct won't immediately release the connection, because we're "cloning" the metadata in ep-engine to let it close itself). This logic has been revised on 1.8.1.

Trond Norbye added a comment - 03/Jul/12 2:17 AM We've changed the shutdown logic for connections in 1.8.2 and 2.0.