Client-side moxi falls over (hung connections)

i am having trouble with client-side moxi.

it chugs along for a while with no problems, then all of a sudden i get a few of my client-to-moxi connections that hang.

the percentage of connections that hang is fairly stable (tho after a reboot that number may be slightly different), and results in an error/timeout rate of about the same percentage.

this is using the default config values, with a transaction rate of 2.5k to 4k requests per second.

there are three downstream couchabase nodes moxi is connecting to.

any idea?


Some thoughs…

Perhaps try cranking up the logging to look for suspicious log messages.

And/or, perhaps telnet into moxi (like port 1121) and do a “stats proxy” command. I usually then I crawl through the output looking error counts for any root cause smoking guns.

thanks for the suggestions.

tried those, no luck.

btw, the moxi logging levels can be describe as such:

v == very concise amount of not-helpful
vv == medium amount of still-not-helpful
vvv == gigabytes per minute of can-you-believe-its-still-not-helpful


Changing the memcached client’s failure mode to Cancel (from Redistribute) seems to have fixed the problem.