Memcached dying on Mac OS X due to rlimit - fix
If you build on Mac OS X, you can get to a state where everything seems, from the web interface, to be running OK, but connections fail.
Looking at the log page, however, reveals that things are dying:
Port server memcached on node 'firstname.lastname@example.org' exited with status 71. Restarting. Messages: failed to set rlimit for open files. Try running as root or requesting smaller maxconns value.
Here's how I fixed it:
In both memcached/memcached.c (around line 6900) and moxi/memcached.c (around 4751) there's some code which tries to set the number of open connections using setrlimit(), based on the maxconns setting. The Mac doesn't like rlim_max being changed, and there's a limit of OPEN_MAX mentioned in the setrlimit man page.
#ifndef min #define min(a,b) ((a) < (b) ? (a) : (b)) #endif
at the top of each file and replaced the lines:
if (rlim.rlim_max < rlim.rlim_cur) rlim.rlim_max = rlim.rlim_cur;
rlim.rlim_cur = min(min(OPEN_MAX, rlim.rlim_max), rlim.rlim_cur);
And then did
make -C memcached make -C moxi make
And it all then worked.
This isn't an ideal solution, because it can ignore the maxconns configuration used internally, but it does get everything working.