Originally, a rule was to keep changes from any "memcached heritage" files to a minimum.
The motivation was so that moxi could keep up with memcached, allowing us to easily merge in any improvements from memcached into moxi.
This is why much of moxi lives in files separate from memcached's files (cproxy_*.c/.h)
Nowadays, that rule is very much relaxed. We'll probably never get easy merges from memcached into moxi (or vice versa).
As much as possible, moxi tries to avoid locks, even though it is multi-threaded, especially in the key/item request proxying codepaths.
Moxi can be considered a store and forward proxy. So, the high level "main loop" in moxi worker threads in moxi wait for i/o activity, via libevent (a higher-level library above epoll/poll/kpoll/select). Some of the i/o events include:
If a client has sent a request, a worker thread wakes up, reads in the client's request fully, and queues each request in memory.
If a downstream cluster resource becomes available, moxi assigns a waiting request off the queue, makes routing decisions (ketama or libvbucket), and writes the request to the appropriate downstream server(s).
If a downstream server has sent a response, a worker thread wakes up, reads in the server's response fully, and asynchronously writes the response back to the waiting client. After all responses are fully written, the client connection is then available for more input request reading.