Membase is unresponsive due to a possible file descriptor leak.

Membase is unresponsive due to a possible file descriptor leak.

The membase logs show the following error:
Feb 6 15:45:23 fv-mb-snapi-091 [membase]: Failed to notify thread: Resource temporarily unavailable

And during this time the membase server is unresponsive.

In between the appearance of the above error the following error is also thrown:
Feb 7 17:26:26 fv-mb-snapi-091 [membase]: Too many open connections


The max connection limit has been set to 64k for the max_conn parameter in membase and as inferred from the graph attached only 8k connections are in use hence we believe that the remaining connections/file descriptors are getting leaked.

We suspect that for each appearance of the "Failed to notify thread: Resource temporarily unavailable" a file descriptor is being leaked.