Couchbase Server

Membase is unresponsive due to a possible file descriptor leak.

Details

  • Type: Bug Bug
  • Status: Resolved Resolved
  • Priority: Major Major
  • Resolution: Incomplete
  • Affects Version/s: 1.6.5
  • Fix Version/s: None
  • Component/s: bucket-engine
  • Security Level: Public
  • Labels:
    None

Description

Membase is unresponsive due to a possible file descriptor leak.

The membase logs show the following error:
Feb 6 15:45:23 fv-mb-snapi-091 [membase]: Failed to notify thread: Resource temporarily unavailable

And during this time the membase server is unresponsive.

In between the appearance of the above error the following error is also thrown:
Feb 7 17:26:26 fv-mb-snapi-091 [membase]: Too many open connections


The max connection limit has been set to 64k for the max_conn parameter in membase and as inferred from the graph attached only 8k connections are in use hence we believe that the remaining connections/file descriptors are getting leaked.

We suspect that for each appearance of the "Failed to notify thread: Resource temporarily unavailable" a file descriptor is being leaked.

Activity

Ascending order - Click to sort in descending order
Hide
Permalink
Alex Ma added a comment -
Pritish,

Can you add detailed steps to reproduce as well as the output of lsof and netstat?

thanks

-Alex.
Show
Alex Ma added a comment - Pritish, Can you add detailed steps to reproduce as well as the output of lsof and netstat? thanks -Alex.
Hide
Permalink
Alex Ma added a comment -
Pritish,

can you also provide a stats snapshot as well?

thanks

-Alex.
Show
Alex Ma added a comment - Pritish, can you also provide a stats snapshot as well? thanks -Alex.
Hide
Permalink
Trond Norbye added a comment -
missing information from lsof and netstat, and I suspect we won't get that (given it was reported in february). There has been a lot of rewrites in the areas affected here (like we're no longer using vbucketmigrator, tap has been rewritten, memcached connection handling changed etc). we would need more information to nail down where the problems comes from..
Show
Trond Norbye added a comment - missing information from lsof and netstat, and I suspect we won't get that (given it was reported in february). There has been a lot of rewrites in the areas affected here (like we're no longer using vbucketmigrator, tap has been rewritten, memcached connection handling changed etc). we would need more information to nail down where the problems comes from..
Trond Norbye made changes -
Field Original Value New Value
Status Open [ 1 ] Resolved [ 5 ]
Resolution Incomplete [ 4 ]
Peter Wansch made changes -
Component/s bucket-engine [ 10010 ]
Component/s memcached-bucket(default-engine) [ 10015 ]

People

  • Assignee:
    Alex Ma
    Reporter:
    Pritish Pratap
Vote (0)
Watch (0)

Dates

  • Created:
    Updated:
    Resolved: