Home | Forums | Membase | Membase Server 1.7.x

Socket timeouts: 1.7.1

3 replies [Last post]
Tue, 10/04/2011 - 07:06
bradrover
Offline
Joined: 10/04/2011
Groups: None

We're running 16 nodes on CentOS, with Windows clients using Enyim client 2.10/Membase 2.13. We're getting a lot of socket timeouts client side with the Enyim stuff on the default bucket, and a lot of unexplained Set command failures with no logs. On the server side we see mnesia overload warnings. Just trying to figure out where we should start troubleshooting this. We do about 20K ops per second on the cluster.

Top
Wed, 10/12/2011 - 03:14
alex
Offline
Joined: 08/29/2011
Groups: None

Hi There,

A few questions for you:
When are you creating new *Clients?
Are they singletons?
Do you have a factory?
Do you create them by request?
Is the client being created multiple times?
How are your clients disposed of?

If clients are not disposed they will consume free connections on the server. These will eventually be cleaned up when garbage collection kicks in but if they are created faster than cleanup the server will run out of connections.

This link outlines how you should and should not connect with the Enyim client.
https://github.com/enyim/EnyimMemcached/wiki/MembaseClient-Usage

Also, make sure you are on the latest Enyim client:
http://memcached.enyim.com/post/6361330653/membase-2-12

-Alex

Top
Wed, 10/12/2011 - 08:32
bradrover
Offline
Joined: 10/04/2011
Groups: None

Hello Alex,

New clients are created once per process, per bucket. We do this in a class library we wrap the Enyim client code in. The instances of Membase client are stored in a static dictionary by bucket name. Generally we are only using the default bucket right now.

They are singletons. They are create with a factory.

We do not dispose of clients. Since they are static per process, They should live for the duration of the process.

Top
Sat, 10/15/2011 - 15:39
ingenthr
Offline
Joined: 03/16/2010
Groups:

I'm not certain this is the cause, but there had been a recent issue with Enyim where a momentary connection problem would accidentally mark a server as dead until application restart. It's fixed now, with better fixes coming. The latest is 2.14 and is available from: http://www.couchbase.org/code/couchbase/net

To see if that's what the problem is, you can probably check at the OS level if you have the number of connections you expect. The other thought would be to recycle the client after a certain number of operations. Both of these would be good debugging steps. Also, turning up the logging.

Then again, just upgrading may take care of it.

Top