Memcache timeouts in app logs
Hey guys I just replaced my old memcache servers with membase in hopes that it would fix the timeout issues we've been seeing in our logs, but it hasn't.
What's the recommended client version to connect to Membase?
We are running CentOS (LAMP) with Drupal (Pressflow 6.22) and php-pecl-memcache-3.0.4 client binary.
Should we update this? And if not, what can we do to fix the timeouts?
PHP Warning: Memcache::connect() [memcache.connect]: Can't connect to 10.129.36.57:13111, Connection timed out (110)
Thanks in advance!
can anyone help here? what can I do to fix this?
*Sigh*.....so much for community support!
Adding more details to this -
Servers are HP with dual quad cores and 32GB of ram with dual gigabit links to the same local network. There should be no reason to have timeouts. These membase servers are pushing 450Mbps at peak traffic.
Running CentOS / Varnish / Apache 2.2 / PHP 5.2 / Mysql 5.5 and the latest version of Membase. I was seeing the same timeouts on memcached and I was hoping that membase would help distribute the load across many servers... It didn't seem to help and I'm seeing the "MYSQL server has gone away!" messages and "memcache timeouts" messages in the logs all the time... please help!
Moving this to the SDK forum -
I am using
php-pecl-memcache-3.0.4-1.el5
libmemcached-0.31-1.1.el5
On Drupal 6.15 through 6.22 (On different sites)
I'm thinking I need to update my drupal module but can someone explain the benefits of using pecl-memcacheD instead of pecl-memcache? Do I really need persistence?
Errors in the logs:
[Wed Nov 02 20:38:23 2011] [error] [client 127.0.0.1] PHP Warning: Memcache::connect() [memcache.connect]: Can't connect to 10.129.36.56:11311, Connection timed out (110) in /usr/local/htdocs/sites/all/modules/contrib/memcache/dmemcache.inc on line 326
And every so often I see this (Even though the DB is fine and up and all other sites are working fine)
When this error appeared when I was using memcache (before I setup membase) the entire site would crash until I restarted memcached and the database.
[Tue Nov 01 14:11:59 2011] [error] [client 10.129.36.42] PHP Warning: MySQL server has gone away\nquery: SELECT * FROM system WHERE type = 'theme' in /usr/local/htdocs/includes/database.mysqli.inc on line 148
[Tue Nov 01 14:12:08 2011] [error] [client 10.129.36.41] PHP Warning: MySQL server has gone away\nquery: SELECT * FROM system WHERE type = 'theme' in /usr/local/htdocs/includes/database.mysqli.inc on line 148
Anyone have any input? I'm glad I'm using membase now and that its replicated so these timeouts aren't as critical as before when I was using memcache by itself..
Thanks
Sorry for the delayed reply. We of course want to help wherever possible, but can't always dedicate time to helping on the forum.
I can say from previous experience, sometimes pecl-memcache sees timeouts that aren't often seen with pecl-memcached. Long story short, pecl-memcache has had long periods of time where it went unmaintained, then updates, then unmaintained, etc. Many PHP users have either modified it themselves or moved on to pecl-memcached. libmemcached, which is under pecl-memcached, is widely used and well maintained.
The thing that's confusing is the error seems to be a warning about the MySQL server going away. Are you sure that's related to the memcached client?
Again, sorry for the delayed reply.
I can say from previous experience, sometimes pecl-memcache sees timeouts that aren't often seen with pecl-memcached. Long story short, pecl-memcache has had long periods of time where it went unmaintained, then updates, then unmaintained, etc. Many PHP users have either modified it themselves or moved on to pecl-memcached. libmemcached, which is under pecl-memcached, is widely used and well maintained.
The thing that's confusing is the error seems to be a warning about the MySQL server going away. Are you sure that's related to the memcached client?
Again, sorry for the delayed reply.
Thanks for getting back to me -
The mysql alerts seem to coincide with the memcache timeouts... thankfully now that i have membase, it doesn't take the entire site down now which has saved my rear end!
What kind of considerations we need to keep in mind when moving from pecl-memcache to pecl-memcached? Do I really need persistence?
Thanks again
I don't know how much traffic you are getting on your network or if you are using iptables for your firewall, but if you are you may be hitting a little known ip_conntrack table limit problem.
We get many unique IPs and the ip_conntrack table fills up and the network won't take any new connections until iptables drops some of the items in the table. We fixed it by adding this to the /etc/sysctl.conf file:
net.ipv4.tcp_max_tw_buckets = 1000000
net.ipv4.netfilter.ip_conntrack_max = 4194304
net.ipv4.netfilter.ip_conntrack_tcp_timeout_established = 120
net.ipv4.tcp_window_scaling = 1
You will see errors in /var/log/messages if this is your problem.
It is only a problem if you get hits from many unique IPs.
Can anyone help here? Also seeing this in the logs which is concerning:
[client 10.129.36.41] PHP Warning: MySQL server has gone away\nquery: SELECT * FROM system WHERE type = 'theme' in /usr/local/htdocs/includes/database.mysqli.inc on line 148
Anyone know what to do here? I'm a bit confused as to why both memcached & membase are giving me the same issue. Which leads me to believe its the memcache client that has a bug.... Please help! thanks!