Moxi with monit - anyone done this successfully?
I'm having a heck of a time getting monit to play nicely with moxi. At this point it always reports that "moxi failed to start", though I see the process running. This is a monit issue I'm sure, but I imagine most people who deal with moxi want to have it monited somehow. I'm posting my monit snippet below. I can confirm that the pidfile is being created and the server does, in fact, start up. Monit reports the process is not monitor and /var/log/syslog shows "moxi failed to start" reported by monit.
Also, I don't use "/bin/kill -9 `cat /var/run/moxi/moxi.pid`" because, surprise surprise, it never works in monit.
Any ideas from people who know monit better than I?
* * *
check process moxi
with pidfile /var/run/moxi/moxi.pid
start program = "/opt/moxi/bin/moxi -P /var/run/moxi/moxi.pid -d -Z usr=Administrator,pwd=XXXX -z /etc/moxi/hosts.cfg -u ubuntu"
stop program = "/usr/bin/killall -9 moxi"
if 5 restarts with 5 cycles then timeout
if cpu is greater than 80% for 5 cycles then alert
Hi, appreciate you posting this -- I'm not a monit user myself, so thanks for making the web more intelligent :-)
Steve
check process moxi_fragments with pidfile /opt/moxi/fragments.pid start program = "/etc/init.d/moxi start fragments" stop program = "/etc/init.d/moxi stop fragments" if failed host 127.0.0.1 port 15000 then restart
i have monit monitor the port that moxi is listening on. i also wrote a wrapper for moxi because i usually like adding a bit of dynamic logic (like where to point moxi, and settings to use, etc) that puppet can place. with this being said, we still had a ton of connect errors to moxi and monit never showed that it was unavailable. so your milage may vary.
For your connect errors, can you describe the behavior a bit better? I'm assuming that Moxi was still running and still listening on the desired port but that it's connection to the Membase servers was somehow broken or malfunctioning and so when your memcache clients tried to connect they had issues.
Is that accurate? Does monit have the capability to do higher level monitoring for this situation?
Perry
the errors where from the memcached client connecting to moxi, moxi appeared to have no problems connecting to any of the membase nodes at any time. our monitoring systems were never unable to connect to moxi (although they sample every 60 seconds or so, which is far less frequent then the app would connect and make requests). monit was just monitoring that the moxi processes were still listening on the correct ports, however nagios would make stats requests every 5 min to ensure the connection is live.
And what were the specific errors from the memcached client? Were they timeouts, connection refused or something else?
connection refused.
N/m, it had to do with the fact that I needed a different place to put my PID files. So, I made a new directory and use /etc/monit/pids/ instead of /var/run/moxi/, which is deleted on reboot. Anyhow, now the web has an example of moxi in a monit file.