Home | Forums | Membase | Membase Server 1.7.x

Membase server stop not stopping all running processes

4 replies [Last post]
Mon, 10/03/2011 - 08:21
chrisribe
Offline
Joined: 09/26/2011
Groups: None

Hi,

When I run the stop command on my membase servers with the following command:
/etc/init.d/membase-server stop

Processes are still present when checking with:
ps aux | grep membase

This is an issue because I cannot seem to add the server back to the server pool until all processes are killed and the service restarted. The only way to stop the processes it seems is to issue a kill -9.

Any ideas why this is happening?
Thanks
Chris

Top
Mon, 10/03/2011 - 09:58
alex
Offline
Joined: 08/29/2011
Groups: None

You can look at Monitor->Server nodes-> and check the disk write queue stats under summary to see the queue of things waiting to be written to disk.

If your disk write queue is high and you stop membase, the process will confirm the shutdown, membase will immediately stop taking incoming requests and continue writing things off the queue to disk before actually shutting down.

thanks

-Alex.

Top
Thu, 10/13/2011 - 09:00
chrisribe
Offline
Joined: 09/26/2011
Groups: None

Hi Alex,

Ah ok, I did not check the disk queue state when I was having issues.
During normal operations my queue is pretty low, about 40 items with an average age of 0.06 seconds.

Membase should at least tell you something like *writing current queue to disk before shutdown* then say *server stopped*. Or at least give you a command to check current shutdown state...

In my case it seems to stay that way for a while. I think its caused by EC2 disk mount corruption or something, my drive became Read Only for some reason, EC2 bug??

Thanks
Chris

Top
Thu, 10/13/2011 - 09:11
alex
Offline
Joined: 08/29/2011
Groups: None

Hi again,

Yes, it should be more obvious that its not actually shut down. I'm filing a bug for that today.

not sure about the EC2 disk corruption thing you're seeing. Do you have it in right scale or normal EC2?

-Alex

Top
Thu, 10/13/2011 - 10:52
chrisribe
Offline
Joined: 09/26/2011
Groups: None

We have it in normal EC2.

We use 2x normal (m1.small) ec2 instance and store the membase data on the default /mnt drive.
The two are replicated, one of the servers failed and I could not restart it. After investigating it was discovered that the /mnt drive was now read only. On reboot the drive was back to read/write, and the server restarted.

Odd issue that seems to have been tracked as a bug at amazon but I could not reproduce the problem.
They seem to blame the problem with ext3 errors but...
https://forums.aws.amazon.com/message.jspa?messageID=230910

Chris

Top