Membase proxy write to downstream error on 1.6.5, out of usable disk space
My membase cluster is down with the errror SERVER_ERROR proxy_write_downstream. The logs report " Service memcached exited " every few seconds. Moreover, there is no usable disk space left, even though my keys are set to expire after 15 minutes. Are there known issues with membase purging data on disk?
I see the long thread that appears to end in resolution for an earlier version in January.
Are there any known causes of this error in 1.6.5? My whole app is down now, I recovered by launching a new cluster, but will presumably hit this error again on my new cluster as my disk usage is growing and will hit max in a few hours again.
Thanks,
Eric
I've trashed the instances as I hit my EC2 limit and had to spin up some new ones to keep going. I'll see if I hit it again, but now I have 400GB of free storage, not 8GB.
FWIW, the default membase install uses the local disk, which on the alestic ubuntu install means you only have about 8GB free, which is not a lot independent of membase's cleanup policy. If you first create a membase-accessible folder on /mnt, you get access to 160-400GB of free disk. That said, your membase-accessible folder will not survive a restart, so you'd have to write a script to create a membase-accessible folder on /mnt before membase launched, or figure out some other way.
This is all to say it would be really nice to have a membase-official instance-store image on EC2 that dealt with these issues.
Thanks,
Eric
Thanks for the feedback Eric.
We've actually got a lot of these features already built into our templates in RightScale (who we've partnered with) so it may make sense for you to take a look at that.
Perry
Eric, 1.6.5 should be properly purging data from disk every hour. We did have a bug in 1.6.4 where this didn't happen properly, but it has been fixed and verified in 1.6.5.
Can you run /opt/membase/bin/ns_server/collect_info <output_file> on the nodes of your cluster and send the resulting files to perry -at- couchbase -dot- com? I cant take a look to see if there is anything else going on but running out of disk space is known to cause problems.
Perry
Forum support is great for free but sometimes you need a guaranteed response time and dedicated resources for your questions or issues.
Consider purchasing enterprise-level support from Couchbase: http://www.couchbase.com/products-and-services/overview
Call or email "sales -at- couchbase-dot- com" today!