Serious issues with Membase 1.7.0
Last night we received an alert from our monitoring service that our Membase server is grinding CPU and RAM at almost full quota.
We've restarted the service and were horrified to find out that all our data has reverted somewhere to the dates between August 8th and the 16th.
Looking at the server it seems that around that date for some reason Membase stopped writing data into it's permanent database and simply worked from Cache all that time.
Attached are screenshots of the Membase logs:
http://imageshack.us/g/823/diskreads.jpg/
Is it possible to somehow fix this issue?
What can we do to prevent this in the future?
Why did this happen in the first place?
Would a replication server prevent this kind of issues in the future?
Please reply as soon as you can,
Ron
Ron, from the logs that you had sent over earlier, it appears that there was a problem writing to the disk. The logs pretty clearly show:
memcached<0.221.0>: sqlite error: attempt to write a readonly database
Although the logs have since rolled over after the problem began, we also see this statistic increasing:
ep_item_begin_failed 1352489250
This is another indication that the server was unable to write its data to disk.
To answer your question, there's not much to do now other than fixing the underlying disk problem. To prevent in the future, it would be good to monitor these "failure" statistics as well as the disk queue. I also see in the stats:
ep_diskqueue_items 1220327
This means that there are over 1 million items in the disk queue waiting to be written, and it has been growing continuously since the initial problem began. This would have been another indication that something was wrong.
Finally, yes, a replication server would help in the sense that you would have all of the data replicated to another node that you could fail over to.
Perry
Forum support is great for free but sometimes you need a guaranteed response time and dedicated resources for your questions or issues.
Consider purchasing enterprise-level support from Couchbase: http://www.couchbase.com/products-and-services/overview
Call or email "sales -at- couchbase-dot- com" today!