How does Membase behave when it runs out of Disk space?
Hi, I had a question about Disk space, suppose you are not monitoring your disk space and the SQLite database grows enough to consume your disk space. how does membase behave at this point? Will it fall over if it needs to swap out to disk?
I looked into vacuuming the SQLite database but I seem to encounter a few problems with the mbdbmaint --vacuum command. Below are the errors returned by the tool and also it only seems to be trying to deal with the default bucket data. how can I ensure it tries to defragment data across all buckets?
Pausing persistence... paused. Vacuuming c:/Program Files/Membase/Server/var/lib/membase/data/default-data\default-0.mb 'sqlite3.exe" "c:' is not recognized as an internal or external command, operable program or batch file. Vacuuming c:/Program Files/Membase/Server/var/lib/membase/data/default-data\default-1.mb 'sqlite3.exe" "c:' is not recognized as an internal or external command, operable program or batch file. Vacuuming c:/Program Files/Membase/Server/var/lib/membase/data/default-data\default-2.mb 'sqlite3.exe" "c:' is not recognized as an internal or external command, operable program or batch file. Vacuuming c:/Program Files/Membase/Server/var/lib/membase/data/default-data\default-3.mb 'sqlite3.exe" "c:' is not recognized as an internal or external command, operable program or batch file. Unpausing persistence.
Note: contrary to the location and name of this tool described in this link: http://www.couchbase.org/wiki/display/membase/Membase+Command-line+Inter...
I executed this command from C:\Program Files\Membase\Server\bin using the mbdbmaint.exe tool. And the membase cluster is made up of standard non-sasl buckets.
Thanks for the reply and workaround Perry. Understood that this will affect persistence and looking forward to 2.0 then! :)
Had another question popup as a result, does the backup/restore process happen while the cluster is still online? And does the vacuum described in the link below happen on the backed up version of the sqlite db if the cluster is online during the process?
http://www.couchbase.org/wiki/display/membase/Membase+Server+version+1.6...
-Backup happens while the cluster is online thanks to the sqlite backup API...this will remain the same in 2.0 but for different reasons ;-)
-Restore can also happen while the cluster is online, but it's a bit different. The mbrestore script simply reads the data from a set of DB files (i.e. default, default-0.mb, default-1.mb, default-2.mb, default-3.mb) and then "restores" it into whatever cluster you want (either the same one or a different one) by writing it through the memcached interface and through Moxi. If you're not using mbrestore (just putting the DB files "in place" and starting up the server) then it's technically not "online" since you need to wait for the server to warmup before accessing the data. This one might get a bit confusing...does that make sense?
-Lastly, vacuum. Using the mbbackup script, the backup files are vacuumed...not the "live" ones. If you use dbmaint (or the procedure I provider above), it will vacuum the "live" files. As I described, there are enough situations where you won't be able to vacuum the live files. Vacuuming the backup files will greatly speed up your recovery in the event that you need to. We highly recommend vacuuming any backup files to prepare for this, and we recommend vacuuming the live ones if possible.
How does that all sound?
Perry
Thanks Perry, that clarifies things well enough for me. Cheers!
I know this thread is a couple months old but I'm running 16 nodes with version 1.7.1 and still running out of disk space even though we are only consuming ~100GB of RAM. We have a total of 960GB of RAM and a little over 1TB of disk space shared across all servers. We've had two cases where in a three day period all disks were consumed but memory consumption was at ~30GB for one case and ~80GB for the other. One situation we needed to flush the cache but forgot that SQLite doesn't give back the space to the OS and we didn't have enough free disk space to run a vacuum. How are clients managing this unless they have unlimited disk space? I'm getting more info from our Devs on how they are expiring data but they are currently using the .Net Enyim client since we are a Windows shop.
That link is a bit out of date, I'll put it on the list to update.
First off, don't run out of disk space ;-) Various things start breaking and we've seen bad things happen. The DB files safe (thanks to sqlite) but configuration and general simplicity in recovering starts to degrade. There is no sense of "swap" on disk space...it will just fill up and stop accepting writes. We've got some new alerts in the UI for when this happens, but in general you should make sure that it doesn't.
Now, as for the mbdbmaint command, I'll have to look into that error in more depth. It doesn't appear to be working at all. I've filed this bug for it: http://www.couchbase.org/issues/browse/MB-4039
You should be able to accomplish this procedure manually in the meantime:
Stop persistence: C:/Program Files/Membase/Server/bin/mbflushctl.exe localhost:11210 stop
Vacuum files: "C:/Program Files/Membase/Server/bin/sqlite3.exe default-0.mb" vacuum (repeat for all files, default-0.mb through default-3.mb)
Start persistence: C:/Program Files/Membase/Server/bin/flushctl localhost:11210 start
Note that using the mbdbmaint script or doing it manually will cause Membase to stop writing to disk while the vacuuming is going on. When under load, this could result in a growing disk write queue and may run you into other problems. It is best to test this first to understand the behavior of your system. 2.0 will have the ability to compact (CouchDB's version of vacuum) while online so as not to have this issue.
Perry
Forum support is great for free but sometimes you need a guaranteed response time and dedicated resources for your questions or issues.
Consider purchasing enterprise-level support from Couchbase: http://www.couchbase.com/products-and-services/overview
Call or email "sales -at- couchbase-dot- com" today!