Strange behaviour when disk is full

Johan, in general, do not run out of disk space ;-) Even traditional RDBMS don't work very well when that happens.

The issue here is that when you shut the service down the first time, the memcached process will keep running as it tries to write the data to disk (which it won't be able to since the disk is full). This is by design, because the server doesn't really "know" that the disk is full, but it doesn't want to shut down while items are in the queue because that would possibly cause data loss.

We would really recommend monitoring the system more closely so that you can both avoid running out of disk, and take action quickly if it does happen. The preferred action would be to add another server to the cluster and rebalance to give it more disk space overall.

Perry

Dear Perry,
this is just a suggestion, not sure if it is possible to implement....
Since "memcached.exe process" do not got killed while the membase service is shut down, during this state, is it possible to make "membase node-init" to move the data file to another location (with more free space)? So that "memcached.exe process" can continue to persisting the data to disk.
I do realize that "membase node-init" moving data file without the data itself, but I think this way the "memcached.exe process" does not becoming "ghost" until the server got restarted or the process got killed, and when membase server get started back, all "servers" are back online...

BTW, I also try to add new server and rebalance.... and no luck.... the rebalance failed...
Any suggestion on this?

Thanks,
Johan

Hi, cjexxi,

I am curious how you could fill up your disk with only 100MB RAM allocated to membase.

I setup a single host test evironment with 1GB RAM + 1TB Disk, I can just insert a little records.
please see http://www.couchbase.org/forums/thread/how-many-records-can-be-holden-me...

I wrote client in C using libmemcached.

- ant

here is the code snippet (C#) I'm using:

            using (MembaseClient client = new MembaseClient("default", ""))
            {
                client.Store(Enyim.Caching.Memcached.StoreMode.Set, "key1", "Hello World awal");
 
                for (int i = 1; i < 800000; i++)
                {
                   client.Store(Enyim.Caching.Memcached.StoreMode.Set, "key" + i, "replace this with very long string, " + DateTime.Now.AddMinutes(2).ToString(), DateTime.Now.AddMinutes(2));
                }
            }

before doing any further, I set the "data file" to a flashdisk with 128MB size only.
then run the code....

Johan, what were the errors you got on rebalance and can you attempt to retry it?

Perry