Can bg flush / drain be forced?
still running into this issue:
http://www.couchbase.com/forums/thread/hard-out-memory-error
one of the suspicions that I have is that the bg thread is not actually running for this bucket, also persistence is on and cbflushctl start has definitely been called (just in case). what i'd like to do now is force a flush to occur, but flush_all sounds scary... particularly since this is a case where I know that 100% of my data is NOT flushed to disk cache. is there a safe way to force a bucket to persist all of it's contents?
good point. i had already noticed that those indeed are BOTH zero. so, a failure of the bg thread to do it's job is not likely. however, the problem remains - my supposed 'persistent' bucket has 2.4G in RAM and 234M on disk. so perhaps whatever is supposed to be enqueuing the items is failing?
could be coincidence or not, but expired items are also not being cleared from RAM for this bucket.
i will send you my stats offline, so as not to clutter up the forum w/ a ton of data. maybe there's something else suspicious in there.
I don't see anything suspicious and I see "ep_flusher_state: running" in the stats. One other thing you can do is to run the "./cbstats localhost:11210 timings" command and this will show you all of the background jobs that I have run. If you post the stats here I can take a look too.
thanks for the response
i have since decided from the limited view that I have of what is happening that there is some sort of bad interaction between expiring items and persistent (Couchbase) buckets. I tried to reproduce in a more controlled environment, but had limited success. Based on this finding, I have confirmed with our development team that it is OK (not preferred) for the data in this bucket to be non-persistent, so I backed up the data and restored it to a memcached bucket. So far this is working ok. Upon restore, all of the items that were expected to be NOT-expired properly showed up in the new bucket, so things WERE being properly persisted to disk.
There's definitely something leaky going on with the situation that we had, which involved items with a 5 day expiration set. Disk was being purged of expired items properly, but memory was not being purged properly in all cases.
First off, flush_all means that you want to delete all of the data in a given bucket so you definitely don't want to do that. Second, flushing should always be happening since we optimize to have items written to disk as quickly as possible. If you think that you data is not being written to disk then you should inspect the "ep_diskqueue_items" and "ep_flusher_todo" stats. You can get them by running the following command.
./cbstats localhost:11210 all
If these two stats are 0 then that means there are no remaining items waiting to be written to disk.