Membase will exhaust free disk space soon - how to determine what is being stored?
I presently have a single m2.xlarge membase instance powering a membase install for my app. This instance provides 15.7GB for membase and 403 GB for storage. I brought this instance up last Wednesday. After approximately 7 days of continuous operation, my membase instance is consuming 195GB of on-disk storage, and reports 13.8M entries in the cache. While I do cache a certain class of data for two weeks, there are only about 200k objects that fall into this class. The remainder are short-lived pieces of data that expire after 15 minutes.
Bottom line is I have absolutely no clue what is represented in those 13.8M entries or 195GB of disk data. I see my disk write queue size hovering around 600/sec, but I do not ever see any disk reads.
If I do not diagnose and correct this issue my node will exhaust disk space in the next week and crash.
How can I inspect what is actually being stored on disk, and determine if this is either a bug in my code, or a bug in the membase code that is supposed to expire old data. Is it possible that the data expiry code of membase is just being really, really, really lazy, given the very large amount of disk space available to it on this kind of EC2 instance?