membase delete bucket doesn't actually free the RAM being used.
I have the following job running every hour:
create 5 buckets with a new version number and delete the 5 buckets created last hour. I assumed that deleting a bucket would free the memory that was used by the bucket but that doesn't seem to happen. And I see that my RAM and swap keeps getting fuller every hour till membase just sort of goes down.
I need the bucket delete to actually delete bucket. I read in some thread that 1.7.x onwards creating and deleting anything less than 20 buckets shouldn't be a problem. But I seem to hit problems with just 5 buckets.
Also can I know if there is a some garbage collection happening that would free unused memory and if yes, when can I bee sure that a free page has been purged.
Matt,
Our data in every bucket changes completely after a few hours(could be a different number of hours/bucket) and hence we reload the data in a new bucket and delete the older one. Why we don't push data in new bucket is that the old data would still occupy space in the bucket quota and prefer a clearer insight into how much memory is really being used per bucket.
When we see issued with creating/deleting buckets we see errors in our logs that "connection has been reset by the peer" in spymemcached logs but I don' think membase or mamcached really goes down. Just that top starts showing huge io wait time and swap usage keeps increasing. After such few runs all buckets go "orange" and used RAM quota shows "0". To do this I don't run do refresh every hour but in a loop one after the other, I hope that is alright thing to do.
Thanks,
Chhaya
Without getting into great detail on how memory allocators work, by design most of them don't release their memory back to the operating system when they free allocated memory. That means that if you look at the 'used' memory from an operating system perspective, it may be higher than what the running program is actually currently using. Exactly how this happens differs based on the OS and memory allocator in use. You usually can know if the page has been "purged" as you say by looking at the memory allocated, the memory actually given to the process, and the resident set using OS tools.
The memory for all buckets is under one shared process, which allows us to be very fast and efficient with memory usage on the system. It's not handled through garbage collection, but rather through allocation and freeing of memory within the core process of each node.
That said, if you're finding that regularly creating new buckets and deleting old buckets is running into the server "going down", that's a problem. It sounds like some data structures related to the bucket just aren't getting released properly. I've just opened an issue here:
http://www.couchbase.org/issues/browse/MB-4317
Please feel free to add any comments to it.
Out of curiosity, why are you constantly creating and deleting buckets?
Thanks!
Matt