"item_alloc_sizes" not correct when using append

Xiaoqin Ma added a comment - 19/Feb/13 7:33 PM We will postpone this one. As we are working on reducing the metadata size in memory, we will revisit it.

Perry Krug added a comment - 20/Feb/13 2:00 AM Curious what you mean by that. As far as I can tell, this issue has nothing to do with metadata size, nor memory usage.

Xiaoqin Ma added a comment - 20/Feb/13 1:26 PM When we redesign the metadata, the code path for set/get/prepend/append will change. I would like do it later. But if you have an urgency for this, let me know. The complexity comes from update/delete an item, should we delete the old item size from the item_alloc_size or not? If it is yes, it is hard to support it without the meta information in memory, and we may not want to do a bgfetch to update this stats.

Mike Wiederhold added a comment - 20/Feb/13 6:53 PM Perry, I looked at the code for this and I think the issue here is that you are misinterpreting what this stat does. In the memcached API there is an api call itemAllocate. When memcached receives an operation is calls itemAllocate with whatever was in the value section of the message sent by the client. Memcached keeps that pointer and then passes it back into ep-engine along with the operation to be done on it. In short this stat is tracking how big of allocations we are making when we call this function. I think what your interpretation of this histogram is that it represents what the value sizes look like in the database. If this is what you want then I have some other concerns about adding that histogram.

Perry Krug added a comment - 21/Feb/13 7:53 AM Thanks Mike and Xioaqin. Xiaoqin, I understand now what you were referring to. Mike, you are right and I also knew that already ;-) This stat was added by Chiyoung a few versions ago specifically to give an approximation of the item size coming through the database...we understand that it's not a perfectly accurate accounting. I filed this bug because I was attempting to use this stat to get an approximate understanding of a customer's workload for performance analysis. However, they were heavily using append and so the actual item sizes were not showing up (obviously). I was hoping there was something easy we could fix, but maybe there needs to be a different approach to approximating the item sizes within the database. It's actually something quite useful for support and our customers to be able to see, especially when things are going wrong.

Mike Wiederhold added a comment - 21/Feb/13 1:26 PM So if you want the stat there is one big issue. Let's say you want to do a set and then an update. In order to keep the stat correct we would want to remove the size of the original item from the histogram and add the new size. This isn't really a big deal right now, but in the near future we plan on evicting all of the meta data from memory. If we do this then that means that we wouldn't keep any information on the value or its size in memory for an evicted key. As a result this means that an update for an evicted key would require us to do a background fetch in order to keep this histogram correct. As a result I don't think it would make sense for us to have such a histogram. Let me know what your thoughts are here.

Perry Krug added a comment - 22/Feb/13 4:32 AM You make a very good point Mike, and i had thought about that. However, for certain updates we're going to have to do a background fetch anyway right? Any append/prepend/incr/decr would definitely need a background fetch (as it does anyway today) and unless we're keeping the CAS ID still resident, CAS would. I'm not sure what the plans are to keep *any* information about the keys resident, but if not, then add and replace would also require finding the key on disk before completing the operation. I think that just about covers all of our operations ;-) (on the side, curious how that will impact our expiration capabilities) Only set() would never require a background fetch and then you have the size of the new item within the operation. So then the question of removing an old size comes up. Given that we don't do it today with item_alloc_sizes, I'd say it's okay to just let that histogram grow and grow. It's able to be reset now, and that would suffice in the future IMO. Two other thoughts I had: -We could keep separate histograms for incoming versus outgoing items. Only needing to update the outgoing when we send data, it would give another approximation of how large the items are in bucket. Biggest downside to that is for write heavy workloads without corresponding reads. I could say here that's sufficient....but you know the first write-heavy customer that I'd like to see the item sizes of will cause a new bug to be opened ;-) -What about taking a totally different approach on this and adding something to our item pager/dispatcher to actually maintain a much more accurate histogram of item sizes? A few approaches to this would be only to deal with items resident in RAM, or to have it part of the ejection process to capture items as they leave, etc. Perhaps something that could be run periodically so you don't ever need to remove sizes from the histogram, just overwrite it with what's there currently... Thoughts?