open service after loading x items during warmup( starting a server with many items can potentially take hours to fully load hashtable and values)

Matt(old) Ingenthron (Inactive) added a comment - 25/Aug/10 10:53 PM

Dustin Sallings added a comment - 08/Sep/10 4:26 AM https://gist.github.com/267d6fd0528be71aa58d

Matt(old) Ingenthron (Inactive) added a comment - 08/Sep/10 10:18 PM re-targetting to next milestone, per discussion of Dustin's options.

Sean Lynch (Inactive) added a comment - 18/Sep/10 10:25 PM Should this stay on beta4?

Matt(old) Ingenthron (Inactive) added a comment - 18/Sep/10 10:31 PM Ideally yes, but it can slip if need be so it's more of a GA release blocker. Retargeting.

Sean Lynch (Inactive) added a comment - 28/Sep/10 1:36 AM Chiyoung or Dustin, is work happening on this?

Chiyoung Seo added a comment - 28/Sep/10 1:44 AM No, I haven't had any chance to work on this, but think this might take some time to be completed and tested. I would suggest to move this to post GA

Perry Krug added a comment - 06/Jan/11 12:06 PM Bringing this back to the forefront as it is still outstanding and something often requested by customers, especially with very large datasets.

Matt Ingenthron added a comment - 03/Aug/11 3:37 PM Assigning to Frank for prioritization. I've (for now) set this to a blocker, mainly to force some decisions to be made on it w.r.t. prioritization in the 2.0 release cycle.

Farshid Ghods added a comment - 03/Aug/11 3:42 PM changed the fix by to 2.0 GA

Matt Ingenthron added a comment - 03/Aug/11 3:46 PM There are two candidate proposals for how to address this. One would be to leverage the new (since 1.7) loading-servicing state during startup. WIth that, we can return errors for things like gets, but allow all idempotent mutation operations to proceed. This works well with the behavior of many applications, as it would allow users for whom data is loaded to start working, would allow newly added data to be immediately available, and it's just the items that aren't yet loaded that would not be available. Another candidate proposal is to add functionality to write the indexes to persistent storage during a clean shutdown. This would allow us to load all data in an ejected state very quickly (though could still be up to 5 minutes). It would move the thrashing a bit further down the road, but for idempotent mutations and failing gets, it'd still be fast. The two solutions can be combined, and I can think of at least a couple of other possible solutions in the 2.0 timeframe.

Perry Krug added a comment - 03/Aug/11 5:00 PM Isn't the second proposal already done with CouchDB under the hood and all we need to do is read in the index and start servicing things (albeit slowly) as we warm up the rest?

Matt Ingenthron added a comment - 03/Aug/11 5:07 PM I don't think the second proposal is already done in that walking the B+Tree is fast, but still seeky rather than linear to get the info we need to have ejected items in memory (key, expiration, flags and a couple of other things). If it's a lot of data in an aged DB file, that could be rather IO intensive. That said, I'll leave that with Frank/Dustin to determine. It should be verified though.

Michaël de Groot added a comment - 19/Mar/12 10:53 AM - edited We - GamePoint - have a membase 1.7.1 server, we have a bucket with 24GB RAM quota, also it has 30.4 million items which was stored in 4 data files of 139GB per data file, 556GB in total. After rebooting the server it took 391577 seconds to reload this bucket - 4 days and 12 hours! This feature would be greatly appreciated.

Peter Wansch added a comment - 05/Jul/12 7:55 PM Already addressed by new warmup implementation.

Dipti Borkar added a comment - 05/Jul/12 8:02 PM Opening service based on a configurable number of items / % of RAMs is not in 2.0 and is meant for a future release.

Dipti Borkar added a comment - 05/Jul/12 8:02 PM will be prioritized for a future release