Couchkvstore read improvements ( BDR- batch disk reads)

Trond Norbye added a comment - 26/Apr/12 3:07 AM Are we sure that we need optimization down within the couchstore library at this time? Isn't part of the problem here the way we actually try to fetch the items from the underlying kvstore by simply scheduling each individual item as a single task using the dispatcher? That would imply a couple of context switches synchronizing between the threads... And the notification logic will go the same way back...) It could be that we should create a separate "service provider" that allows for fetching items.. it should then try to batch up all of these individual tasks to perform a more optimized fetch request (I'm assuming that we're feeding the system with read requests faster than the underlying kvstore may service them).. With the batching in place on a higher level we may get some number on where we're spending our time? Just my 0.5 nok...

Jin Lim added a comment - 07/May/12 3:38 PM Bulk of implementation is currently done by Aaron. Assign it to him.

Aaron Miller added a comment - 07/May/12 3:46 PM - edited @Trond: Not sure how much overhead is currently caused by contention on the dispatcher queues, but they look to be implemented in a way that minimizes that, and I've been told we're underperforming on reads compared to SQLite, so that places at least part of it in the KVStore. @Jin is #1 done already or do I need to do that? The reason I guess is because its more efficient for us to do lookups by Seq than by key/ID.

Trond Norbye added a comment - 08/May/12 5:31 AM @aaron are that comparison done with 2.0 running sqlite backend compared to 2.0 running couch backend? If not we're not comparing apples with apples.. THis seems like something that would be extremely easy to create a test program to compare _without_ running the full system (to get the delta in random read lookup).

Aaron Miller added a comment - 09/May/12 1:09 PM @Trond It's 1.8-sqlite vs. 2.0-couchkv unfortunately, so the numbers could pretty much be fake. Don't know how much has changed. I don't know much about how to run this stuff in isolation, can you advise on how I'd get a test like that running?

Trond Norbye added a comment - 09/May/12 1:24 PM to compare the difference in read performance you can just hack up a small test program that first stores a lot of objects in a database file, then does completely random read of objects back in... if you use the same dataset and the same random sequence you'll get a pretty good idea on how much slower one is compared to the other. that would be the "pure read" test. next up would be to have another thread performing write operations while you're doing the benchmark. The test program should have a set and get method and inside there you'll branch out to couchstore/libsqlite.. that'll make it easy to adapt to new backends. (ideally it should have been using our kvstore interface, but I think its not that easy to create a small isolated test program using that interface anymode ;)

Aaron Miller added a comment - 09/May/12 1:32 PM Couchstore and SQLite operate at different layers of abstraction, so that's not an apples to apples comparison either. It's an intentional decision that couchstore itself doesn't do any caching and that its on the user to cache and batch in order to get good performance. Pretty much every couchstore API directly maps to an operation on the file, it's quite low level. Gist is, the only apples-to-apples comparison is SQLiteKVStore vs. CouchKVStore accessed through the same layers and anything else will give us numbers that are fake.

Trond Norbye added a comment - 09/May/12 1:43 PM then you have to play around with the linker to figure out how to link everything up. Btw. ep-engine is doing caching itself so it will most likely not hit any buffers. From what I can see from the exposed couchstore API there isn't much I can do in order to improve performance from reading out an object from the ID (that will work across compaction etc)

Aaron Miller added a comment - 09/May/12 1:54 PM There's batched reads in the API now, so you can look up multiple items in a single call. It's cheaper because it can collect them all in the same pass over the b-tree. There are also ID and Seq variants of all the lookup ops (docinfo_by_*). The Seq being used for lookups helps a lot (hence item #1 on this bug), probably more than batching, as the sequence index will have a much shallower b-tree (and key comparisons are cheaper).

Trond Norbye added a comment - 09/May/12 2:03 PM Part of the problem there is that you need to refactor the way ep-engine use the kvstore in order to have any effect of that.. So the couchstore instance doesn't keep a cache of any of the btree blocks in memory and reads them off disk _every_ time?

Aaron Miller added a comment - 09/May/12 3:10 PM - edited That refactoring is what item #2 of this bug is. I'm already partway through coding this. ep-engine already queues items that come in for fetching, they just live as opaque callback objects sitting on the IO dispatcher queue. I'm making them have their own queue, and having the IO thread consume the whole thing when it becomes available instead of one item at a time (assuming the underlying store supports batched reads.).

Aaron Miller added a comment - 09/May/12 3:15 PM Also yes, couchstore does not cache those b-tree nodes at all, they're always read off disk (or the OS filsystem cache.) I do have some code I'm working on for the compactor to have it use fadvise to make it behave well with the FS cache. Right now that's just for preventing compaction from ejecting good data from cache when it scans the source file, but could also advise it to cache the range of the result file that contains the b-trees.

Trond Norbye added a comment - 09/May/12 3:28 PM Great. I'll be interesting to see how this improves the performance on both sqlite and couchstore. I'm always a bit scared about thinking the OS filesystem cache helps us too much. It doesn't know what's B-tree nodes or what's data nodes so all it can do is do a wild guess on whats important or not.. and then filesystems vary a lot between OSs. Anyway, looking forward to see the before and after numbers.

Aaron Miller added a comment - 09/May/12 3:36 PM True. But the OS knows a lot more than we do about the file system, its own I/O layer, and the underlying hardware, so there's a tradeoff there. I think fadvise can get us pretty far, but we can add our own caching later if we still need to squeeze more out of it. Would be interesting to see how much batched reads help the SQLite KVstore as well, I believe it'll be pretty easy to add them.

Trond Norbye added a comment - 09/May/12 4:05 PM I've never used posix_fadvise on a windows box, but it probably have similar functionality. Are you thinking of calling something like posix_fadvise() with DONTNEED and WILLNEED every time you write a block? I've never tested heavily use of posix_fadvice on sections in multiple files on a system so this is just a question out of curiosity.. How many of these will the kernel keep track of? anyway. time will show :)

Damien Katz added a comment - 09/May/12 4:19 PM - edited We should avoid for as long as possible any btree caching/file level caching, as filesystems tend to cache stuff whether you like it or not (there are some ways to control it, but they don't always work like you want and vary platform to platform). So if we are caching file level things, and the FS is also caching the same things, then we'll have half as much RAM dedicated to cache than if we just leave it to the FS.

Aaron Miller added a comment - 09/May/12 4:23 PM On linux, WILLNEED and DONTNEED are explicitly "load all these pages right now", and "eject all these pages right now", and will only operate on pages your specified range covers fully. I've done some testing just advising DONTNEED from 0 to the position of the item i've read on the compaction source file, and it seems to work pretty well. Would like to see how this works on some other OSes. OSX doesn't seem to have posix_fadvise at all =/. Incidentally NOREUSE is what I actually wanted, but it doesn't do anything on linux: https://github.com/torvalds/linux/blob/master/mm/fadvise.c

Aaron Miller added a comment - 09/May/12 4:33 PM Ah, I'm wrong. WILLNEED operates on partial pages. DONTNEED only operates on full pages. That seems sane.

Chiyoung Seo added a comment - 01/Jun/12 2:47 AM Jin, Let's take the initial implementation from Aaron, and review and adapt it if necessary and then do the extensive performance tests. Let's discuss this further once we resolve the windows memory corruption issue in the current 2.0 build.

Jin Lim added a comment - 10/Jul/12 7:10 PM The read batch is merged, http://review.couchbase.org/#change,17593

Thuan Nguyen added a comment - 10/Jul/12 11:33 PM Integrated in github-ep-engine-2-0 #346 (See [ http://qa.hq.northscale.net/job/github-ep-engine-2-0/346/ ])      MB-5164 Optimize background fetch by batching multiple gets (Revision ece63f5c7ebc28eaf2bc05b4d573dde8d37bf022)      Result = SUCCESS Jin Lim : Files : * Makefile.am * mc-kvstore/mc-kvstore.cc * priority.hh * blackhole-kvstore/blackhole.cc * ep_engine.cc * stored-value.hh * callbacks.hh * ep_testsuite.cc * bgfetcher.hh * bgfetcher.cc * ep.cc * priority.cc * vbucket.hh * ep.hh * stats.hh * queueditem.hh * kvstore.hh * sqlite-kvstore/sqlite-kvstore.cc * couch-kvstore/couch-kvstore.cc * couch-kvstore/couch-kvstore.hh * vbucket.cc * win32/Makefile.mingw