Couchbase View Engine Internals

Version 5 by sarath
on Mar 07, 2014 03:39.

compared with
Version 6 by sarath
on Mar 07, 2014 03:41.

This line was removed.
This word was removed. This word was added.
This line was added.

Changes (3)

View Page History
h2. Index updater operation

The index updater is a component within the view engine which can be called on demand to update the indexes up to date with the KV server. Updater is part of design doc group server and one updater instance per group server. The updater is triggered by stale=false query or the periodic trigger update daemon by the ns server. The updater loads a batch of mutations that happened between last indexed sequence and latest mutation sequence and updates the view btrees.

!view-engine updater.png|border=1!
Once these sort files are created for the given batch, it is processed by an external sorting component, which sorts these files with 64MB inmemory buffer. Sorted records will be bulk inserted into the corresponding view btree in the index file and appends a new index header with new root node information for all btrees.

h3. *Initial index builder and incremental updates*

Within the updater we have two types of update. Initial index build and incremental update. If an index for a design document is build for the first time, it is called initial index build. In the case of initial index build, the writer do not batch the incoming mutations, all mutations written into the sort record files and is used to build the initial view btrees. The btree is build in bottom up fashion and is very efficient (Refer: [|])
h2. View query operation

The query for view engine operates in a distributed manner. The node that is receiving the query request acts as distributed query coordinator and broadcasts the query requests to all the other nodes. A k-way merge queue is used to enqueue streaming results from the nodes. Each node gets a request handler process which reads the query rows on arrival from http response and enqueues to the k-way merge queue. The k-way merge queue blocks the enqueue operation by each process based on k-way mergesort property. The results coming out of k-way merge queue is sort ordered and they are streamed to query response as they arrive.