Couchbase View Engine Internals

Version 4 by sarath
on Mar 07, 2014 03:34.

compared with
Version 5 by sarath
on Mar 07, 2014 03:39.

This line was removed.
This word was removed. This word was added.
This line was added.

Changes (9)

View Page History
h2. Index updater operation

The index updater is a component within the view engine which can be called on demand to update the indexes up to date with the KV server. Updater is part of design doc group server and one updater instance per group server. The updater is triggered by stale=false query or the periodic trigger update daemon by the ns server. The updater loads a batch of mutations that happened between last indexed sequence and latest mutation sequence and updates the view btrees. !view-engine updater.png|border=1!

!view-engine updater.png|border=1!

The updater has mainly three components Loader, Mapper and Writer. They operate in a pipeline fashion. The loader is responsible for collecting a batch of changes from last indexed sequence of mutation to the latest mutation sequence and load it into a queue. The updater stores the highest sequence number for every partition it has already indexed . This way it's easy to find out whether an update it needed or not. The updater loops through all partitions and requests all the updates that happened since the last time the updater ran.

h2. View query operation

The query for view engine operates in a distributed manner. The node that is receiving the query request acts as distributed query coordinator and broadcasts the query requests to all the other nodes. A k-way merge queue is used to enqueue streaming results from the nodes. Each node gets a request handler process which reads the query rows on arrival from http response and enqueues to the k-way merge queue. The k-way merge queue blocks the enqueue operation by each process based on k-way mergesort property. The results coming out of k-way merge queue is sort ordered and they are streamed to query response as they arrive. !query.png|border=1!



The query server in each node is responsible for retrieving data from the view btrees. When a query request arrives, the query server will request for the latest view group information from the view group server corresponding to its design document. The group info request is a blocking request. When stale = false, it will block until current vbuckets seq num <= indexed seqs num by triggering an updater in the background.&nbsp;

| {color:#000000}Events{color} | {color:#000000}Partition state on Node A{color} | {color:#000000}Partition state on Node B{color} |
| {color:#000000}Initial state{color} | {color:#000000}main index: active, indexable{color}\\ {color:#000000}replica index: not-present{color} | {color:#000000}main index: not-present{color}\\ {color:#000000}replica index: passive{color} |
| {color:#000000}Failover initiated{color} | {color:#000000}not available{color} | {color:#000000}main index: not-present{color}\\ {color:#000000}replica index: active, unindexable{color} |
| {color:#000000}Started indexing partition on main index.{color} | {color:#000000}not available{color} | {color:#000000}main index: passive, indexable{color}\\ {color:#000000}replica index: active, unindexable{color} |
| {color:#000000}Indexing on main index completed{color} | {color:#000000}not available{color} | {color:#000000}main index: active, indexable{color}\\ {color:#000000}replica index: not-present{color} |

h2. Index compaction in view engine