Each design document maps to one indexer, so when the indexer runs it updates all views defined in the corresponding design document. Indexing takes resources (CPU, disk IO, memory), therefore Couchbase Server limits the maximum number of indexers that can run in parallel. There are 2 configuration parameters to specify the limit, one for regular (main/active) indexers and other for replica indexers (more on this in a later section). The default for the former is 4 and for the later is 2. They can be queried like this:
shell> curl -s 'http://Administrator:asdasd@localhost:9000/settings/maxParallelIndexers' {"globalValue":4,"nodes":{"n_0@192.168.1.80":4}}
maxParallelIndexers is for main indexes and
maxParallelReplicaIndexers is for replica
indexes. When there are more design documents (indexers) than
maxParallelIndexers, some indexers are blocked until there's a
free slot, and the rule is simple as first-come-first-served.
These slots are controlled by 2 barriers processes, one for main
indexes, and the other for replica indexes. Their current state
can be seen from_active_tasks(per node), for
example when there's no indexing happening:
shell> curl -s 'http://localhost:9500/_active_tasks' | json_xs [ { "waiting" : 0, "started_on" : 1345642656, "pid" : "<0.234.0>", "type" : "couch_main_index_barrier", "running" : 0, "limit" : 4, "updated_on" : 1345642656 }, { "waiting" : 0, "started_on" : 1345642656, "pid" : "<0.235.0>", "type" : "couch_replica_index_barrier", "running" : 0, "limit" : 2, "updated_on" : 1345642656 } ]
The waiting fields tells us how many indexers
are blocked, waiting for their turn to run. Queries with
stale=false have to wait for the indexer to be
started (if not already), unblocked and to finish, which can lead
to a long time when there are many design documents in the system.
Also take into account that the indexer for a particular design
document might be running for one node but it might be blocked in
another node - when it's blocked it's not necessarily blocked in
all nodes of the cluster nor when it's running is necessarily
running in all nodes of the cluster - you verify this by querying
_active_tasks for each node (this API is not meant for direct user
consumption, just for developers and debugging/troubleshooting).
Through_active_tasks(remember, it's per node,
so check it for every node in the cluster), you can see which
indexers are running and which are blocked. Here follows an
example where we have 5 design documents (indexers) and
>maxParallelIndexers is 4:
shell> curl -s 'http://localhost:9500/_active_tasks' | json_xs [ { "waiting" : 1, "started_on" : 1345644651, "pid" : "<0.234.0>", "type" : "couch_main_index_barrier", "running" : 4, "limit" : 4, "updated_on" : 1345644923 }, { "waiting" : 0, "started_on" : 1345644651, "pid" : "<0.235.0>", "type" : "couch_replica_index_barrier", "running" : 0, "limit" : 2, "updated_on" : 1345644651 }, { "indexer_type" : "main", "started_on" : 1345644923, "updated_on" : 1345644923, "design_documents" : [ "_design/test" ], "pid" : "<0.4706.0>", "signature" : "4995c136d926bdaf94fbe183dbf5d5aa", "type" : "blocked_indexer", "set" : "default" }, { "indexer_type" : "main", "started_on" : 1345644923, "progress" : 0, "initial_build" : true, "updated_on" : 1345644923, "total_changes" : 250000, "design_documents" : [ "_design/test4" ], "pid" : "<0.4715.0>", "changes_done" : 0, "signature" : "15e1f576bc85e3e321e28dc883c90077", "type" : "indexer", "set" : "default" }, { "indexer_type" : "main", "started_on" : 1345644923, "progress" : 0, "initial_build" : true, "updated_on" : 1345644923, "total_changes" : 250000, "design_documents" : [ "_design/test3" ], "pid" : "<0.4719.0>", "changes_done" : 0, "signature" : "018b83ca22e53e14d723ea858ba97168", "type" : "indexer", "set" : "default" }, { "indexer_type" : "main", "started_on" : 1345644923, "progress" : 0, "initial_build" : true, "updated_on" : 1345644923, "total_changes" : 250000, "design_documents" : [ "_design/test2" ], "pid" : "<0.4722.0>", "changes_done" : 0, "signature" : "440b0b3ded9d68abb559d58b9fda3e0a", "type" : "indexer", "set" : "default" }, { "indexer_type" : "main", "started_on" : 1345644923, "progress" : 0, "initial_build" : true, "updated_on" : 1345644923, "total_changes" : 250000, "design_documents" : [ "_design/test7" ], "pid" : "<0.4725.0>", "changes_done" : 0, "signature" : "fd2bdf6191e61af6e801e3137e2f1102", "type" : "indexer", "set" : "default" } ]
The indexer for design document _design/test is represented by a
task with a type field of
blocked_indexer, while other indexers have a
task with type indexer, meaning they're
running. The task with type
couch_main_index_barrier confirms this by
telling us there are currently 4 indexers running and 1 waiting
for its turn. When an indexer is allowed to execute, its active
task with type blocked_indexer is replaced by a
new one with type indexer.