Total mutations remaining / drain rate

Couchbase Server
2

@naftali,

Slow drain rate can be observed in the following cases:

a. Indexer memory quota is full and indexer is throttled on disk
b. The ingestion rate into indexer is slow (Possibly due to large number of indexes, less network bandwidth)
c. Indexer is throttled on CPU

Looks like the first scenario is not the case. Is the drain rate stuck at zero (or) is it fluctuating slowly? Also, how many indexes are you building?

You can look at some of the below:

  1. The CPU utilisation on the indexer node while the index build is happening - Is there enough CPU (or) does the CPU look saturated

  2. Can you check the mutation_queue_size stat for the bucket for which the index is building - If the mutation queue size is close to zero, then it is possible that the network bandwidth is less. Can you also check the network bandwidth

  3. In the indexer.log for the node on which index is being built (Located at /opt/couchbase/var/lib/couchbase/logs/), check if you are seeing the “Indexer <> blocked” messages

After you eliminate the above 3 possibilities, if you still see less drain rate, we may need the cbcollect logs to understand the reason why index building is slow. The procedure to do cbcollect is mentioned in the following link:

Thanks,
Varun