Ongoing (~6 hours) high-load scanning of index items after creating primary index

Hello

I created two simple indexes and built them together. Here are the commands:

CREATE PRIMARY INDEX `primary-index` ON `my_bucket` WITH { "defer_build":true };
CREATE INDEX `idx_type` ON `my_bucket`(type) WITH { "defer_build":true };
BUILD INDEX ON `my_bucket`(`primary-index`,`idx_type`) USING GSI;

The indexes were built in about 20 minutes. Since then I see high cpu usage and high network usage, especially on the server which keeps scanning items of the primary index. In the last hour this server had this statistics on the Couchbase dashboard: 584K items scanned/sec. The other last ~6 hours look similarly. If the scan average is 500k items per second then it should easily scan all items in the bucket within 1 minute.

Some more info:

  • There are basically no mutations remaining. It seems they only appear when new documents are created but they are drained immediately.
  • The traffic between CB servers and SG servers stayed the same over the last 6 hours. There is only increased traffic for the last 6 hours between CB servers.
  • The scans are on average 500k items per second. But there are actually about 4 individual scans per minute. Each scan shows several million documents scanned.
  • The gets views show about 5k gets per second for each server. That is about 2 magnitudes higher than normal.
  • In the cluster are 5 CB servers running Couchbase Server Community Edition 6.0.0 build 1693.

A stop and start of the couchbase server service on the particular server did not resolve the issue. Please advice.

Thanks!

-Ben

This issue was solved by restarting one SG service. This in turn started the build of about three SG indexes. This stopped the high traffic, the high ongoing scans, high CPU usage, etc. All is back to normal and has been for the past few minutes. Since I took part in all 3 Couchbase courses offered by Couchbase but wasn’t aware of this simple fix I’d be happy to be pointed to some documentation where I can read about this.

Thanks!