Large time to build and move index

We are trying to deploy new version of Couchbase using Couchbase Operator for Kubernetes.

In our old Couchbase 6.0 CE single-node cluster we have around 60 millions of documents in one of the buckets and have indexes built on this bucket. Usual build time for biggest index is less than 1 hour.

As for new cluster memory-optimized index storage would require too much RAM we switched the engine to standard GSI (plasma storage).

Now the problem is that index build time is very long - more than 24 hours for a set of 5 indexes. We have indexes varying in size from 100K to 5G.

As we initially used standard storage class for persistent volumes we thought the speed could be related to disk throttling. So we decided to give a try and use premium (SSD) storage option for persistent volumes, but now after moving data from one of the data nodes to new one during re-balancing some indexes “got stuck” in moving state (which is also taking enormous amount of time similar to build time). If moving 240G of data from single node took ~1 hour, expected time according to the remaining mutations graph would take ~24hours or more.

At the same time neither of data, not index nodes are showing significant CPU usage - maximum ~20% despite we use smallest node type for Couchbase cluster (n2d-highmem2, 2vCPU, 16G RAM).

Could you give any recommendation what can be a bottleneck in this case? Which resource metrics we should pay attention at?

Hi @Nikolay_Kushin,

I am sorry to hear that you are facing issues with index build speed.

As far as I understand, plasma storage is not supported with CE. It is EE only feature.

Still assuming plasma storage backend, it is recommended to allocate enough memory quota so that index service can maintain minimum of 20% resident percent. So, please check the memory quota for the index service, as well as resident percentage of the indexes. With lesser resident percentage, index build can get slow.

Memory quota should be visible on settings page on the UI. Index resident percent should be visible either on index UI page or can be retrieved using /stats indexing rest endpoint (https://:9102/stats).

If above explanation does not help, please provide the cbcollect logs