I have an Index node, and i have a set of handful number of indexes ( 5 indexes that consume around 28 GB of disk space and I have 10 GB of RAM on this node.)
I am creating a new Index on this node and this is supposed to take around 12 GB of disk.
This new Index creation is not successful and all my other indexes in this node goes into Warmup mode and comes back and this is happening again n again.
Can you help me understand the behavior.
After building a certain percentage, It is not moving forward and I see this warning in console …
“Warning: Cannot communicate with indexer process. Information on indexes may be stale. Will retry.”
Here’s my take: The indexing process is consuming I/O and CPU resources to such an extent that the existing indexes cannot be maintained properly, which is why they go into the Warm Up state and you see those indexer-process warnings. But I would expect that to only happen if the data services are sending data mutations to the index node.
If that’s not the case and the system is quiet, perhaps it’s a configuration issue.
What version of Couchbase Server are you using?
What’s the CPU utilization when you create the index?
What is your “Indexer Threads” count (4.6) under Settings/Index Settings/Advanced Index Settings?
I have dedicated Index and Data nodes (3 each), I am assuming this will Isolate the workloads.
there is no mutations happening on data nodes at the time of index creation and building.
version -> 5.0 Beta Build
Indexer Threads -> 0 ( using all available cores)
swappiness -> 0 ( using default setting)
Regards
Shiv[quote=“jkurtz, post:2, topic:13564”]
resources to such an extent that the existing indexes cannot be maintained properly, which is why they go into the Warm Up state and you see those indexer-process warnings. But I would expect that to only happen if the data services are sending data mutations to the index node.
[/quote]
@sgudavalli, it looks like the Indexer process is crashing/restarting over and over. Warmup state of indexes indicate the indexer trying to recover these indexes. Can you check in the indexer.log for any panics/errors or you can share the log here and we can take a look.