Index Not Keeping up with Activity

We are using 4.11 CC Edition for session management. We use 3 indexes: a primary index and two GSI indexes that are geared toward application queries. We created backups of each index to bring our total index count to 6. We created 9 servers and we put a single index on 6 servers.

The problem we are having is that the index drain rate doesn’t keep up with our mutation rate. We are left with a very high ‘total items remaining’ in the index stats. This is leaving us with very stale data when running the queries.

What are some recommended steps for improving index performance?

We typically have around 300-400k items in our bucket. We run around 1.8k ops / sec.

Unfortunately it would be very difficult to move our HDs to SSD with our current VMWare setup.

Here are some things we have considered doing:

Indexer threads are set at 4 and the box has 4 cores. I could increase the cores to 8 and set the indexer threads to 8?

Index RAM Quota. This is currently set to 1000 MB. On the indexer stats it says the data size is 65MB. I am not sure if more memory would help?

Move to Community 4.5 and look into in-memory indexes.

Probably around 60% of the documents in the bucket are main session objects. The other 40% are session search related documents. We could move the session search related documents to another bucket.

Get rid of the primary index and create a third GSI index that would eliminate the need for the primary.

Move to Enterprise and have dedicated boxes for indexing. This would probably take several months to roll it into the budget.

Any other advice would be appreciated. Thanks!

4.5 or 4.5.1 with MOI indexes.

Sorry to excavate this thread, but I have the same problem now. I have a 6 core machine with 64gb ram as index node in a cluster evaluating enterprise 5.1.1 and 3 MOI indexes (~400k documents each).

On the dip I restarted the server.

The server had 2 threads running at 100% load and the rest of the cores close to idle. Also when querying the server, the load is very low, is that normal? How to debug such scenario?

PS: I am running the node on Ubuntu 16.04

So the virtual memory seems to be maxed out for every thread. But what could be the reason for that?

Please check the “Total Items Remaining” stats on the UI mini graphs for the index to check if the index is behind. MOI indexes need sufficient memory for all the index data to be memory resident. Otherwise, the indexes would move to “Paused” state and not keep up to date with data service.