The DATP blocked messages mean that mutations are being pushed but indexer channels are completely full. So, indexer is not receiving new mutations on DATP till the channels become free.
What is the index build time with out changing the index threads while you change the hash key. I am asking this because you did two changes: (a) Change the partition key (b) Increase the number of threads.
We don’t know what is causing the perf improvement. Can you try not changing the indexer threads and change the hash key. I expect this to be around ~20Min. Also, what is the indexer CPU percentage you see after changing to 1024 threads? Does it saturate at 4800%?
mem_throttled: true means that the storage is throttled as there is not enough memory available. You can increase the memory quota. of the indexer node.