One possibility is it’s related to the buffers where we build operations. We cache and reuse these buffers (up to 32 of them) to reduce allocations on the LOH. However, to ensure we don’t use too much RAM we don’t reuse buffers greater than 1MB in size. We dispose of those. My original thought was that we might make these thresholds tunable via configuration if there was a need, but it isn’t currently tunable.
Since you’re doing approx 1MB operations, some of them may be exceeding that threshold and causing the LOH to thrash. I can’t say for sure, it’s just a loose theory, but maybe worth checking. Unfortunately, there isn’t any logging around disposing of these buffers because they are global and outside the scope of logging.
Possible tests:
- Make all operations less than 1MB (not exactly 1MB, we need 30-ish bytes for headers)
- Adjust the thresholds in the linked file above and build from source
- Use a memory tracing tool like JetBrains dotMemory to see how much this is happening and if garbage collection might be the culprit.