Compaction impacting performance on high load
We have been load testing CB 2.0 for a while now. And the following scenario kept erroring out:
+ CB 2.0 Cluster of 5 nodes.
+ Each node has 6.69 GB allocated to CB out of the 8GB of total VM memory.
The test bed was generating 500 concurrent users (read threads) from 6 client hosts firing get:put in 1:1 ratio.
The test would run for about 5 min, and then on the console, we can see compaction kicking in. And boom. Entire cluster would come crashing down. Sometimes 2 nodes would fail, sometimes 1. The web (admin) console would hang/become unresponsive.
This is reproducible quite easily.
After trying various combinations (as we thought there was a problem with our VMs or network initially), we finally landed at compaction being the culprit.
Right now the tests are running without any issue with compaction turned off.
Any thoughts/comments from others or couchbase team?