I’ll post Eli’s private response to close the loop:
The issue was compaction was not able to finish as the total disk size was not configured to the recommended multiple (3x) of the raw data size. After increasing the free space on the nodes, compaction was able to proceed.
To do more obvious alerting on low disk space, add the following issue for us to track. https://issues.couchbase.com/browse/MB-17479
Thanks
-cihan