Does anyone know what Couchbase server does every 1 hr 45 mins (i.e. 105 minutes)?
Looking at our monit graphs, our couchbase box’s load average shoots up exactly every 1 hr 45 mins. For example: 07:30, 09:15, 11:00, 12:45, 14:30…
Normally the load average floats around 0.25-0.50 on the box. At that point every 105 minutes, it shoots up to 0.75-1.00.
Couchbase has a few background tasks which run periodically, however nothing which by default is every 105 minutes. The main tasks are:
The Expiry Pager (Couchbase Admin Guide) which actually delete any items which have expired from disk. It runs by default every 60 minutes but can be adjusted.
Compaction (Database and View Compaction is performed to compact the on-disk files, essentially removing deleted / old versions of documents. By default this runs when fragmentation of data / views reaches a given threashold and so the exact time this is run will vary depending on your workload.
In your situation I would guess that your disk/index fragmentation is hitting 30% (the default threshold to start compaction) every 105 minutes, and this is contributing to the load average you see. This is a normal, automatic process and so shouldn’t be a concern. There’s more detail on compaction in the link above.