Hi,
Just a little bit about our setup
We are running CB cluster on AWS EC2 Graviton2 (ARM arch) with version 7.1.0 build 2556. There are 6 nodes in cluster:
- 3x indexer
- 2x data+query
- 1x data+query+backup
We have setup a backup using S3 as an archive dir and using staging directory on our backup node. On every Sunday morning, we create new repo (archive previous one) and create hourly backup during the week…and this hourly backups are making my hairs grey
Every backup is taking longer to complete and in the final days (Friday, Saturday…before making new repo), it can take more then an hour in which case a backup scheduled is skipped.
This is a list of the last 10 backups… the one marked is the backup from log file attached…
If I look at the S3 it looks like folder on S3 was created few minutes before backup was completed…
Nodes are running on m6gd.xlarge instances and we have S3 gateway set up in VPC in which instances are running.
CBLogs.zip (3.5 KB)
Can someone please help us understand this? At least a hint, what is going on that 47+ min, before directory on S3 is create?
I can’t find a reason why backups are taking so long to complete.
I know we can create a new repo during the week as a work-around but I would like to at least understand before doing that…
Just FYI…we also have another cluster (same version, hourly backups using S3) also with 6 nodes with same services set up but it is running on x86_64 arch… backups are completed in few minutes all week around…