OK. CB does pool connections on a bucket level (hence the multi-bucket question). Obviously this means they could have been established quite some time before the errors - which gives ample opportunity for a hiccup to affect them.
You might also try “SELECT * FROM system:indexes” and see if that helps - should refresh the connection pool, which if it works, would let us know that something external (i.e. n/w layer) is affecting the existing (but likely idle) connections. (N/W idle timeout settings perhaps?)
Good info and suggestions. I have a suspicion that it is an idle timeout thing - and thus I don’t expect to see the problem until tomorrow morning.
I have written to the data center admin - and when it reoccurs I’ll try the select on indexes.
Thanks!
On my hunt for explanations I fell over this in the messages log:
Jul 20 02:20:40 AQUA-pFangstDB1 chronyd[712]: Can't synchronise: no majority
Jul 20 02:28:40 AQUA-pFangstDB1 chronyd[712]: Selected source 130.226.56.69
Jul 20 02:28:40 AQUA-pFangstDB1 chronyd[712]: System clock wrong by 236.291160 seconds, adjustment started
Could this kind of clock offset be the root cause??
I know time sync. has been set up - but the message above could indicate that something is not working correctly. The times I have tested it (through date run on two terminals at a time) the time on the two database servers in the cluster have been within the second - so probably Ok.
I don’t think they would. I can’t see time-sync affecting establishment or use of TCP connections; perhaps some programme functions but those wouldn’t be the errors you’re seeing. (I can’t really even see that impacting firewall/firewall management, if that’s applicable.)
Ahh… there was an explanation. They “pause” the VM at night to allow them to take a snapshot. Most likely that is what is stalling the communications (and causing the time lack).
They are on it - and we’ll see if it is fixed in the morning.
Thanks for the discussion and insight!
Ah - that would do it.
No problem.