Time on Database node

Couchbase Server
1

Hi

After my servers were upgraded to version 7.0…0 (CE version) and the server was restarted, I have started receiving mails like this:

The time on node 'ns_1@db2.xxxx.dk' is not synchronized. Please ensure that NTP is set up correctly on all nodes and that clocks are synchronized.

The server is CentOS 7 running in a VM under VMware ESX. I’m not sure why this has suddenly started. The servers have been running Ok for 3-4 years. It is a small cluster with 2 servers - and I receive the messages from one of them. If I force re-sync. of time then after a short time it gets out of sync. again - and then I may receive the messages from the other server.

Is the reporting of this issue new with version 7? In other words - is my problem new - or has it been there all the time without my knowledge?

Just for reference if others search for this:

I used ntpd for keeping time - but after having read a lot over the weekend I found that Chrony seems to be the better choice with CentOS 7. I have reconfigured the servers - but still seem to have the problem.

When I use the command timedatectl I see the “no” in NTP synchronized:

      Local time: Mon 2021-08-02 08:37:20 CEST
  Universal time: Mon 2021-08-02 06:37:20 UTC
        RTC time: Mon 2021-08-02 06:39:50
       Time zone: Europe/Copenhagen (CEST, +0200)
     NTP enabled: yes
NTP synchronized: no
 RTC in local TZ: no
      DST active: yes
 Last DST change: DST began at
                  Sun 2021-03-28 01:59:59 CET
                  Sun 2021-03-28 03:00:00 CEST
 Next DST change: DST ends (the clock jumps one hour backwards) at
                  Sun 2021-10-31 02:59:59 CEST
                  Sun 2021-10-31 02:00:00 CET

If I restart Chrony using systemctl restart chronyd the timedatectl will give me (after a couple of seconds):

      Local time: Mon 2021-08-02 08:46:09 CEST
  Universal time: Mon 2021-08-02 06:46:09 UTC
        RTC time: Mon 2021-08-02 06:46:24
       Time zone: Europe/Copenhagen (CEST, +0200)
     NTP enabled: yes
NTP synchronized: yes
 RTC in local TZ: no
      DST active: yes
 Last DST change: DST began at
                  Sun 2021-03-28 01:59:59 CET
                  Sun 2021-03-28 03:00:00 CEST
 Next DST change: DST ends (the clock jumps one hour backwards) at
                  Sun 2021-10-31 02:59:59 CEST
                  Sun 2021-10-31 02:00:00 CET

But then after a couple of minutes the NTP synchronized will have changed to “no” again - and the mails start again… :wink:

2

I use NTP on my VM cluster (Ubuntu); it is always out of sync if I allow the DHCP config to override my server pool in /etc/ntpd.conf (0.pool.ntp.org,1.pool.ntp.org,2.pool.ntp.org,3.pool.ntp.org). Disabling the DHCP hook (comment out ntp_servers_setup in /etc/dhcp/dhclient-exit-hooks.d/ntp as per Prevent DHCP overriding NTP configuration - Ask Ubuntu) has worked for me.

It sounds like you need to change your chrony refresh interval; something like setting maxpoll in /etc/chrony.conf (possibly adding the same ntp.org server pool too). (Ref: Is there a way to determine how often Chrony is querying NTP servers to sync time? - Server Fault)

As for the origin of the issue… would have been there all along but via MB-25123 it is now being checked and reported in 7.0.

HTH.

3

Thanks for the response. Apparently, there is a small difference in Ubuntu and CentOS 7 as the /etc/dhcp/dhclient-exit-hooks.d/ntp file does not exist in my servers (as is actually the “best” solution in the article you refer to). So this “fix” seems to already be applied (or must be configured differently).

4

Ok, I had to change the way the server updates time using chrony with the parameter makestep 1 -1 to force it to update the time…

So when I check it on the server:

timedatectl
      Local time: Tue 2021-08-03 11:17:50 CEST
  Universal time: Tue 2021-08-03 09:17:50 UTC
        RTC time: Tue 2021-08-03 09:18:04
       Time zone: Europe/Copenhagen (CEST, +0200)
     NTP enabled: yes
NTP synchronized: yes
 RTC in local TZ: no
      DST active: yes
 Last DST change: DST began at
                  Sun 2021-03-28 01:59:59 CET
                  Sun 2021-03-28 03:00:00 CEST
 Next DST change: DST ends (the clock jumps one hour backwards) at
                  Sun 2021-10-31 02:59:59 CEST
                  Sun 2021-10-31 02:00:00 CET

I get the right info that it is NTP synchronized. However, I still get mails from the server that the time is not synchronized. I have seen a couple of times that it said “no” for a short period of time before but soon after going back to “yes”.

Do I need to restart Couchbase Server to make it aware of the change to NTP setup?

5

Ok, so it turned out that the servers in the production environment had “time-sync” with the ESXi host enabled. They should not! Disabling this made everything behave more as expected - and Couchbase has stopped spamming me with mails :grinning::+1:

Just for reference if others have similar issues. See the solution for my servers in the “Edit” part of this response: centos7 - NTP/Chrony not keeping time synchronized on CentOS 7.9 (VM running on VMware ESXi) - Server Fault

1 Like