Having a server clock that is always correct is crucial for anybody running a server. An incorrect system date and time can cause untold problems and it seems a lot of people have encountered a bug with NTP in Ubuntu 16.04 Xenial with the service either not starting up after a reboot or simply crashing all the time. In fact I am quite sure a lot of people out there are not even aware that their NTP service is probably not starting up or is crashing and they are blindly thinking their system time is correct. So here’s a very quick walk through of how to Fix NTP on Ubuntu.
I had a very nasty surprise last week after upgrading my 2 nameservers from Ubuntu 14.04 to 16.04. During the upgrade this NTP bug surfaced and I was not aware of it. What this in turn caused was the following. When my DNSSEC on bind re-signed a bunch of domain names using DNSSEC it signed them with a date about 16 hours into the future, this resulted in a bunch of signed records in the domains being detected as BOGUS and immediately caused several web sites of mine to go down.
Not immediately knowing the cause of the issue I temporarily disabled DNSSEC on the domains until I could investigate further.
Then it dawned upon me to check NTP as I have been experiencing problems with NTP crashing on both upgrades from Ubuntu 14.0 > 16.04 and even on fresh 16.04 installs. I immediately saw what had caused the issue, NTP was not running and the system time had run a good 16 hours into the future causing a major DNSSEC failure on my part.
I eventually found a very lengthy bug report on Ubuntu regarding this issue and the fix suggested actually doesn’t work and is not the correct way to deal with ths problem. I won’t go into any length or detail of the bug but you can read about it here if you like. I will simply just provide you with the correct fix and it will take you all of a minute or two to fix.
First let me confirm 100% that ntpdate is deprecated in Ubuntu 16.04 which means it is no longer used, yet some genius at Canonical decided not only to include it with fresh new installs of Ubuntu 16.04 despite the fact that it has been replaced with something called timedatectl, but also to keep it when people upgrade instead of warning them during the upgrade process about this very mission critical thing that can totally screw up your server time as happened to me.
This correct fix below comes after the initial fix I provided last week (below) which after a few days STILL saw NTP randomly crash on my 2 nameservers causing time to jump ahead several hours due to a clock problem when ntp dies. So it is crucial to understand how this all works and what NOT to go messing with. Having a server clock go 8 hours into the future on a nameserver especially is so bad that it will kill all your DNSSEC signed domains as they will be signed at a date that does not exist and all signed records will be considered BOGUS !!!
So here’s how to get NTP correctly setup on Ubuntu 16.04 once and for all.
STEP 1: I am assuming you already have ntp and ntpdate installed. If you do not have ntp installed, then install it using
sudo apt-get install ntp
STEP 2: Now configure NTP with accurate timeservers as follows
STEP 3: Change the following section of the file to the servers I have below, leave everything else as is.
# Use servers from the NTP Pool Project. Approved by Ubuntu Technical Board # on 2011-02-08 (LP: #104525). See http://www.pool.ntp.org/join.html for # more information. server time.nist.gov burst iburst server 0.pool.ntp.org burst iburst server 1.pool.ntp.org burst iburst server 2.pool.ntp.org burst iburst server 3.pool.ntp.org burst iburst
STEP 4: Save the ntp.conf file using CTRL+X > Y > ENTER
STEP 5: Restart the NTP service
sudo service ntp restart
STEP 6: Confirm NTP is running and syncing with the time servers.
sudo ntpq -p
This will give you output similar to this. The server with the * is the one you are currently syncing with.
remote refid st t when poll reach delay offset jitter ============================================================================== *utcnist2.colora .NIST. 1 u 571 1024 217 55.804 4.462 0.077 -ntp2.torix.ca 192.168.100.251 3 u 193 1024 377 26.450 12.704 0.062 -time.srv.ualber 220.127.116.11 2 u 514 1024 377 46.636 3.115 0.122 +penguin.hopcoun 18.104.22.168 2 u 954 1024 377 16.160 -5.821 0.229 +147.ip-144-217- 22.214.171.124 2 u 596 1024 377 0.296 -3.495 1.894
STEP 7: Now make sure ntpdate is uninstalled
sudo apt-get remove ntpdate
Remove the ntpdate startup file
sudo rm /etc/network/if-up.d/ntpdate
Now enable timedatectl
sudo timedatectl set-ntp true
STEP 10: Now check the status of timedatectl and you should see the following. PLEASE NOTE: my time zone is Africa/Johannesburg +0200 so what you see below as the output from timedatectl is correct. Your Universal Time and RTC time should both be the same and then your local time zone, in my case is adjusted with NTP to be 2 hours ahead. Your actual hardware clock should be in UTC time and not synced with local time, otherwise NTP will not sync properly. Be very careful of guides telling you to run hwclock —systohc, unless your system time and entire server is running in UTC time, do not set your hardware clock with local time using hwclock —systohc.
sudo timedatectl status
You should see output as follows
Local time: Tue 2017-04-11 10:38:48 SAST Universal time: Tue 2017-04-11 08:38:48 UTC RTC time: Tue 2017-04-11 08:38:48 Time zone: Africa/Johannesburg (SAST, +0200) Network time on: yes NTP synchronized: yes RTC in local TZ: no
STEP 11: Reboot your server
STEP 12: After reboot confirm the NTP service is running
sudo ntpq -p
and you will once again see output like this
remote refid st t when poll reach delay offset jitter ============================================================================== *nist1-lnk.binar .ACTS. 1 u 872 1024 377 32.028 4.001 0.026 +126.96.36.199 188.8.131.52 2 u 481 1024 377 23.152 -11.972 0.237 -ntp1.torix.ca 192.168.100.252 2 u 1026 1024 377 26.160 -7.960 0.032 +ntp2.torix.ca 192.168.100.251 3 u 798 1024 377 19.403 -0.363 0.029 -ntp3.torix.ca 192.168.100.251 3 u 316 1024 377 14.377 9.362 4.976
STEP 13: Run the following command again to make sure timedatectl is showing the correct information.
sudo timedatectl status
Local time: Tue 2017-04-11 10:46:59 SAST Universal time: Tue 2017-04-11 08:46:59 UTC RTC time: Tue 2017-04-11 08:46:59 Time zone: Africa/Johannesburg (SAST, +0200) Network time on: yes NTP synchronized: yes RTC in local TZ: no
STEP 14: If everything checks out we want one final step now to make sure NTP is monitored and if it ever happens to fail for some reason it will get restarted and you will get notified by email about it. For this we are going to use monit to monitor our NTP service.
sudo apt-get install monit
after installing monit configure the basic monit configuration file as follows.
sudo nano /etc/monit/monitrc
If you are only using monit for monitoring NTP you can replace all the contents of the file with just this below, put your correct email address in to make sure you receive notifications from Monit.
######################################################################### ## Monit control file ############################################################################### set daemon 60 # check services at 2-minute intervals set logfile /var/log/monit.log # set pidfile /var/run/monit.pid set idfile /var/lib/monit/id set statefile /var/lib/monit/state set eventqueue basedir /var/lib/monit/events # set the base directory where events will be stored slots 100 # optionally limit the queue size set alert email@example.com # receive all alerts set mailserver localhost # include /etc/monit/conf.d/* include /etc/monit/conf-enabled/*
Now create the ntp monitoring file for monit
sudo nano /etc/monit/conf-enabled/ntp
paste the contents below into that file
check process ntpd with pidfile "/var/run/ntpd.pid" start program = "/etc/init.d/ntp start" stop program = "/etc/init.d/ntp stop" if 3 restarts within 3 cycles then alert
Now restart monit
sudo service monit restart
Now check monit is running correctly
systemctl status monit
and check your email for startup alerts from monit to make sure Monit is emailing out correctly.
STEP 15: Reboot you server again
STEP 16: After reboot make sure NTP, monit and timedatectl are all correct. Run all of the following commands
sudo systemctl status ntp
sudo systemctl status monit
sudo timedatectl status
sudo ntpq -p
STEP 17: That’s it, you now have NTP correctly configured on Ubuntu 16.04. But if you are a little paranoid like I am I have written a little shell script running with CRON below which, for now, emails me every 6 hours to confirm everything is 100% in tune and synced correctly. It’s just my little extra step of caution over the next few weeks and once I am happy I will disable this cronjob and just leave Monit to take over.
sudo nano /usr/sbin/timecheck.sh
paste the contents below and change to your valid email address and adjust the subject line “Time Check Server 1” of the email to suit you.
#!/bin/bash # First clear the contents of our timecheck log file sudo truncate -s 0 /var/log/timecheck.log # Run all our time check commands date >> /var/log/timecheck.log printf "\n" >> /var/log/timecheck.log ntpq -p >> /var/log/timecheck.log printf "\n" >> /var/log/timecheck.log sudo hwclock --debug >> /var/log/timecheck.log printf "\n" >> /var/log/timecheck.log sudo timedatectl status >> /var/log/timecheck.log # Send an email report mail -s "Time Check Server 1" firstname.lastname@example.org < /var/log/timecheck.log exit 0
Make the shell script executable
sudo chmod +x /usr/sbin/timecheck.sh
Create the empty logfile
sudo touch /var/log/timecheck.log
Make the logfile writeable
sudo chmod 755 /var/log/timecheck.log
Run the script once manually.
Check that you receive the email and then add it to cron to run every 6 hours.
sudo crontab -e
and add the following cron job
0 */6 * * * /usr/sbin/timecheck.sh
One you are happy after a few days or weeks, you can simply remove the cronjob or disable it with a # at the beginning
If you want to keep running it forever but only have one alert per day, let’s say in the morning only, then your cron could be setup like this
30 08 * * * /usr/sbin/timecheck.sh
Then this will only do the timecheck every morning at 8:30 am.
UPDATE: The original fix I suggested below is actually NOT the correct solution, I have left it here however to once again show how one can be lead astray by people suggesting stupid solutions on Ubuntu bug reports that actually do NOT fix the problem because those suggesting the solutions don’t understand the actual problem. The correct fix is above this message.
Simply edit the following file.
sudo nano /etc/network/if-up.d/ntpdate at line number 2 of the file, just below “#!/bin/sh” add the “exit 0” so it looks like this.
#!/bin/sh exit 0 set -e Save the file CRTL+X+Y+ENTER Make sure the file is executable
sudo chmod 755 /etc/network/if-up.d/ntpdate and reboot your system Once rebooted, simply run the following terminal command and you will see that ntp is now running on startup.
sudo ntpq -p
That’s IT !!! As for the bug, the Ubuntu people really need to sort this out and Fix NTP on Ubuntu as it involves the deprecated ntpdate package which should actually be removed altogether and this is causing a lot of problems for people, most who seem totally unaware of this issue. As I said it does not only happen on upgrades from Ubuntu 14.04 > 16.04, the same bug exists on a fresh installation of Ubuntu 16.04.