Fix NTP on Ubuntu
How to Fix NTP on Ubuntu

Having a server clock that is always correct is crucial for anybody running a server. An incorrect system date and time can cause untold problems and it seems a lot of people have encountered a bug with NTP in Ubuntu 16.04 Xenial with the service either not starting up after a reboot or simply crashing all the time. In fact I am quite sure a lot of people out there are not even aware that their NTP service is probably not starting up or is crashing and they are blindly thinking their system time is correct. So here’s a very quick walk through of how to Fix NTP on Ubuntu.

I had a very nasty surprise last week after upgrading my 2 nameservers from Ubuntu 14.04 to 16.04. During the upgrade this NTP bug surfaced and I was not aware of it. What this in turn caused was the following. When my DNSSEC on bind re-signed a bunch of domain names using DNSSEC it signed them with a date about 16 hours into the future, this resulted in a bunch of signed records in the domains being detected as BOGUS and immediately caused several web sites of mine to go down.

Not immediately knowing the cause of the issue I temporarily disabled DNSSEC on the domains until I could investigate further.

Then it dawned upon me to check NTP as I have been experiencing problems with NTP crashing on both upgrades from Ubuntu 14.0 > 16.04 and even on fresh 16.04 installs. I immediately saw what had caused the issue, NTP was not running and the system time had run a good 16 hours into the future causing a major DNSSEC failure on my part.

I eventually found a very lengthy bug report on Ubuntu regarding this issue and the fix suggested actually doesn’t work and is not the correct way to deal with ths problem. I won’t go into any length or detail of the bug but you can read about it here if you like. I will simply just provide you with the correct fix and it will take you all of a minute or two to fix.

First let me confirm 100% that ntpdate is deprecated in Ubuntu 16.04 which means it is no longer used, yet some genius at Canonical decided not only to include it with fresh new installs of Ubuntu 16.04 despite the fact that it has been replaced with something called timedatectl, but also to keep it when people upgrade instead of warning them during the upgrade process about this very mission critical thing that can totally screw up your server time as happened to me.

This correct fix below comes after the initial fix I provided last week (below) which after a few days STILL saw NTP randomly crash on my 2 nameservers causing time to jump ahead several hours due to a clock problem when ntp dies. So it is crucial to understand how this all works and what NOT to go messing with. Having a server clock go 8 hours into the future on a nameserver especially is so bad that it will kill all your DNSSEC signed domains as they will be signed at a date that does not exist and all signed records will be considered BOGUS !!!

So here’s how to get NTP correctly setup on Ubuntu 16.04 once and for all.

STEP 1: I am assuming you already have ntp and ntpdate installed. If you do not have ntp installed, then install it using

sudo apt-get install ntp

STEP 2: Now configure NTP with accurate timeservers as follows

nano /etc/ntp.conf

STEP 3: Change the following section of the file to the servers I have below, leave everything else as is.

# Use servers from the NTP Pool Project. Approved by Ubuntu Technical Board
# on 2011-02-08 (LP: #104525). See http://www.pool.ntp.org/join.html for
# more information.
server time.nist.gov burst iburst
server 0.pool.ntp.org burst iburst
server 1.pool.ntp.org burst iburst
server 2.pool.ntp.org burst iburst
server 3.pool.ntp.org burst iburst

STEP 4: Save the ntp.conf file using CTRL+X > Y > ENTER

STEP 5: Restart the NTP service

sudo service ntp restart

STEP 6: Confirm NTP is running and syncing with the time servers.

sudo ntpq -p

This will give you output similar to this. The server with the * is the one you are currently syncing with.

remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*utcnist2.colora .NIST.           1 u  571 1024  217   55.804    4.462   0.077
-ntp2.torix.ca   192.168.100.251  3 u  193 1024  377   26.450   12.704   0.062
-time.srv.ualber 129.128.153.62   2 u  514 1024  377   46.636    3.115   0.122
+penguin.hopcoun 142.66.101.13    2 u  954 1024  377   16.160   -5.821   0.229
+147.ip-144-217- 206.248.144.162  2 u  596 1024  377    0.296   -3.495   1.894

STEP 7: Now make sure ntpdate is uninstalled

sudo apt-get remove ntpdate

STEP 8:

Remove the ntpdate startup file

sudo rm /etc/network/if-up.d/ntpdate

STEP 9:

Now enable timedatectl

sudo timedatectl set-ntp true

STEP 10: Now check the status of timedatectl and you should see the following. PLEASE NOTE: my time zone is Africa/Johannesburg +0200 so what you see below as the output from timedatectl is correct. Your Universal Time and RTC time should both be the same and then your local time zone, in my case is adjusted with NTP to be 2 hours ahead. Your actual hardware clock should be in UTC time and not synced with local time, otherwise NTP will not sync properly. Be very careful of guides telling you to run hwclock —systohc, unless your system time and entire server is running in UTC time, do not set your hardware clock with local time using hwclock —systohc.

sudo timedatectl status

You should see output as follows

Local time: Tue 2017-04-11 10:38:48 SAST
Universal time: Tue 2017-04-11 08:38:48 UTC
RTC time: Tue 2017-04-11 08:38:48
Time zone: Africa/Johannesburg (SAST, +0200)
Network time on: yes
NTP synchronized: yes
RTC in local TZ: no

STEP 11: Reboot your server

sudo reboot

STEP 12: After reboot confirm the NTP service is running

sudo ntpq -p

and you will once again see output like this

remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*nist1-lnk.binar .ACTS.           1 u  872 1024  377   32.028    4.001   0.026
+208.80.96.70    142.3.100.2      2 u  481 1024  377   23.152  -11.972   0.237
-ntp1.torix.ca   192.168.100.252  2 u 1026 1024  377   26.160   -7.960   0.032
+ntp2.torix.ca   192.168.100.251  3 u  798 1024  377   19.403   -0.363   0.029
-ntp3.torix.ca   192.168.100.251  3 u  316 1024  377   14.377    9.362   4.976

STEP 13: Run the following command again to make sure timedatectl is showing the correct information.

sudo timedatectl status
Local time: Tue 2017-04-11 10:46:59 SAST
Universal time: Tue 2017-04-11 08:46:59 UTC
RTC time: Tue 2017-04-11 08:46:59
Time zone: Africa/Johannesburg (SAST, +0200)
Network time on: yes
NTP synchronized: yes
RTC in local TZ: no

STEP 14: If everything checks out we want one final step now to make sure NTP is monitored and if it ever happens to fail for some reason it will get restarted and you will get notified by email about it. For this we are going to use monit to monitor our NTP service.

sudo apt-get install monit

after installing monit configure the basic monit configuration file as follows.

sudo nano /etc/monit/monitrc

If you are only using monit for monitoring NTP you can replace all the contents of the file with just this below, put your correct email address in to make sure you receive notifications from Monit.

#########################################################################
## Monit control file
###############################################################################
set daemon 60            # check services at 2-minute intervals
set logfile /var/log/monit.log
# set pidfile /var/run/monit.pid
set idfile /var/lib/monit/id
set statefile /var/lib/monit/state
set eventqueue
basedir /var/lib/monit/events # set the base directory where events will be stored
slots 100                     # optionally limit the queue size
set alert yourname@youremail.com                       # receive all alerts
set mailserver localhost
#   include /etc/monit/conf.d/*
include /etc/monit/conf-enabled/*

Now create the ntp monitoring file for monit

sudo nano /etc/monit/conf-enabled/ntp

paste the contents below into that file

check process ntpd
with pidfile "/var/run/ntpd.pid"
start program = "/etc/init.d/ntp start"
stop program = "/etc/init.d/ntp stop"
if 3 restarts within 3 cycles then alert

Now restart monit

sudo service monit restart

Now check monit is running correctly

systemctl status monit

and check your email for startup alerts from monit to make sure Monit is emailing out correctly.

STEP 15: Reboot you server again

sudo reboot

STEP 16: After reboot make sure NTP, monit and timedatectl are all correct. Run all of the following commands

sudo systemctl status ntp
sudo systemctl status monit
sudo timedatectl status
sudo ntpq -p

STEP 17: That’s it, you now have NTP correctly configured on Ubuntu 16.04. But if you are a little paranoid like I am I have written a little shell script running with CRON below which, for now, emails me every 6 hours to confirm everything is 100% in tune and synced correctly. It’s just my little extra step of caution over the next few weeks and once I am happy I will disable this cronjob and just leave Monit to take over.

sudo nano /usr/sbin/timecheck.sh

paste the contents below and change to your valid email address and adjust the subject line “Time Check Server 1” of the email to suit you.

#!/bin/bash
# First clear the contents of our timecheck log file
sudo truncate -s 0 /var/log/timecheck.log
# Run all our time check commands
date >> /var/log/timecheck.log
printf "\n" >> /var/log/timecheck.log
ntpq -p >> /var/log/timecheck.log
printf "\n" >> /var/log/timecheck.log
sudo hwclock --debug >> /var/log/timecheck.log
printf "\n" >> /var/log/timecheck.log
sudo timedatectl status >> /var/log/timecheck.log
# Send an email report
mail -s "Time Check Server 1" yourname@youremail.com < /var/log/timecheck.log
exit 0

Make the shell script executable

sudo chmod +x /usr/sbin/timecheck.sh

Create the empty logfile

sudo touch /var/log/timecheck.log

Make the logfile writeable

sudo chmod 755 /var/log/timecheck.log

Run the script once manually.

sudo /usr/sbin/timecheck.sh

Check that you receive the email and then add it to cron to run every 6 hours.

sudo crontab -e

and add the following cron job

0 */6 * * * /usr/sbin/timecheck.sh

One you are happy after a few days or weeks, you can simply remove the cronjob or disable it with a # at the beginning

If you want to keep running it forever but only have one alert per day, let’s say in the morning only, then your cron could be setup like this

30 08 * * * /usr/sbin/timecheck.sh

Then this will only do the timecheck every morning at 8:30 am.

 

UPDATE: The original fix I suggested below is actually NOT the correct solution, I have left it here however to once again show how one can be lead astray by people suggesting stupid solutions on Ubuntu bug reports that actually do NOT fix the problem because those suggesting the solutions don’t understand the actual problem. The correct fix is above this message.

Simply edit the following file.

sudo nano /etc/network/if-up.d/ntpdate

at line number 2 of the file, just below “#!/bin/sh” add the “exit 0” so it looks like this.

#!/bin/sh
 exit 0
 set -e

Save the file CRTL+X+Y+ENTER

Make sure the file is executable

sudo chmod 755 /etc/network/if-up.d/ntpdate

and reboot your system

Once rebooted, simply run the following terminal command and you will see that ntp is now running on startup.

sudo ntpq -p

That’s IT !!! As for the bug, the Ubuntu people really need to sort this out and Fix NTP on Ubuntu as it involves the deprecated ntpdate package which should actually be removed altogether and this is causing a lot of problems for people, most who seem totally unaware of this issue. As I said it does not only happen on upgrades from Ubuntu 14.04 > 16.04, the same bug exists on a fresh installation of Ubuntu 16.04.

3 thoughts on “Fix NTP on Ubuntu 16.04 Not Starting and Crashing (ntpdate deprecated)

  1. Jeff G says:

    It’s actually worse than you think.

    I’m thinking that systemd-timesyncd is supposed to completely replace ntp.

    Try removing the ntp package completely and rely solely on timesyncd and see if that keeps time properly.

    I’m getting the feeling that ntp in addition to ntpdate are deprecated. ntpq command isn’t needed as a result.

    My $0.02

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.