Time Synchronization

Every night at two in the morning, a systemd timer on the backup server runs an rsync script pointed at the internal DNS server we built in Chapter 9. One morning, the Sysadmin found the backup folder filled with duplicate copies for the same date, while the previous day's archive had disappeared without a trace. After investigating, the cause turned out to be trivial: the backup server's system clock was eight hours behind the DNS server because it had never been synchronized since initial installation. As a result, the script that should have run once a day triggered multiple times and overwrote archives on the wrong date. Issues like this are usually only discovered when a needed backup is missing, right when production data is actually lost. The root cause is almost always the same: a server whose time accuracy was never maintained.

This chapter covers how to maintain time accuracy on Linux servers using chrony, starting from why this topic matters despite often being overlooked, verifying the status of chrony as the default NTP service on Ubuntu Server, configuring one internal server as a time source for other servers on the same network, to verifying synchronization results via chronyc tracking.

11.1 Why Accurate Time Matters on Servers

Every computer has a hardware clock that runs using an internal crystal oscillator. This component is never truly precise, causing the system clock to drift little by little from actual time, a phenomenon known as clock drift. On physical machines, the shift is usually small and slow. On virtual machines, however, this drift tends to be worse because virtual clocks rely on hypervisor scheduling, which can delay or pause vCPU execution without the guest operating system's knowledge. In the field, a VM recently restored from an old snapshot is one of the most common causes of system clocks suddenly drifting far off.

Without a correction mechanism, this drift accumulates continuously and has tangible impacts on server operations:

• TLS certificate validation: any browser or client application will reject HTTPS certificates whose Not Before or Not After dates make no sense compared to the system clock, even if the certificate itself is actually valid. We will cover this topic further when setting up Let's Encrypt in Chapter 17.
• Cross-server log correlation: as in the opening scenario, security incident investigations rely heavily on a consistent chronological order of events across all servers. Chapter 39 will cover centralized logging, and that effort is of little use if the timestamps from each source are out of sync.
• Time-based authentication protocols: Kerberos, one of the centralized authentication methods touched upon in Chapter 4, rejects authentication requests if the time difference between client and server exceeds a certain tolerance threshold (typically around 5 minutes) as protection against replay attacks.
• Reliability of services we built: DHCP lease durations in Chapter 10 and DNS cache Time To Live (TTL) values in Chapter 9 are both calculated based on the system clock. An incorrect clock causes both mechanisms to behave unexpectedly, such as considering a lease expired when it is not, or vice versa.
• Automated scheduling: systemd timers from Chapter 5 and scheduled backup scripts in later chapters rely entirely on an accurate system clock to run tasks on time.

The industry standard solution for this problem is NTP (Network Time Protocol), a protocol that enables a computer to align its system clock with a more accurate time source over the network. Before diving into configuration, let us check the server's current time state first.

date
timedatectl status

Pay attention to the System clock synchronized and NTP service lines in the output of timedatectl status. Both lines serve as quick indicators of the server's synchronization status before we dive deeper into chrony as the NTP implementation used by Ubuntu Server.

11.2 Chrony as the Default NTP Client in Ubuntu Server

chrony is an NTP implementation consisting of two main components: chronyd, the daemon running in the background to continuously adjust the system clock, and chronyc, the command-line interface for checking status and managing chronyd. According to official Ubuntu Server documentation, chrony is installed by default for automatic time synchronization starting from Ubuntu 25.10, and that status is maintained in Ubuntu Server 26.04 LTS, which serves as the reference for this series. This step replaces the old combination of ntpd or systemd-timesyncd that still frequently appears in older tutorials, because chrony has proven to achieve synchronization faster and is more accurate at handling networks with unstable connections. Both characteristics are critical for production servers.

11.2.1 Verifying Chrony Installation and Service Status

Since chrony is already installed by default, our first step is not installation, but verifying that its service is actually running.

Practical Steps

1. Ensure the chrony package is indeed installed.

   dpkg -l chrony

   If it turns out not to be installed, for example due to a minimal installation that removed it, install the package manually.

   sudo apt update
   sudo apt install chrony

2. Check the status of chronyd through its systemd service.

   systemctl status chrony.service

3. Reconfirm via timedatectl that the system recognizes chrony as the active NTP synchronization provider.

   timedatectl status

Verification and Troubleshooting

• Ideally, the output of timedatectl status shows System clock synchronized: yes and NTP service: active. If either still shows no or inactive, explicitly enable chrony.

  sudo systemctl enable --now chrony.service

• If timedatectl instead shows systemd-timesyncd as the active NTP provider, it indicates both services are competing for control over the system clock. Disable systemd-timesyncd to avoid conflicts with chrony.

  sudo systemctl disable --now systemd-timesyncd.service

11.2.2 Configuration Structure and Network Time Security (NTS)

The main chrony configuration resides in /etc/chrony/chrony.conf. Since recent releases, Ubuntu separates the list of NTP sources into its own directory under /etc/chrony/sources.d/, rather than writing them directly in chrony.conf. The default file is /etc/chrony/sources.d/ubuntu-ntp-pools.sources, which contains:

pool 1.ntp.ubuntu.com iburst maxsources 1 nts prefer
pool 2.ntp.ubuntu.com iburst maxsources 1 nts prefer
pool 3.ntp.ubuntu.com iburst maxsources 1 nts prefer
pool 4.ntp.ubuntu.com iburst maxsources 1 nts prefer
pool ntp-bootstrap.ubuntu.com iburst maxsources 1 nts certset 1

Notice the nts flag on each line. Since Ubuntu enabled NTS (Network Time Security) by default, all communication to the Ubuntu NTP pool is encrypted and authenticated using TLS, rather than plain UDP packets like classic NTP. This is important from a security perspective because traditional NTP is vulnerable to spoofing. An attacker on the same network could forge NTP responses to shift the target clock to an incorrect time, for instance to make an expired TLS certificate appear still valid. NTS closes this vulnerability through a key negotiation process over port 4460/TCP, while the time data exchange itself continues over port 123/UDP as with standard NTP. The prefer flag indicates that chronyd prioritizes that source when multiple sources are equally reachable.

Verification and Troubleshooting

• View the list of time sources currently used by chronyd along with their selection status.

  chronyc -N sources

  The ^* symbol in the leftmost column marks the source currently selected as the primary reference, while ^+ marks acceptable candidate sources that have not yet been selected.
• Ensure NTS negotiation succeeds by checking authentication data for each source.

  sudo chronyc -N authdata

  Successful sources show Mode: NTS with non-zero KeyID and KLen values. If the values remain zero after a few minutes, there is likely a firewall on your network blocking port 4460/TCP for NTS key negotiation.

11.3 Configuring a Server as an Internal NTP Source

While having every server query public NTP pools individually still functions, it is less than ideal for an internal network that already has multiple servers, such as the DNS server from Chapter 9 and the DHCP server from Chapter 10 that we built. Ideally, only one internal server synchronizes with public pools, and then other servers and devices on the same network synchronize with that internal server. This approach offers several clear advantages: latency to the time source is significantly lower because it resides on the local network, internet requests are drastically reduced, and equally important, the internal server can continue providing consistent time to the entire local network even during internet connection outages.

In this practice, we will configure the same server used for the internal DNS server in Chapter 9 (IP 192.168.1.10) as the internal NTP source for the 192.168.1.0/24 network, maintaining consistency with the network scheme used since Chapters 9 and 10.

11.3.1 Allowing Client Access with the allow Directive

By default, chronyd rejects all external NTP requests, even when running as a synchronized client. For this server to serve time requests to other devices, we need an allow directive that explicitly specifies which subnets are permitted.

Practical Steps

1. Edit /etc/chrony/chrony.conf on the server acting as the internal NTP source.

   sudo nano /etc/chrony/chrony.conf

2. Add the following allow directive at the end of the file.

   allow 192.168.1.0/24

3. Also add the local stratum 10 directive if we want this server to continue serving time to the local network even when its connection to the internet pool is disconnected. This directive makes chronyd "pretend" to be synchronized from the perspective of requesting clients, even if its own upstream source is unavailable.

   local stratum 10

   It should be emphasized that time shared via local mode is not truly accurate relative to the outside world once upstream remains disconnected for a prolonged period; it merely maintains time consistency among devices within the local network itself. Do not rely on this mode as a permanent replacement for a legitimate NTP source; use it only as a temporary safety net.
4. Because allow and local are access and server behavior directives (not just time source lists in sources.d/), apply the changes by performing a full restart, not just reload sources.

   sudo systemctl restart chrony.service

   This restart causes a very brief pause in chronyd's own synchronization process. The impact is usually imperceptible, but it should still be performed outside peak hours on production servers.
5. Open access for NTP ports in UFW and restrict it strictly to the relevant internal subnet so this server does not serve NTP requests from the internet arbitrarily.

   sudo ufw allow from 192.168.1.0/24 to any port 123 proto udp

Verification and Troubleshooting

• After several clients begin synchronizing with this server (see Section 11.3.2), check the list of clients that have requested time from this internal server.

  chronyc clients

• If clients still cannot access this server, the most common cause is that the subnet in the allow directive does not match the actual client subnet, or port 123/UDP is not opened in UFW as in the last step above.

11.3.2 Directing Clients to the Internal NTP Server

The internal server is now ready to serve time requests. The next step is directing other servers on the same network to synchronize with this server, rather than directly with public pools.

Practical Steps

1. On each client server, create a new time source file in the sources.d/ directory instead of editing the default ubuntu-ntp-pools.sources directly.

   sudo nano /etc/chrony/sources.d/internal-ntp.sources

2. Populate the file with a server directive pointing to the internal NTP server's IP address, along with the iburst flag for faster initial synchronization and the prefer flag to prioritize it over default public pools.

   server 192.168.1.10 iburst prefer

   We intentionally do not delete default ubuntu-ntp-pools.sources so that public pools remain available as a fallback if the internal NTP server goes down. The prefer flag is sufficient to make chronyd prioritize the internal server as long as it is reachable.
3. Apply this new time source without needing to fully restart chronyd.

   sudo chronyc reload sources

Verification and Troubleshooting

• Check whether chronyd on the client has selected the internal server as its primary source.

  chronyc -N sources

  The line with address 192.168.1.10 should be marked with ^* in the leftmost column, indicating that the server is currently actively used as the reference.
• If the internal server has not been selected despite being listed in the sources, wait a few minutes as chronyd requires several polling cycles to evaluate a source's quality before deciding to use it, even with the prefer flag.

11.4 Verifying Synchronization with chronyc tracking

After configuring both server and clients, chronyc tracking is the most important command for ensuring synchronization is actually functioning as expected, rather than just being an active systemd service with no real result.

Practical Steps

1. Run the following command on both the internal NTP server and its clients.

   chronyc -N tracking

   The output will look similar to this:

   Reference ID    : B97DBE7B (2.ntp.ubuntu.com)
   Stratum         : 3
   Ref time (UTC)  : Mon Jun 16 13:06:04 2025
   System time     : 0.000000004 seconds slow of NTP time
   Last offset     : +0.001758954 seconds
   RMS offset      : 0.017604901 seconds
   Frequency       : 3.889 ppm slow
   Residual freq   : +0.202 ppm
   Skew            : 1.458 ppm
   Root delay      : 0.022837413 seconds
   Root dispersion : 0.003050051 seconds
   Update interval : 1032.5 seconds
   Leap status     : Normal

The following lines are most commonly paid attention to during verification or troubleshooting:

• Reference ID: the identity of the time source currently in use. On our internal server, this value should point to one of the Ubuntu pools, whereas on a client, it should point to the IP address of the internal NTP server we just configured.
• Stratum: the distance of the time source from a physical reference clock (atomic clock or GPS located at stratum 0). A server connected directly to stratum 0 is called stratum 1. Clients synchronizing with our internal server are automatically one stratum higher than the server itself, e.g., stratum 4 if the internal server is at stratum 3.
• System time: the offset of the system clock relative to actual NTP time at the moment, reported in seconds. The closer to zero, the more accurate the time.
• Root delay and Root dispersion: total network delay and total accumulated uncertainty across all hops to stratum 0. Small values indicate the path to the time source is short and stable.
• Leap status: ideally shows Normal. A value of Not synchronized means chronyd has not yet successfully synchronized with any source, which is the clearest sign that something needs investigation.

Verification and Troubleshooting

• To view the quality and reachability history of each source in greater detail, use the following command:

  chronyc sourcestats

• Pay attention to the Reach column in the output of chronyc -N sources, which is printed in octal notation. A value of 377 means 8 out of 8 polls were successfully answered by that source, while significantly lower values indicate the source frequently fails to respond, due either to network issues or target server problems.
• If Leap status persistently shows Not synchronized on a client, the most efficient inspection sequence is: ensure the internal server itself is Normal first, verify the allow directive on the server includes the client subnet, then ensure port 123/UDP is unblocked on both sides.
• One important note for container environments covered in Chapters 26 and 27: Docker containers share the kernel clock with the host, so running a separate chronyd instance inside each container is unnecessary and uncommon. Simply ensure the host machine running those containers is properly synchronized.

Up to this point, we have understood why time accuracy is not just a cosmetic detail on servers, but a core foundation for TLS, authentication, logs, and services built in previous chapters. We have also practiced verifying chrony as Ubuntu Server's default NTP client, building an internal server as a time source for the local network, and interpreting chronyc tracking output to confirm everything is synchronized properly. Chapter 12 will pivot to a new topic serving as the foundation for Part IV: setting up our first web server using Nginx.