The NTP Pool is a volunteer organization that provides time synchronization service to hundreds of millions of computers worldwide. A typical client might query a particular NTP Pool server ~10-60 times/hour. Wikipedia lists some abusive clients that far exceeded the normal rate. This wastes NTP server resources, may interfere with other clients, and can trigger DDoS protections. In late 2019, a software update made some FortiGate firewalls very unfriendly to the NTP Pool.
Fortinet is a multinational corporation that produces computer security devices, including the FortiGate firewall. The firewall product documentation describes how to use the NTP Pool for time synchronization, e.g., by using:
0.europe.pool.ntp.org 1.europe.pool.ntp.org 2.europe.pool.ntp.org
DNS will resolve 0.europe.pool.ntp.org into one of the hundreds of NTP servers in the NTP Pool. Let’s say that at firewall startup 1.2.3.4 was the NTP server selected by DNS. What happens if sometime later the administrator of 1.2.3.4 decides to stop participating in the NTP Pool and shuts down the public NTP service? We were told that older FortiGate NTP software did not handle that condition gracefully. A software update, FortiOS 6.2.3, was issued in December 2019 that periodically reissued the DNS lookup. In our example, if 0.europe.pool.ntp.org no longer resolved into 1.2.3.4 some re-initialization was done by the new software. That new code had a serious flaw. Bursts of NTP requests with rates that sometimes exceeded 20,000/sec were sent for up to 10 seconds duration. As DNS resolution changed, which is common for the NTP pools, a single FortiGate firewall would send bursts to different NTP pool servers.
A burst from a single FortiGate firewall is shown below
In early 2020, NTP pool operators noticed these distinctive bursts. They sometimes represented over 20 % of the total NTP traffic on a single server! Diagnostic work by several NTP Pool volunteers identified FortiGate firewalls as the source of these bursts. In April 2020, we contacted Fortinet support who acknowledged the bug and said that the next software release containing a fix would soon be deployed. However, the new software had unrelated bugs that prevented it from being widely deployed. On August 22, 2020, FortiOS 6.2.5 was released to eliminate the bursts.
The fix for FortiGate administrators is simple:
- Do not use FortiOS release 6.2.3, or
- if using release 6.2.3, do not use the NTP Pool.
FortiGate notified a handful of customers of the problem. However, as of this writing (November 2020), the NTP Pool is still receiving frequent NTP bursts from hundreds (at least) of FortiGate firewalls. While Fortinet was responsive in email and phone discussions, on the whole, their corrective actions were disappointing. Hopefully, FortiGate administrators will read this note and take corrective action as appropriate.
The diagnostic work was done by Miroslav Lichvar, Hal Murray, and Steven Sommars with contributions from several other NTP Pool volunteers.
Photo by Heather Zabriskie on Unsplash.