IKEv2 IPsec VPN Tunnel Palo Alto FortiGate
Using a FortiGate for Bitcoin Mining
File Blocking Shootout – Palo Alto vs. Fortinet
FortiGate Out-of-Band Management
Basic MP-BGP Lab: Cisco Router, Palo Alto, Fortinet
Trying to change an IPv6 Link-Local Address on a FortiGate
I got an email where someone asked whether I know how to change the link-local IPv6 addresses on a FortiGate similar to any other network/firewall devices. He could not find anything about this on the Fortinet documentation nor on Google.
Well, I could not find anything either. What’s up? It’s not new to me that you cannot really configure IPv6 on the FortiGate GUI, but even on the CLI I couldn’t find anything about changing this link-local IPv6 address from the default EUI-64 based one to a manually assigned one. Hence I opened a ticket at Fortinet. It turned out that you cannot *change* this address at all, but that you must *add* another LL address which will be used for the router advertisements (RA) after a reboot (!) of the firewall. Stupid design!
Again and again and again I am not happy at all with the IPv6 implementation on the FortiGates. Too many bugs and features missing, while everything is too complicated to configure. (Have a look at my Fortinet feature requests.) For the following tests I used a FortiGate FG-90D with firmware v5.6.5 build1600 (GA).
Before (Default Behaviour)
Before I touched the config the state of IPv6 was the following. Have a look at the “fg-trust” interface with its link-local address in line 12:
fg # diagnose ipv6 address list dev=31 devname=vsys_fgfm flag=P scope=254 prefix=128 addr=::1 dev=29 devname=vsys_ha flag=P scope=254 prefix=128 addr=::1 dev=28 devname=fg-server flag=P scope=0 prefix=64 addr=2003:de:2016:220::1 dev=27 devname=fg-trust2 flag=P scope=0 prefix=64 addr=2003:de:2016:211::1 dev=26 devname=fg-trust flag=P scope=0 prefix=64 addr=2003:de:2016:210::1 dev=24 devname=root flag=P scope=254 prefix=128 addr=::1 dev=5 devname=wan1 flag=P scope=0 prefix=64 addr=2003:de:2016::2 dev=6 devname=wan2 flag=P scope=253 prefix=10 addr=fe80::a5b:eff:fea1:8360 dev=28 devname=fg-server flag=P scope=253 prefix=10 addr=fe80::a5b:eff:fea1:835e dev=27 devname=fg-trust2 flag=P scope=253 prefix=10 addr=fe80::a5b:eff:fea1:835e dev=26 devname=fg-trust flag=P scope=253 prefix=10 addr=fe80::a5b:eff:fea1:835e dev=5 devname=wan1 flag=P scope=253 prefix=10 addr=fe80::a5b:eff:fea1:835f
The configuration at this point was:
config system interface edit "fg-trust" set vdom "root" set ip 192.168.210.1 255.255.255.0 set allowaccess ping https ssh set role lan set snmp-index 5 config ipv6 set ip6-address 2003:de:2016:210::1/64 set ip6-allowaccess ping https ssh set ip6-send-adv enable config ip6-prefix-list edit 2003:de:2016:210::/64 set autonomous-flag enable set onlink-flag enable next end end set interface "internal1" set vlanid 210 next end
And a Linux machine got the following routing table, in which the default route had a gateway of
fe80::a5b:eff:fea1:835e:
weberjoh@jw-vm05-Ubuntu-Test-3:~$ ip -6 r s 2003:de:2016:210::/64 dev ens32 proto kernel metric 256 expires 2591699sec pref medium fe80::/64 dev ens32 proto kernel metric 256 pref medium default via fe80::a5b:eff:fea1:835e dev ens32 proto ra metric 1024 expires 1499sec pref medium
Configuration of the Link-Local Address
To add a link-local address you need the “config ip6-extra-addr” submenu. I added the quite simple
fe80::1/64address to that interface, that is:
config system interface edit fg-trust config ipv6 config ip6-extra-addr edit fe80::1/64 next end end end
Now, in order to have the router advertisements sent from this newly created link-local address, you have to reboot the firewall! Come on Fortinet, you need a complete reboot for this?!? (Note that the support ticket told me to disable the “ip6-send-adv” before adding the LL address, and enabling it again after that. But this was not successful. At this point the RAs were still sent from the old EUI-64 based LL address.) Hence a reboot:
execute reboot
After
After this changes and the reboot the added link-local IPv6 was present (line 6):
fg # diagnose ipv6 address list dev=31 devname=vsys_fgfm flag=P scope=254 prefix=128 addr=::1 dev=29 devname=vsys_ha flag=P scope=254 prefix=128 addr=::1 dev=28 devname=fg-server flag=P scope=0 prefix=64 addr=2003:de:2016:220::1 dev=27 devname=fg-trust2 flag=P scope=0 prefix=64 addr=2003:de:2016:211::1 dev=26 devname=fg-trust flag=SP scope=253 prefix=64 addr=fe80::1 dev=26 devname=fg-trust flag=P scope=0 prefix=64 addr=2003:de:2016:210::1 dev=24 devname=root flag=P scope=254 prefix=128 addr=::1 dev=5 devname=wan1 flag=P scope=0 prefix=64 addr=2003:de:2016::2 dev=6 devname=wan2 flag=P scope=253 prefix=10 addr=fe80::a5b:eff:fea1:8360 dev=28 devname=fg-server flag=P scope=253 prefix=10 addr=fe80::a5b:eff:fea1:835e dev=27 devname=fg-trust2 flag=P scope=253 prefix=10 addr=fe80::a5b:eff:fea1:835e dev=26 devname=fg-trust flag=P scope=253 prefix=10 addr=fe80::a5b:eff:fea1:835e dev=5 devname=wan1 flag=P scope=253 prefix=10 addr=fe80::a5b:eff:fea1:835f
The complete configuration section for this interface looked like this:
config system interface edit "fg-trust" set vdom "root" set ip 192.168.210.1 255.255.255.0 set allowaccess ping https ssh set role lan set snmp-index 5 config ipv6 set ip6-address 2003:de:2016:210::1/64 set ip6-allowaccess ping https ssh config ip6-extra-addr edit fe80::1/64 next end set ip6-send-adv enable config ip6-prefix-list edit 2003:de:2016:210::/64 set autonomous-flag enable set onlink-flag enable next end end set interface "internal1" set vlanid 210 next end
And the Linux machine (after a reboot as well) got the correct next hop for its default route:
weberjoh@jw-vm05-Ubuntu-Test-3:~$ ip -6 r s 2003:de:2016:210::/64 dev ens32 proto kernel metric 256 expires 2591853sec pref medium fe80::/64 dev ens32 proto kernel metric 256 pref medium default via fe80::1 dev ens32 proto ra metric 1024 expires 1653sec pref medium
Accordingly I could verify that the router advertisements were sent from my added link-local address
fe80::1:
Competitors
That’s it. I am not happy with this approach from Fortinet in “changing” the link-local address. On other firewalls such as the Palo Alto Networks firewall you can clearly change the behaviour of the interface ID portion, and it even works without rebooting the firewall:
Cheers.
Featured image “Buy Local” by Mariano Mantel is licensed under CC BY-NC 2.0.
Fortinet FortiGate (not) using NTP Authentication
A security device such as a firewall should rely on NTP authentication to overcome NTP spoofing attacks. Therefore I am using NTP authentication on the FortiGate as well. As always, this so-called next-generation firewall has a very limited GUI while you need to configure all details through the CLI. I hate it, but that’s the way Fortinet is doing it. Furthermore the “set authentication” command is hidden unless you’re downgrading to NTPv3 (?!?) and it only supports MD5 rather than SHA-1. Not that “next-generation”!
Finally, you have no chance of knowing whether NTP authentication is working or not. I intentionally misconfigured some of my NTP keys which didn’t change anything in the NTP synchronization process while it should not work at all. Fail!
I am using a FortiGate FG-100D with FortiOS version v5.6.6 build1630 (GA). If you want to configure custom NTP servers you have to go through the CLI at all:
Then, configuring on the CLI, it took me quite some time to realize that the NTP authentication commands are completely hidden unless you are using NTPv3. Don’t know why this is a requirement at all since NTP authentication, of course, works with NTPv4 as well. And why isn’t this documented?
However, here are the commands I used to set up my three NTP servers with authentication:
config system ntp set ntpsync enable set type custom config ntpserver edit 1 set server "ntp1.weberlab.de" set ntpv3 enable set authentication enable set key ENC 3xZj6FcN+Hg0ltR3BIQevJR3G+umyFrzN4mXeRRoxlTXM9HwKMMb1wo/t3AscNHjuuVkC58OTXP30U6rPce7RvGXfVfBA81s92JQ9duTKZv3be+N4KPiOM8EbTxYFN9irk/Kf8VuNDVZITsVGW+m6qaJewHycIk4wRypuHbA4s2/6GtL4ryYXHvksoB9bckwqOCqAw== set key-id 1 next edit 2 set server "ntp2.weberlab.de" set ntpv3 enable set authentication enable set key ENC wdqOtz4Q6HAe+RSzpGpx0nqZmRImT2gH3nwGStdDJn93EOLNv+kP5fxxjazyT+ArjRVWZVFYZnT/8fFqujwWP2GhyyALS4FdYPExaKTFAe/9m6DpIzTod1k8m8LbAJT0PnOG+8O3CgqLnhpnHm8v8Cp2oly/iORJ/ajVPQzvuvCuDzHX1fDQxsO4fJhFOVKlMgn/RQ== set key-id 2 next edit 3 set server "ntp3.weberlab.de" set ntpv3 enable set authentication enable set key ENC 0XXZMf6zshlsRxbElifoqXJXRxuM4Pti92wIYHq3pKKjvsHLuGPYx3wpqhylITZcabVS49X6EE6JwmHS22BTrCJLTVoO8TAvKaq/ZXHsawBLLme7WO7VQA5SumIx88q9VCj7Bd9aYKoevn4oBl5VRomY3I78DvoQ015nK8J+zReuWXWGL5LgL9qo3mM7j0YJTTGsgw== set key-id 3 next end end
In order to view any live values the
get system ntpis not quite helpful. At least you can see the sync interval:
fg # get system ntp ntpsync : enable type : custom syncinterval : 1 ntpserver: == [ 1 ] id: 1 == [ 2 ] id: 2 == [ 3 ] id: 3 source-ip : 0.0.0.0 server-mode : disable
diagnose sys ntp statushelps a bit more:
fg # diagnose sys ntp status synchronized: yes, ntpsync: enabled, server-mode: disabled ipv6 server(ntp3.weberlab.de) 2003:de:2016:330::dcfb:123 -- reachable(0xff) S:1 T:8 selected server-version=3, stratum=1 reference time is e03fd1c3.96a0308 -- UTC Fri Mar 22 21:27:31 2019 clock offset is 0.019739 sec, root delay is 0.000000 sec root dispersion is 0.000153 sec, peer dispersion is 623 msec ipv6 server(ntp2.weberlab.de) 2003:de:2016:330::6b5:123 -- reachable(0xff) S:0 T:7 server-version=3, stratum=1 reference time is e03fd1bb.d7cf8fae -- UTC Fri Mar 22 21:27:23 2019 clock offset is 0.015482 sec, root delay is 0.000000 sec root dispersion is 0.001114 sec, peer dispersion is 504 msec ipv6 server(ntp1.weberlab.de) 2003:de:2016:336::dcf7:123 -- reachable(0xff) S:0 T:7 server-version=3, stratum=1 reference time is e03fd18c.e184d3e8 -- UTC Fri Mar 22 21:26:36 2019 clock offset is -0.023505 sec, root delay is 0.000000 sec root dispersion is 0.004059 sec, peer dispersion is 411 msec
Trivia: Failed Upgrade
Initially, I wanted to upgrade the FortiGate for this blogpost to its latest version from v5.6.6 to v5.6.8. Just a minor upgrade, right? However, this upgrade destroyed my VPN that was needed for the NTP servers. Even downgrading the version and restoring hasn’t worked. Just another example why I don’t really like those FortiGates. Details:
I just did a minor upgrade on a @Fortinet #FortiGate FG-100D (5.6.6 to 5.6.8) with the result that my configured IPsec VPN tunnel is completely lost. Fortinet at its best. #fail #Ihaveit pic.twitter.com/76pwBYyFMM
— Johannes Weber (@webernetz) March 22, 2019
Featured image “handypics August 2015 087” by PercyGermany is licensed under CC BY-NC-ND 2.0.
Using a FortiGate with a 6in4 Tunnel
For some reason, I am currently using a FortiGate on a location that has no native IPv6 support. Uh, I don’t want to talk about that. ;) However, at least the FortiGate firewalls are capable of 6in4 tunnels. Hence I am using the IPv6 Tunnel Broker from Hurricane Electric again. Quite easy so far.
But note, as always: Though FortiGate supports these IPv6 features such as a 6in4 tunnel or stateful/-less DHCPv6 server, those features are NOT stable or well designed at all. I had many bugs and outages during my last years. Having “NAT enabled” on every new IPv6 policy is ridiculous. Furthermore, having independent security policies for legacy IP and IPv6 is obviously a really bad design. One single policy responsible for both Internet protocols is a MUST. Anyway, let’s look at the 6in4 tunnel:
Configuring this IPv6-in-IPv4 tunnel is quite easy since HE itself offers the configuration:
Of course, you need an internal layer 3 interface as well. That is, a complete configuration (6in4 tunnel, default route, inside interface with RDNSS) looks like that:
config system sit-tunnel edit "HE" set destination 216.66.80.30 set ip6 2001:470:1f0a:16b0::2/64 set source 194.247.4.10 next end config router static6 edit 1 set device "HE" next end config system interface edit "internal" config ipv6 set ip6-address 2001:470:1f0b:16b0::1/64 set ip6-allowaccess ping https ssh set ip6-send-adv enable config ip6-prefix-list edit 2001:470:1f0b:16b0::/64 set autonomous-flag enable set onlink-flag enable set rdnss 2620:fe::fe set dnssl "weberlab.de" next end end next end
Finally, you need some IPv6 policy entries to permit traffic. Again, note that you MUST NOT select the NAT, which is stupidly pre-selected by Fortinet:
On every new IPv6 policy on a @Fortinet FortiGate firewall the default selection is "NAT". #fail #IPv6 pic.twitter.com/BfS4qKEfc6
— Johannes Weber (@webernetz) March 1, 2017
Stumbling Blocks
I am using a FortiGate FG-90D with FortiOS 5.6.8 build1672 (GA).
Note that this “HE” interface, as it is named in the example configuration above, is NOT visible in the interface section in the GUI:
while it IS visible in the routing section:
Honestly: Who is approving such decisions at Fortinet? This is not sound at all, isn’t it?
Verifying
You can have a look at the routing monitor to see the default route in place:
Some CLI commands are as follows. Getting information about the tunnel interface you can use this kind of hidden command:
fnsysctl ifconfigsuch as:
fg2 # fnsysctl ifconfig HE HE Link encap:Unknown HWaddr C2:F7:04:0A:00:00 inet addr6: 2001:470:1f0a:16b0::2 prefixlen 64 link-local6: fe80::c2f7:40a prefixlen 128 UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1480 Metric:1 RX packets:664858 errors:0 dropped:0 overruns:0 frame:0 TX packets:1015185 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:179754139 (171.4 MB) TX bytes:114587175 (109.3 MB)
IPv6 routing table:
fg2 # get router info6 routing-table IPv6 Routing Table Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 I - IS-IS, B - BGP * - candidate default Timers: Uptime S* ::/0 [10/0] via ::, HE, 01w2d12h C ::1/128 via ::, root, 04w2d19h C 2001:470:1f0a:16b0::/64 via ::, HE, 01w2d12h C 2001:470:1f0b:16b0::/64 via ::, internal, 01w2d11h C fe80::/10 via ::, internal, 01w2d11h C fe80::c2f7:40a/128 via ::, HE, 01w2d12h
And some basic network connectivity test, aka ping:
fg2 # execute ping6-options reset fg2 # execute ping6 weberblog.net PING weberblog.net(2a01:488:42:1000:50ed:8588:8a:c570) 56 data bytes 64 bytes from 2a01:488:42:1000:50ed:8588:8a:c570: icmp_seq=1 ttl=56 time=9.13 ms 64 bytes from 2a01:488:42:1000:50ed:8588:8a:c570: icmp_seq=2 ttl=56 time=11.4 ms 64 bytes from 2a01:488:42:1000:50ed:8588:8a:c570: icmp_seq=3 ttl=56 time=9.57 ms 64 bytes from 2a01:488:42:1000:50ed:8588:8a:c570: icmp_seq=4 ttl=56 time=10.3 ms 64 bytes from 2a01:488:42:1000:50ed:8588:8a:c570: icmp_seq=5 ttl=56 time=10.1 ms --- weberblog.net ping statistics --- 5 packets transmitted, 5 packets received, 0% packet loss, time 4045ms rtt min/avg/max/mdev = 9.136/10.145/11.487/0.806 ms
That’s it. Thanks for watching. ;) Don’t forget to hit the subscribe button.
Featured image “Make It Count” by Mr. Nixter is licensed under CC BY-NC 2.0.
iperf3 on a FortiGate
This is a really nice feature: you can run iperf3 directly on a FortiGate to speed-test your network connections. It’s basically an iperf3 client. Using some public iperf servers you can test your Internet bandwidth; using some internal servers you can test your own routed/switched networks, VPNs, etc. However, the maximum throughput for the test is CPU dependent. So please be careful when interpreting the results. Here we go:
I am using a FortiGate FG-90D with FortiOS v6.0.10. I don’t know whether this iperf implementation is present on all FortiOS releases on all FortiGates. On mine, it is. ;) Here is more information about iperf3.
You have to set at least the iperf client and server interface on the FortiGate in order to run it. The server interface is NOT used when testing the bandwidth to an external server. However, you have to specify it, otherwise, you’re getting an error. (You can test internal paths within the FortiGate <- that’s why you have to set the client and server interface. However, I don’t know whether these tests will have any value.) To test your ISP connection, you have to find a public iperf server, e.g., here: https://iperf.cc/. The FortiGate implementation of iperf does not accept hostnames, but only IP addresses.
Test, Test, Test
A basic run looks like this. Using port 5200 (in my example) and testing in both directions:
diagnose traffictest client-intf wan1 diagnose traffictest server-intf wan1 diagnose traffictest port 5200 diagnose traffictest run -c 213.209.106.95 diagnose traffictest run -R -c 213.209.106.95
That is:
fg2 # diagnose traffictest client-intf wan1 client-intf: wan1 fg2 # diagnose traffictest server-intf wan1 server-intf: wan1 fg2 # diagnose traffictest port 5200 port: 5200 fg2 # diagnose traffictest show server-intf: wan1 client-intf: wan1 port: 5200 proto: TCP fg2 # diagnose traffictest run -c 213.209.106.95 Connecting to host 213.209.106.95, port 5200 [ 8] local 194.247.4.10 port 1489 connected to 213.209.106.95 port 5200 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 8] 0.00-1.03 sec 15.6 MBytes 127 Mbits/sec 0 359 KBytes [ 8] 1.03-2.00 sec 16.2 MBytes 140 Mbits/sec 0 410 KBytes [ 8] 2.00-3.05 sec 18.8 MBytes 150 Mbits/sec 0 385 KBytes [ 8] 3.05-4.01 sec 16.2 MBytes 143 Mbits/sec 0 392 KBytes [ 8] 4.01-5.06 sec 18.8 MBytes 149 Mbits/sec 0 380 KBytes [ 8] 5.06-6.04 sec 16.2 MBytes 140 Mbits/sec 0 389 KBytes [ 8] 6.04-7.04 sec 17.5 MBytes 146 Mbits/sec 0 387 KBytes [ 8] 7.04-8.05 sec 16.2 MBytes 135 Mbits/sec 0 404 KBytes [ 8] 8.05-9.06 sec 17.5 MBytes 145 Mbits/sec 0 386 KBytes [ 8] 9.06-10.06 sec 17.5 MBytes 148 Mbits/sec 0 386 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 8] 0.00-10.06 sec 171 MBytes 142 Mbits/sec 0 sender [ 8] 0.00-10.06 sec 171 MBytes 142 Mbits/sec receiver iperf Done. iperf3: interrupt - the server has terminated fg2 # diagnose traffictest run -R -c 213.209.106.95 Connecting to host 213.209.106.95, port 5200 Reverse mode, remote host 213.209.106.95 is sending [ 8] local 194.247.4.10 port 1491 connected to 213.209.106.95 port 5200 [ ID] Interval Transfer Bandwidth [ 8] 0.00-1.00 sec 8.02 MBytes 67.0 Mbits/sec [ 8] 1.00-2.00 sec 8.13 MBytes 68.4 Mbits/sec [ 8] 2.00-3.00 sec 8.27 MBytes 69.5 Mbits/sec [ 8] 3.00-4.00 sec 8.19 MBytes 68.7 Mbits/sec [ 8] 4.00-5.00 sec 8.51 MBytes 71.2 Mbits/sec [ 8] 5.00-6.00 sec 8.46 MBytes 71.1 Mbits/sec [ 8] 6.00-7.00 sec 8.08 MBytes 67.7 Mbits/sec [ 8] 7.00-8.02 sec 8.32 MBytes 68.7 Mbits/sec [ 8] 8.02-9.03 sec 8.32 MBytes 69.1 Mbits/sec [ 8] 9.03-10.01 sec 7.96 MBytes 68.1 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 8] 0.00-10.01 sec 83.7 MBytes 70.1 Mbits/sec 0 sender [ 8] 0.00-10.01 sec 82.4 MBytes 69.0 Mbits/sec receiver iperf Done. iperf3: interrupt - the server has terminated
Other useful commands are:
diagnose traffictest show diagnose traffictest run -v diagnose traffictest run -h
They show the current configuration on the FortiGate (first one) and some more details about iperf itself:
fg2 # diagnose traffictest show server-intf: wan1 client-intf: wan1 port: 5200 proto: TCP fg2 # diagnose traffictest run -v iperf 3.0.9 fg2 # diagnose traffictest run -h -f, --format [kmgKMG] format to report: Kbits, Mbits, KBytes, MBytes -i, --interval # seconds between periodic bandwidth reports -F, --file name xmit/recv the specified file -A, --affinity n/n,m set CPU affinity -V, --verbose more detailed output -J, --json output in JSON format -d, --debug emit debugging output -v, --version show version information and quit -h, --help show this message and quit -b, --bandwidth #[KMG][/#] target bandwidth in bits/sec (0 for unlimited) (default 1 Mbit/sec for UDP, unlimited for TCP) (optional slash and packet count for burst mode) -t, --time # time in seconds to transmit for (default 10 secs) -n, --bytes #[KMG] number of bytes to transmit (instead of -t) -k, --blockcount #[KMG] number of blocks (packets) to transmit (instead of -t or -n) -l, --len #[KMG] length of buffer to read or write (default 128 KB for TCP, 8 KB for UDP) -P, --parallel # number of parallel client streams to run -R, --reverse run in reverse mode (server sends, client receives) -w, --window #[KMG] TCP window size (socket buffer size) -C, --linux-congestion <algo> set TCP congestion control algorithm (Linux only) -M, --set-mss # set TCP maximum segment size (MTU - 40 bytes) -N, --nodelay set TCP no delay, disabling Nagle's Algorithm -4, --version4 only use IPv4 -6, --version6 only use IPv6 -S, --tos N set the IP 'type of service' -L, --flowlabel N set the IPv6 flow label (only supported on Linux) -Z, --zerocopy use a 'zero copy' method of sending data -O, --omit N omit the first n seconds -T, --title str prefix every output line with this string --get-server-output get results from server [KMG] indicates options that support a K/M/G suffix for kilo-, mega-, or giga-
Caveats, Caveats, Caveats
Unfortunately, here are some (major!) caveats: At first, the iperf implementation on the FortiGate is heavily CPU related. My FG-90D has a 1 Gbps uplink to the Internet. Running iperf3 on the Forti reveals only about 150 Mbps (see above), while the CPU usage immediately peaked at 100 %. Ouch:
Testing my ISP speed *through* the FortiGate from a Linux system behind it, iperf3 showed about 900 Mbps, while the CPU usage on the Forti stayed by about 3-5 %. Following is the bandwidth widget from the Forti during my tests:
Certainly this behavior is different on other FortiGates hardware. To be fair, my FG-90D is not the newest nor the biggest model. I have tested the traffictest feature on a FG-501E with FortiOS v6.2.5 which was able to receive 900 Mbps while only one out of eight cores peaked at about 25 %.
Second caveat: it’s not working with IPv6, but only with legacy IP. :(
fg2 # diagnose traffictest run -c 2a02:2028:ff00::f9:2 iperf3: error - unable to connect to server: Invalid argument iperf3: interrupt - the server has terminated fg2 # diagnose traffictest run -6 -c 2a02:2028:ff00::f9:2 iperf3: error - unable to connect to server: iperf3: interrupt - the server has terminated
Conclusion
Uh, that’s hard. In theory, this is a cool hidden feature. If you’re keeping track of your CPU usage you can probably use it for getting realistic results. Especially on links with small bandwidth.
However, if you really want to test your big ISP connection, you shouldn’t rely on it. Or to say it differently: If you’re getting the expected results with iperf on the Forti, you’re ok. If not, you don’t know why. ;(
PS: Happy Birthday Nicolai!
Photo by Harley-Davidson on Unsplash.
FortiGate bug: firewalls sending excessive requests to the NTP Pool
The NTP Pool is a volunteer organization that provides time synchronization service to hundreds of millions of computers worldwide. A typical client might query a particular NTP Pool server ~10-60 times/hour. Wikipedia lists some abusive clients that far exceeded the normal rate. This wastes NTP server resources, may interfere with other clients, and can trigger DDoS protections. In late 2019, a software update made some FortiGate firewalls very unfriendly to the NTP Pool.
Fortinet is a multinational corporation that produces computer security devices, including the FortiGate firewall. The firewall product documentation describes how to use the NTP Pool for time synchronization, e.g., by using:
0.europe.pool.ntp.org 1.europe.pool.ntp.org 2.europe.pool.ntp.org
DNS will resolve 0.europe.pool.ntp.org into one of the hundreds of NTP servers in the NTP Pool. Let’s say that at firewall startup 1.2.3.4 was the NTP server selected by DNS. What happens if sometime later the administrator of 1.2.3.4 decides to stop participating in the NTP Pool and shuts down the public NTP service? We were told that older FortiGate NTP software did not handle that condition gracefully. A software update, FortiOS 6.2.3, was issued in December 2019 that periodically reissued the DNS lookup. In our example, if 0.europe.pool.ntp.org no longer resolved into 1.2.3.4 some re-initialization was done by the new software. That new code had a serious flaw. Bursts of NTP requests with rates that sometimes exceeded 20,000/sec were sent for up to 10 seconds duration. As DNS resolution changed, which is common for the NTP pools, a single FortiGate firewall would send bursts to different NTP pool servers.
A burst from a single FortiGate firewall is shown below
In early 2020, NTP pool operators noticed these distinctive bursts. They sometimes represented over 20 % of the total NTP traffic on a single server! Diagnostic work by several NTP Pool volunteers identified FortiGate firewalls as the source of these bursts. In April 2020, we contacted Fortinet support who acknowledged the bug and said that the next software release containing a fix would soon be deployed. However, the new software had unrelated bugs that prevented it from being widely deployed. On August 22, 2020, FortiOS 6.2.5 was released to eliminate the bursts.
The fix for FortiGate administrators is simple:
- Do not use FortiOS release 6.2.3, or
- if using release 6.2.3, do not use the NTP Pool.
FortiGate notified a handful of customers of the problem. However, as of this writing (November 2020), the NTP Pool is still receiving frequent NTP bursts from hundreds (at least) of FortiGate firewalls. While Fortinet was responsive in email and phone discussions, on the whole, their corrective actions were disappointing. Hopefully, FortiGate administrators will read this note and take corrective action as appropriate.
The diagnostic work was done by Miroslav Lichvar, Hal Murray, and Steven Sommars with contributions from several other NTP Pool volunteers.
Photo by Heather Zabriskie on Unsplash.
Route-Based VPN Tunnel FortiGate Cisco ASA
More than 6 years ago (!) I published a tutorial on how to set up an IPsec VPN tunnel between a FortiGate firewall and a Cisco ASA. As time flies by, ASA is now able to terminate route-based VPN tunnels (which is great!), we have IKEv2 running everywhere and enhanced security proposals. Hence, it’s time for an update:
My Setup
This is my setup for this tutorial: (Yes, public IPv4 addresses behind the Forti.)
I am using a Fortinet FortiWiFi FWF-61E with FortiOS v6.2.5 build1142 (GA) and a Cisco ASA 5515 with version 9.12(3)12 and ASDM 7.14(1). These are the VPN parameters:
- Route-based VPN, that is: numbered tunnel interface and real route entries for the network(s) to the other side. But no proxy-IDs aka traffic selection aka crypto map. Thank goodness for that.
- The tunnel interface on the Forti is added during the VPN setup automatically. However, you have to set the IP address on the tunnel interface manually after that. The static route on the ASA needs an IP address as the gateway.
- IKEv2 (no distinction anymore between main or aggressive mode as with IKEv1)
- PSK: 30 chars alphanumeric, generated with a password generator! (ref)
- IKE crypto/policies:
- Diffie-Hellman group 21
- AES-256-GCM
- SHA-512 (you could use SHA-256 if you like)
- 8 hours
- IPsec crypto/proposals/transform sets:
- AES-256-GCM
- SHA-512 (again, you can use SHA-256 as well)
- Diffie-Hellman group 21
- 1 hour
- No NAT between the internal networks (of course not ;))!
FortiGate
You can do the configuration through the GUI:
or through the CLI: (incl. the zone commands <- can be omitted if you aren’t using zones)
config system interface edit "asa" set vdom "root" set ip 10.1.37.1 255.255.255.255 set allowaccess ping set type tunnel set remote-ip 10.1.37.2 255.255.255.252 set interface "wan1" next end config system zone edit "s2s-vpns" set interface "asa" next end config vpn ipsec phase1-interface edit "asa" set interface "wan1" set ike-version 2 set keylife 28800 set peertype any set net-device enable set proposal aes256gcm-prfsha512 set dhgrp 21 set nattraversal disable set remote-gw 185.23.77.7 set psksecret ThisIsThePreSharedKey next end config vpn ipsec phase2-interface edit "asa" set phase1name "asa" set proposal aes256gcm set dhgrp 21 set keylifeseconds 3600 next end config router static edit 5 set dst 172.16.37.0 255.255.255.0 set device "asa" next end
Cisco ASA
Same on the ASA, either via the “GUI”:
or via classical CLI commands: (The ACL is omitted.)
interface Tunnel1 nameif fg2 ip address 10.1.37.2 255.255.255.252 tunnel source interface outside tunnel destination 194.247.4.10 tunnel mode ipsec ipv4 tunnel protection ipsec profile aes256gcm-sha512-dh21-3600s ! route fg2 194.247.5.0 255.255.255.224 10.1.37.1 1 ! crypto ipsec ikev2 ipsec-proposal aes256gcm-sha512 protocol esp encryption aes-gcm-256 protocol esp integrity sha-512 crypto ipsec profile aes256gcm-sha512-dh21-3600s set ikev2 ipsec-proposal aes256gcm-sha512 set pfs group21 set security-association lifetime seconds 3600 crypto ikev2 policy 1 encryption aes-gcm-256 integrity null group 21 prf sha512 lifetime seconds 28800 ! group-policy 194.247.4.10 internal group-policy 194.247.4.10 attributes vpn-tunnel-protocol ikev2 tunnel-group 194.247.4.10 type ipsec-l2l tunnel-group 194.247.4.10 general-attributes default-group-policy 194.247.4.10 tunnel-group 194.247.4.10 ipsec-attributes ikev2 remote-authentication pre-shared-key ThisIsThePreSharedKey ikev2 local-authentication pre-shared-key ThisIsThePreSharedKey
Monitoring
Some screenshots from the FortiGate:
as well as CLI outputs:
fg2 # get vpn ike gateway asa vd: root/0 name: asa version: 2 interface: wan1 6 addr: 194.247.4.10:500 -> 185.23.77.7:500 created: 3158587s ago IKE SA created: 1/111 established: 1/111 time: 0/3/100 ms IPsec SA created: 1/973 established: 1/973 time: 0/0/100 ms id/spi: 2040 7be16624b6a980a3/b107958ab150a4fb direction: initiator status: established 23585-23585s ago = 10ms proposal: unknown-256-unknown SK_ei: c301af190feb89e7-e89076489227f77e-73a80ecd3692c0c7-925c73a84a30c063-618eb9af SK_er: 5362b4bc6103b45f-776a3e817a61026f-75b7cd0220fb8d70-05f32a71240799e6-f1441bb6 SK_ai: SK_ar: lifetime/rekey: 28800/4914 DPD sent/recv: 00000000/00000000 fg2 # fg2 # fg2 # get vpn ipsec tunnel name asa gateway name: 'asa' type: route-based local-gateway: 194.247.4.10:0 (static) remote-gateway: 185.23.77.7:0 (static) mode: ike-v2 interface: 'wan1' (6) rx packets: 110976 bytes: 145943836 errors: 0 tx packets: 64092 bytes: 3004962 errors: 0 dpd: on-demand/negotiated idle: 20000ms retry: 3 count: 0 selectors name: 'asa' auto-negotiate: disable mode: tunnel src: 0:0.0.0.0/0.0.0.0:0 dst: 0:0.0.0.0/0.0.0.0:0 SA lifetime/rekey: 3600/2171 mtu: 1446 tx-esp-seq: 27 replay: enabled qat: 0 inbound spi: 15ad154f enc: aes-gc e1831416107c6ca5c1d6da624269ba4e21b7d45c95d5a16da8c0f9200b598ebbab76f5b9 auth: null outbound spi: 9573f1de enc: aes-gc 3d6e5ab8c1ac1de02a230095d76778dd5b88aeeff7dfae8b25df26c265bdec56710d040e auth: null NPU acceleration: none fg2 # fg2 # fg2 # diagnose vpn tunnel list name asa list ipsec tunnel by names in vd 0 ------------------------------------------------------ name=asa ver=2 serial=3 194.247.4.10:0->185.23.77.7:0 dst_mtu=1500 bound_if=6 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/536 options[0218]=npu create_dev frag-rfc accept_traffic=1 overlay_id=0 proxyid_num=1 child_num=0 refcnt=14 ilast=12 olast=12 ad=/0 stat: rxp=110977 txp=64094 rxb=145943972 txb=3005118 dpd: mode=on-demand on=1 idle=20000ms retry=3 count=0 seqno=440 natt: mode=none draft=0 interval=0 remote_port=0 proxyid=asa proto=0 sa=1 ref=3 serial=2 src: 0:0.0.0.0/0.0.0.0:0 dst: 0:0.0.0.0/0.0.0.0:0 SA: ref=3 options=10226 type=00 soft=0 mtu=1446 expire=2114/0B replaywin=1024 seqno=29 esn=0 replaywin_lastseq=00000014 itn=0 qat=0 hash_search_len=1 life: type=01 bytes=0/0 timeout=3298/3600 dec: spi=15ad154f esp=aes-gcm key=36 e1831416107c6ca5c1d6da624269ba4e21b7d45c95d5a16da8c0f9200b598ebbab76f5b9 ah=null key=0 enc: spi=9573f1de esp=aes-gcm key=36 3d6e5ab8c1ac1de02a230095d76778dd5b88aeeff7dfae8b25df26c265bdec56710d040e ah=null key=0 dec:pkts/bytes=20/1600, enc:pkts/bytes=40/5360 npu_flag=20 npu_rgwy=185.23.77.7 npu_lgwy=194.247.4.10 npu_selid=5 dec_npuid=0 enc_npuid=0 fg2 # fg2 # fg2 # get router info routing-table all Routing table for VRF=0 Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2 i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area * - candidate default S* 0.0.0.0/0 [10/0] via 194.247.4.1, wan1 C 10.1.37.0/30 is directly connected, asa C 10.1.37.1/32 is directly connected, asa S 172.16.37.0/24 [10/0] via 10.1.37.2, asa S 192.168.11.0/24 [10/0] is directly connected, ssg5-weberhom S 193.24.227.224/27 [10/0] is directly connected, pa C 194.247.4.0/27 is directly connected, wan1 C 194.247.5.0/27 is directly connected, internal fg2 #
And some screenshots from the ASA: (the third one showing the logs after a manual “logout”)
as well as CLI outputs:
asa# show crypto ikev2 sa detail IKEv2 SAs: Session-id:16, Status:UP-ACTIVE, IKE count:1, CHILD count:1 Tunnel-id Local Remote Status Role 1219040189 185.23.77.7/500 194.247.4.10/500 READY INITIATOR Encr: AES-GCM, keysize: 256, Hash: N/A, DH Grp:21, Auth sign: PSK, Auth verify: PSK Life/Active Time: 28800/298 sec Session-id: 16 Status Description: Negotiation done Local spi: E82116F37CF38D12 Remote spi: 3D48FE4CB448BA6B Local id: 185.23.77.7 Remote id: 194.247.4.10 Local req mess id: 26 Remote req mess id: 0 Local next mess id: 26 Remote next mess id: 0 Local req queued: 26 Remote req queued: 0 Local window: 1 Remote window: 1 DPD configured for 10 seconds, retry 2 NAT-T is not detected IKEv2 Fragmentation Configured MTU: 576 bytes, Overhead: 28 bytes, Effective MTU: 548 bytes Child sa: local selector 0.0.0.0/0 - 255.255.255.255/65535 remote selector 0.0.0.0/0 - 255.255.255.255/65535 ESP spi in/out: 0x5f713ed2/0x15ad1552 AH spi in/out: 0x0/0x0 CPI in/out: 0x0/0x0 Encr: AES-GCM, keysize: 256, esp_hmac: N/A ah_hmac: None, comp: IPCOMP_NONE, mode tunnel Parent SA Extended Status: Delete in progress: FALSE Marked for delete: FALSE asa# asa# asa# show crypto ipsec sa peer 194.247.4.10 detail peer address: 194.247.4.10 Crypto map tag: __vti-crypto-map-5-0-1, seq num: 65280, local addr: 185.23.77.7 local ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0) remote ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0) current_peer: 194.247.4.10 #pkts encaps: 29, #pkts encrypt: 29, #pkts digest: 29 #pkts decaps: 45, #pkts decrypt: 45, #pkts verify: 45 #pkts compressed: 0, #pkts decompressed: 0 #pkts not compressed: 29, #pkts comp failed: 0, #pkts decomp failed: 0 #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0 #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0 #TFC rcvd: 0, #TFC sent: 0 #Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0 #pkts no sa (send): 0, #pkts invalid sa (rcv): 0 #pkts encaps failed (send): 0, #pkts decaps failed (rcv): 0 #pkts invalid prot (rcv): 0, #pkts verify failed: 0 #pkts invalid identity (rcv): 0, #pkts invalid len (rcv): 4009213712 #pkts invalid pad (rcv): 0, #pkts invalid ip version (send): 0, #pkts invalid ip version (rcv): 0 #pkts invalid len (send): 0, #pkts invalid len (rcv): 0 #pkts invalid ctx (send): 0, #pkts invalid ctx (rcv): 0 #pkts invalid ifc (send): 0, #pkts invalid ifc (rcv): 0 #pkts failed (send): 0, #pkts failed (rcv): 0 #pkts replay rollover (send): 0, #pkts replay rollover (rcv): 0 #pkts replay failed (rcv): 0 #pkts min mtu frag failed (send): 0, #pkts bad frag offset (rcv): 0 #pkts internal err (send): 0, #pkts internal err (rcv): 0 local crypto endpt.: 185.23.77.7/500, remote crypto endpt.: 194.247.4.10/500 path mtu 1500, ipsec overhead 55(36), media mtu 1500 PMTU time remaining (sec): 0, DF policy: copy-df ICMP error validation: disabled, TFC packets: disabled current outbound spi: 15AD1552 current inbound spi : 5F713ED2 inbound esp sas: spi: 0x5F713ED2 (1601257170) SA State: active transform: esp-aes-gcm-256 esp-null-hmac no compression in use settings ={L2L, Tunnel, PFS Group 21, IKEv2, VTI, } slot: 0, conn_id: 51, crypto-map: __vti-crypto-map-5-0-1 sa timing: remaining key lifetime (kB/sec): (3962873/3231) IV size: 8 bytes replay detection support: Y Anti replay bitmap: 0xAAAAAAAA 0xAAAAB8AA outbound esp sas: spi: 0x15AD1552 (363664722) SA State: active transform: esp-aes-gcm-256 esp-null-hmac no compression in use settings ={L2L, Tunnel, PFS Group 21, IKEv2, VTI, } slot: 0, conn_id: 51, crypto-map: __vti-crypto-map-5-0-1 sa timing: remaining key lifetime (kB/sec): (4193275/3231) IV size: 8 bytes replay detection support: Y Anti replay bitmap: 0x00000000 0x00000001 asa# asa# asa# show route static Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2 E1 - OSPF external type 1, E2 - OSPF external type 2, V - VPN i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2 ia - IS-IS inter area, * - candidate default, U - per-user static route o - ODR, P - periodic downloaded static route, + - replicated route Gateway of last resort is 185.23.77.1 to network 0.0.0.0 S* 0.0.0.0 0.0.0.0 [1/0] via 185.23.77.1, outside S 194.247.5.0 255.255.255.224 [1/0] via 10.1.37.1, fg2 asa#
PS: Sorry for being legacy IP only this time. ;(
Photo by Casey Horner on Unsplash.
FortiGate Syslog via TLS
As we have just set up a TLS capable syslog server, let’s configure a Fortinet FortiGate firewall to send syslog messages via an encrypted channel (TLS). Let’s go:
I am using a Fortinet FortiGate (FortiWiFi) FWF-61E with FortiOS v6.4.7 build1911 (GA) for this tutorial. My syslog-ng server with version 3.13.2 is running on Ubuntu 18.04.6 LTS.
You cannot configure any syslog server details (rather than the address itself) via the GUI on this so-called “Next Generation Firewall”. Log & Report -> Log Settings -> Remote Logging and Archiving:
Hence we have to use the CLI:
config log syslogd setting set status enable set server "syslog2.weberlab.de" set mode reliable set port 6514 set enc-algorithm high end
Some notes:
- You can’t “set enc-algorithm high” unless you have “set mode reliable”.
- You don’t have to “set port 6514” manually because it is set automatically by doing the “set enc-algorithm high” command.
- Using an FQDN as the server (as I did in the listing), the FortiGate will use legacy IP though an AAAA record is present. If you want to use IPv6 you must use an IPv6 address here.
- ;)
- Official Fortinet CLI reference
If your syslog server uses a self-signed or untrusted certificate it won’t work right now. This is good from a security point of view. You’ll see the following syslog messages on your syslog server: “tlsv1 alert unknown ca”:
weberjoh@nb17-lx2:~$ tail -f /var/log/syslog Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: Syslog connection accepted; fd='96', client='AF_INET6([2001:470:1f0b:16b0::1]:15831)', local='AF_INET6([::]:6514)' Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: Syslog connection accepted; fd='99', client='AF_INET6([2001:470:1f0b:16b0::1]:15830)', local='AF_INET6([::]:6514)' Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: SSL error while reading stream; tls_error='SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca' Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: I/O error occurred while reading; fd='96', error='Connection reset by peer (104)' Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: Syslog connection closed; fd='96', client='AF_INET6([2001:470:1f0b:16b0::1]:15831)', local='AF_INET6([::]:6514)' Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: SSL error while reading stream; tls_error='SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca' Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: I/O error occurred while reading; fd='99', error='Connection reset by peer (104)' Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: Syslog connection closed; fd='99', client='AF_INET6([2001:470:1f0b:16b0::1]:15830)', local='AF_INET6([::]:6514)'
Or looking at it with Wireshark you’ll see this “Alert Message”:
Import the syslog x.509 certificate at System -> Certificates -> Import -> CA Certificate:
Logging via TLS will immediately start after that. No further configuration is needed.
Confirmed with Wireshark again: “Change Cipher Spec” followed by Application Data. Nice!
Very good. Keep in mind that you should not use unencrypted management protocols at all. Hence this one here is a step in the right direction.
Photo by Robin Spielmann on Unsplash.
Publishing IPv6 NTP Servers with DHCPv6
During the last weeks, I had an interesting request to publish NTP servers to client systems by using DHCPv6 in an IPv6 only network. Our Fortigate (or me?) had to learn how to publish the information. Hence this post is not only about NTP and IPv6, but a small guide on how to walk through RFCs and how to get out the relevant information. I’m very happy I got the possibility to share my experience here. Thank you, Johannes!
It started with a small question from my co-worker: “Can you send our NTP servers with DHCP in the IPv6 only network?” And my first idea was: “I’m sure, I can configure this within of minutes.” But 10 minutes later I had to change my mind: it takes me more time to understand how I have to configure this. The simple reason for this is the missing option in the system configuration, the vendor didn’t implement (yet)…
I will show how to manage this with a FortiGate (FortiOS 6.0.14, there is no newer version available for this model) operating as a DHCPv6 server. At the time of beginning, the DHCP6 server configuration looks like this:
config system dhcp6 server edit 641 set rapid-commit enable set lease-time 3602 set domain "example.com"IP set subnet 2001:db8:c:641::/64 set interface "IPv6only-01" config ip-range edit 1 set start-ip 2001:db8:c:641::6401 set end-ip 2001:db8:c:641::64ff next end set dns-server1 2001:db8:c:a53::53 set dns-server2 2001:db8:c:b53::53 next end
As you can see, the DHCPv6 server is configured to publish a domain name and two DNS servers for recursive name resolution. Further on there’s an IPv6 range for stateful DHCPv6. But FortiOS didn’t allow me to configure the NTP server directly. But it allows me to send three self-defined options in DHCPv6:
config system dhcp6 server edit < id > set option1 {string} Option 1. set option2 {string} Option 2. set option3 {string} Option 3. next end
Could this be the way to fulfil my co-worker’s request? And how to create the {string} to send an IPv6 address or hostname to the client system? Having a look at the CLI of the FortiGate:
config system dhcp6 server edit < id > (server) # set option1 ? < option-code > [< options >] options must be even number of hexadecimal characters(optional)
This information didn’t help me. Now it looks like nothing else than reading RFCs helps. :-(
Digging through RFCs
In the list of FortiOS 6.0 supported RFCs, RFC 3315 is mentioned. But in RFC 3315 is no useful information about NTP… Reading the new RFC for DHCPv6, RFC 8415, you can find in section 24 a list of DHCPv6 options and the link to the complete list of DHCPv6 parameters assigned by IANA. Looks like that’s the information I need.
There is an option with the name “NTP Server”, number 56 (decimal), but the information about the order and possible values are somewhere else: in RFC 5908.
In RFC typical style there (RFC 5908 on page 4) you can find the required format of the option to send NTP-server information to the client system:
The format of the NTP Server Option is: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | OPTION_NTP_SERVER | option-len | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | suboption-1 | : : +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | suboption-2 | : : +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ : : +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | suboption-n | : : +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ option-code: OPTION_NTP_SERVER (56), option-len: Total length of the included suboptions.
For the ones who never had the need to read something like this, a few hints which helped me:
- The headline of the “table” are the bits, starting from 00 to 31 = 32bit.
- The field OPTION_NTP_SERVER is a 16-bit value and the field option-len is a 16bit value, too.
- The next fields suboption-1, suboption-2 to suboption-n has, at this point, no fixed length; but at least 32bit. This is because there is a pipe character on the left and right side, that indicates: this line is required. The next line, which has a colon at both sides, tells us it has a variable length, 0 or more lines.
To find the next piece of the puzzle, we continue reading the next page in RFC 5908 (page 5). Here we find how to format the sub-options:
The format of the NTP Server Address Suboption is:
0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | NTP_SUBOPTION_SRV_ADDR | suboption-len = 16 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | | | IPv6 address of NTP server | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ IPv6 address of the NTP server: An IPv6 address, suboption-code: NTP_SUBOPTION_SRV_ADDR (1), suboption-len: 16.
Next hint:
- The field NTP_SUBOPTION_SRV_ADDR is a 16-bit value and the field suboption-len is a 16bit value, too.
- The next field (IPv6 address of NTP server) has a fixed length of 128 bit (4 lines with 32 bit). I hope you already expected this length. :-)
OK, we try to understand the information we have until now: We have to set the option number for the NTP server, which is 56 (dec) and 0x38 (hex). The NTP_SUBOPTION_SRV_ADDR has a sub-option code of 1 (0x0001 as 16bit hex value) and a suboption-length of 16 octets (if it is easier for you: 16 bytes). Then we have to put in the IPv6 (unicast-) address of the server. We take 2001:0db8:2800:0000:0000:0000:0ac3:0123 as an example. In the end, we calculate the complete length of the option and put the value in the field option-len:
- 2 octets for the NTP suboption field
- 2 octets for the NTP suboption length
- 16 octets for the NTP IPv6 server address
In summary 20 octets (0x14 in hex) is the calculated value for the field option-len. If we put in all the data, the fields look like this:
0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | OPTION_NTP_SERVER = 0x0038 | option-len = 0x0014 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |SUBOPTION_NTP_SRV_ADDR = 0x0001| suboption-len = 16dec = 0x0010| +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | 2001:0db8: | | 2800:0000: | | 0000:0000: IPv6 address of NTP server | | 0ac3:0123 | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
But in FortiOS this is different than expected (by me?): we don’t have to calculate and put in the option-length here! We just need the option code (in decimal = 56) and the option value in hexadecimal characters as in the following line:
(server) # set option1 56 '0001001020010db828000000000000000ac30123' NTP_SUBOPTION_SRV_ADDR | | suboption-lenght | IPv6 unicast address of NTP server
It took me a few hours with Wireshark to realize this…
Publishing FQDNs
My co-worker was quite happy about my phone call with the good news. But: is publishing an IPv6 unicast address a good solution for the future? I decided no, I want to publish an FQDN.
Now, as I did understand the format of the NTP-server option, I had a look for how to publish the name of the NTP server. For this we have to read section 4.3 on page 6 of RFC 5908. The format of the NTP Server FQDN Suboption is:
0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | NTP_SUBOPTION_SRV_FQDN | suboption-len | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | FQDN of NTP server | : : +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ suboption-code: NTP_SUBOPTION_SRV_FQDN (3), suboption-len: Length of the included FQDN field, FQDN: Fully-Qualified Domain Name of the NTP server or SNTP server. This field MUST be encoded as described in [RFC3315], Section 8. Internationalized domain names are not allowed in this field.
Let’s have a try to send the hostname 3.de.pool.ntp.org to the client systems.
The FQDN must be encoded as described in section 8 of RFC 3315. The few lines there tell us to read section 3.1 of RFC 1035 but MUST NOT use the compressed form as described in section 4.1.4 of RFC 1035. And we MUST NOT use an IDN (internationalized domain name)!
This is how encode the hostname 3.de.pool.ntp.org based on section 3.1 in RFC 1035: First of all, we have to specify the length of the next part of the hostname (also called: label):
- one character which results in value 1 (0x01) followed by the ascii-code of the character 3: 0x33.
- Repeat this for the next label: length of 0x02 followed by 0x6465 for de
- For the next label: length (4 octets) and the word pool results in data 0x04706f6f6c
- And so on: length (3 octets) and data are 0x036e7470 for ntp
- One time similar again: length (3 octets) and data are 0x036f7267 for org
- And now, very important: notification for end of hostname: 0x00
I repeat the whole hexadecimal characters (separated with space characters for a better understanding):
01 33 02 6465 04 706f6f6c 03 6e7470 03 6f7267 00 3 . d e . p o o l . n t p . o r g
This is the FQDN for the NTP suboption. Now we have to calculate the length of the suboption: 19 octets, 0x13 in hex.
My line for configuring the hostname 3.de.pool.ntp.org in FortiOS looks is this one:
(server) # set option2 56 '00030013013302646504706f6f6c036e7470036f726700' suboption type = FQDN | | suboption length = 19dec = 0x0013 | FQDN as described a few lines above
Do you want to publish two FQDNs? For example 2.de.pool.ntp.org and 3.de.pool.ntp.org? No problem, just do it:
(server) # set option2 56 '00030026013202646504706f6f6c036e7470036f726700013302646504706f6f6c036e7470036f726700'
But this wasn’t a good solution for my co-worker. He was not able to configure his client with the information I sent by DHCPv6. (Please don’t ask me for details!) Next, he asked me to send the SNTP address.
What about SNTP?
A fast search and I found the next interesting RFC with the name Simple Network Time Protocol (SNTP) Configuration Option for DHCPv6: RFC 4075. Looking on page 2:
The format of the Simple Network Time Protocol servers option is as shown below: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | OPTION_SNTP_SERVERS | option-len | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | SNTP server (IPv6 address) | | | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | | | SNTP server (IPv6 address) | | | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ : ... : +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ option-code: OPTION_SNTP_SERVERS (31) option-len: Length of the 'SNTP server' fields, in octets; it must be a multiple of 16 SNTP server: IPv6 address of SNTP server
SNTP is option code 31, FortiOS is calculating the option length itself, we just need the IPv6 unicast (only unicast?) address of the SNTP server
(server) # set option3 31 '20010db828000000000000000ac30123'
Do you want to publish two SNTP servers? No problem, do it like this:
(server) # set option3 31 '20010db828000000000000000ac2012320010db828000000000000000ac30123'
FortiOS and RouterOS Configuration
Today, the DHCP6 server configuration of the FortiGate looks like this:
config system dhcp6 server edit 641 set rapid-commit enable set lease-time 3602 set domain "example.com" set subnet 2001:db8:c:641::/64 set interface "IPv6only-01" set option1 56 '0001001020010db828000000000000000ac30123' set option2 56 '00030013013302646504706f6f6c036e7470036f726700' set option3 21 '20010db828000000000000000ac30123' config ip-range edit 1 set start-ip 2001:db8:c:641::6401 set end-ip 2001:db8:c:641::64ff next end set dns-server1 2001:db8:c:a53::53 set dns-server2 2001:db8:c:b53::53 next end
FortiOS in 6.0.14 allows only 3 custom values, I can’t publish more custom options. One more feature request to Fortinet: please allow more (maybe 9?) custom options here. :-)
In the same way, it’s possible to publish other information with DHCPv6, if the operating systems of the server allow the administrator to publish custom values in DHCPv6. In some operating systems, at the time of writing these lines, it’s the only way to send even basic information to a client system. For example, if you have to do a similar configuration in MikroTik RouterOS (tested with 6.48.5), it looks like this:
/ipv6 dhcp-server option add code=23 name=DNSa value="'2001:db8:c:a53::53'" add code=23 name=DNSb value="'2001:db8:c:b53::53'" add code=24 name=DNSsearch value="0x07'example'0x03'com'0x00" add code=31 name=SNTPboth value="'2001:db8:2800::ac2:123''2001:db8:2800::ac3:123'" add code=56 name=NTPaddr value="'000120010db8000c0b53000000000053'" /ipv6 dhcp-server add dhcp-option= DNSa,DNSb,DNSsearch,SNTPboth,NTPaddr interface=eth3 lease-time=29m name=IPv6test
In my experience, RouterOS is sending only requested options, other than FortiOS. In RouterOS, we don’t have to calculate the length of the DHCPv6 option, too.
Please always keep in mind: if you don’t send at least the other-flag in RA packets, most client systems don’t try to get DHCPv6 information from the network.
Photo by Bank Phrom on Unsplash.
Accessing IPv6-only Resources via Legacy IP: NAT46 on a FortiGate
In general, Network Address Translation (NAT) solves some problems but should be avoided wherever possible. It has nothing to do with security and is only a short-term solution on the way to IPv6. (Yes, I know, the last 20 years have proven that NAT is used everywhere every time. 😉) This applies to all kinds of NATs for IPv4 (SNAT, DNAT, PAT) as well as for NPTv6 and NAT66.
However, there are two types of NATs that do not only change the network addresses but do a translation between the two Internet Protocols, that is IPv4 <-> IPv6 and vice versa. Let’s focus on NAT46 this time. In which situations is it used and why? Supplemented by a configuration guide for the FortiGates, a downloadable PCAP and Wireshark screenshots.
NAT46 Concept
Basically, if you have IPv6-only servers (to avoid the unnecessary dual-stack burden for at least some of your infrastructure), but still want to have those servers accessible for IPv4-only clients, you have to use some kind of protocol translation somewhere. Either through reverse proxies or load balancers or through a NAT46 gateway like this:
Note that the NAT46 proxy, pronounced NAT-four-six by the way, does not necessarily have to be in the direct path between the client and server – it only has to be accessible by appropriate routes. Nevertheless, having one central firewall in place which does the job fits perfectly for small installations.
Furthermore, note that you need a hostname in the public DNS with an A record for the IPv4 address on your NAT46 gateway. But you don’t need a special DNS device such as a DNS64 box for NAT64 to work.
The basic concept of translating IP and ICMP between IPv4 and IPv6 aka “Stateless IP/ICMP Translation Algorithm (SIIT)” is described in RFC 7915. Funnily enough, the keyword “NAT46” is not said in the document at all.
Since I have at least one “server” (it’s a Raspi) running IPv6-only, I was able to test this NAT46 gateway. My True Random PSK Generator at https://random.weberlab.de/ has only an AAAA record, while I used https://random46.weberlab.de/ to access it via legacy IP:
NAT46 on a FortiGate
I’m not a big fan of FortiGate firewalls because they are neither reliable nor sound in many situations. However, they offer many cool and new features that other vendors don’t have. So let’s configure NAT46. I’m using a FortiWiFi-61E with FortiOS v7.0.9 for this setup.
At first, it requires a NAT object aka “Virtual IP” of type IPv4 which maps the (public) IPv4 address to the (internal) IPv6 address:
Second, you need an IPv6 Pool for the IPv6 source from which the firewall will initiate the internal IPv6 connections. Note the NAT46 checkbox. Also note that the “pool” is not really a pool but only capable of one or two IPv6 addresses. It took me a while to figure it out. All ranges I tested weren’t valid until I reduced the pool to one single IPv6 address. As you can see, I chose a very special-looking one: an IPv6 address with 4646 at the very end to spot it easily:
Finally, you need an appropriate Firewall Policy. Note that you must select the “NAT46” feature before you can select the destination which is the virtual IP object:
I’m a little scared when looking at this policy in the overview since the destination IPv6 address object says “any” though nothing was selected when creating the policy. AHH. It looks like this policy now allows all IPv6 destinations. Hopefully, it doesn’t. (That’s what I mean when stating that FortiGate firewalls are not that sound.)
Having a look at the Forward Traffic Log you can indeed see both Internet Protocols:
The appropriate CLI commands for this feature are: (You have to adjust some of them according to your needs/setup, e.g., the profile-group, the inspection-mode, and the like.)
config firewall vip edit "random46" set comment "Test NAT46" set extip 194.247.5.23 set nat44 disable set nat46 enable set extintf "wan1" set ipv6-mappedip 2001:470:1f0b:16b0:6986:b8d4:3649:9cbe next end config firewall ippool6 edit "SNAT46" set startip 2001:470:1f0b:16b0::4646 set endip 2001:470:1f0b:16b0::4646 set nat46 enable next end config firewall policy edit 41 set name "random46" set srcintf "wan1" set dstintf "internal" set action accept set nat46 enable set srcaddr "all" set dstaddr "random46" set srcaddr6 "all" set dstaddr6 "all" set schedule "always" set service "HTTP" "HTTPS" "PING" "PING6" set utm-status enable set inspection-mode proxy set profile-type group set profile-group "app-only" set logtraffic all set ippool enable set poolname6 "SNAT46" next end
Having a look at the sessions via CLI, you can see both ones, legacy IP and IPv6. Note the “peer” line for each IP in which the other IP is referenced. Nice! You have to use two different commands to show those sessions, though. (That’s what I mean when stating that FortiGate firewalls are not that sound.)
fg2 # diagnose sys session list session info: proto=6 proto_state=05 duration=1 expire=0 timeout=3600 flags=00000000 socktype=0 sockport=0 av_idx=0 use=3 origin-shaper= reply-shaper= per_ip_shaper= class_id=0 ha_id=0 policy_dir=0 tunnel=/ vlan_cos=0/255 state=log may_dirty npu f00 statistic(bytes/packets/allow_err): org=1343/11/1 reply=7375/9/1 tuples=2 tx speed(Bps/kbps): 1316/10 rx speed(Bps/kbps): 7230/57 orgin->sink: org pre->post, reply pre->post dev=6->20/20->6 gwy=194.247.5.23/194.247.4.1 hook=pre dir=org act=noop 85.215.94.29:53928->194.247.5.23:443(0.0.0.0:0) hook=post dir=reply act=noop 194.247.5.23:443->85.215.94.29:53928(0.0.0.0:0) peer=2001:470:1f0b:16b0::4646:53928->2001:470:1f0b:16b0:6986:b8d4:3649:9cbe:443 naf=1 hook=pre dir=org act=noop 2001:470:1f0b:16b0::4646:53928->2001:470:1f0b:16b0:6986:b8d4:3649:9cbe:443(:::0) hook=post dir=reply act=noop 2001:470:1f0b:16b0:6986:b8d4:3649:9cbe:443->2001:470:1f0b:16b0::4646:53928(:::0) pos/(before,after) 0/(0,0), 0/(0,0) misc=0 policy_id=41 pol_uuid_idx=606 auth_info=0 chk_client_info=0 vd=0 serial=018cd679 tos=ff/ff app_list=0 app=0 url_cat=0 rpdb_link_id=00000000 ngfwid=n/a npu_state=0x4040400 ofld-O npu info: flag=0x00/0x00, offload=0/0, ips_offload=0/0, epid=0/0, ipid=0/0, vlan=0x0000/0x0000 vlifid=0/0, vtag_in=0x0000/0x0000 in_npu=0/0, out_npu=0/0, fwd_en=0/0, qid=0/0 no_ofld_reason: ofld_fail_reason(kernel, drv): none/not-established, none(0)/none(0) npu_state_err=00/04 total session 1 fg2 # diagnose sys session6 list session6 info: proto=6 proto_state=15 duration=1 expire=0 timeout=3600 flags=00000000 sockport=0 socktype=0 use=3 origin-shaper= reply-shaper= per_ip_shaper= class_id=0 ha_id=0 policy_dir=0 tunnel=/ vlan_cos=0/0 state=log may_dirty npu app_valid statistic(bytes/packets/allow_err): org=1563/11/0 reply=7802/12/0 tuples=2 tx speed(Bps/kbps): 893/7 rx speed(Bps/kbps): 4458/35 orgin->sink: org pre->post, reply pre->post dev=20->23/23->20 hook=pre dir=org act=noop 2001:470:1f0b:16b0::4646:53928->2001:470:1f0b:16b0:6986:b8d4:3649:9cbe:443(:::0) hook=post dir=reply act=noop 2001:470:1f0b:16b0:6986:b8d4:3649:9cbe:443->2001:470:1f0b:16b0::4646:53928(:::0) peer=194.247.5.23:443->85.215.94.29:53928 naf=2 dst_mac=b8:27:eb:03:a0:ac misc=0 policy_id=41 auth_info=0 chk_client_info=0 vd=0 serial=00312c35 tos=ff/ff ips_view=0 app_list=2000 app=40568 url_cat=0 rpdb_link_id = 00000000 ngfwid=n/a npu_state=0x4041808 ofld-R npu info: flag=0x00/0x81, offload=0/0, ips_offload=0/0, epid=0/64, ipid=0/76, vlan=0x0000/0x0000 vlifid=0/76, vtag_in=0x0000/0x0000 in_npu=0/1, out_npu=0/1, fwd_en=0/0, qid=0/3 no_ofld_reason: ofld_fail_reason(kernel, drv): none/not-established, none(0)/none(0) npu_state_err=00/04
Finally, note that this setup required several other things around the mere network config which I have not shown here. That is:
- a hostname for random46.weberlab.de with
onlyat least an A record - ServerAliases on the apache2 config for the virtual host
- an adjusted rewrite condition to forward HTTP -> HTTPS for this ServerAlias
- running certbot again to have a valid X.509 certificate with this hostname in the subject alternative name field
Deeper Look on the Wire
I’ve captured some basic runs, that is: doing an HTTP request, getting redirected to HTTPS, as well as a ping aka echo-request. I captured on the client as well as on the server simultaneously and merged them later on. This capture is within the Ultimate PCAP already, but you can download it solely as well:
You can easily filter for ip or ipv6 to see only one of those Internet Protocols. Here they are side-by-side, looking at the SYN for the HTTP session:
As expected, the upper-layer protocol stuff is exactly the same after the NAT46 proxy, such as the TLS handshake with its ECDH client key exchange:
However, looking at ICMPv4 vs. ICMPv6 messages for echo-requests/-replies, the data portion looks a little different. ICMPv4 in this example used a timestamp which should be silently dropped according to RFC 7915, section 4.2. Looks like the FortiGate isn’t doing it that way but keeps the timestamp information within the data portion:
However, it’s working quite good. Nice! If you can spot any other differences between those translated protocols, please write a comment!
Fun Fact: NAT646
The other day I was on a german train using my T-Mobile tethering on my iPhone which gives perfect IPv6-native access, incl. DNS64/NAT64. Now, when surfing to this IPv4-only NAT46 domain, it eventually does a 646 translation. ;)
Of course, that would not have happened if the hostname would have had an AAAA record as well, which would be the case for real-world purposes in which your server hostname has an AAAA record (since it is IPv6-only) *and* the additional A record for the NAT46 translation.
Photo by Joshua Sortino on Unsplash.
Optimized NAT46 Config on a FortiGate
Johannes published a basic NAT46 configuration for a Fortigate firewall with FortiOS 7.0 some time ago. I run such a service (legacy IPv4 access to IPv6-only resources) since FortiOS 5.6, which means more than six years; lastly with FortiOS 6.4. It’s running for more than 100 servers without any other problems as we see them with IPv4 only or dual stack services.
But we weren’t happy with the basic configuration example by Fortinet. We wanted some NAT46 sample configuration with more details, that is: including the original source IPv4 address within the synthesized/SNATted IPv6 address. More in this post, after a short story about my way to a running nat46 configuration with port forwarding in FortiOS 7.2.x.
(Please note: this post is about NAT46, not NAT64, which is more common. :))
We operate a firewall which is doing NAT46 for many of our hosting services. All servers have IPv6 GUA (Global Unicast Addresses) and we are happy with the configuration. It looks like this:
Bear the Challenge and Update the FortiGate
Now, Fortinet support asked me, because of an issue we opened a ticket, to update my productive firewall (with the nat46 configuration!) to FortiOS 7.0 or 7.2 soon. I didn’t want to do this step since I read the first 7.0.1 release notes in summer 2021. This is because Fortinet changed the whole NAT64 and NAT46 configuration with policy46, vip46 and vip64 and so on. You will lose all of your configuration for IPv4-IPv6 translation features, not written there word by word, but as a result. They call this “simplify policy and routing configurations” at Fortinet in their Release Notes. And of course, as you already expect: the transfer of the nat46/nat64 configuration from a 6.x operating system to the new 7.x configuration is not a simple clicking around.
During the update process, there is no automatic transfer of the ip pools for nat64 and nat46, and no transfer of the nat64 and nat46 policies. After the first boot with 7.0.x, you only get a hint like this on the console:
The config file may contain errors, Please see details by the command 'diagnose debug config-error-log read'
As I always strongly recommend when you get this line: run the debug command before the next reboot! There you find a short hint about the ignored transfer of some configuration sections; in this case, as announced in the release notes:
fgtos70 # diagnose debug config-error-log read → → "set" "gui-nat46-64" "enable" @ root.system.settings:command parse error (error -61) → → "config" "firewall" "vip46" @ root:command parse error (error -61) → → "config" "firewall" "policy64" @ root:command parse error (error -61) → → "config" "firewall" "policy46" @ root:command parse error (error -61) → → "config" "system" "nat64" @ root:command parse error (error -61) fgtos70 #
In my case, the configuration had around 800 lines less… Ouch!
I didn’t want to wait and keep many services of our hosting offline for a longer time, so an additional reboot and an update from 7.0.13 to 7.2.6 was the next task.
With the help of a self-written search-and-replace script and some additional, manual changes, I had a text file with my whole nat46 configuration which I could import without any warnings or errors. BE HAPPY.
Optimized nat46 Configuration
Unfortunately, there are no examples with the well-known NAT46/SIIT IPv6-prefix 64:ff9b::/96 in many documentations. This 64:ff9b::/96 range is 32-bit wide (96 + 32 = 128 bit) and represents the whole (32-bit wide) IPv4 range in IPv6, used for NAT46 (and NAT64) and SIIT (see RFC 6052 and RFC 7755 and many others).
The IPv4 address is shown as a hexadecimal address in the last two hextets of the IPv6 address. This means: if you have the (example) IPv4 address 9.9.9.9 it converts to 64:ff9b::909:909 in IPv6 if you use it for NAT46/NAT64; IPv4 address 10.11.12.13 is translated to 64:ff9b::a0b:c0d. And one more example: IPv4 192.0.2.240 is 64:ff9b::c000:2f0. This is also called a synthesized IPv4 address when the IPv4 address is included in the 64:ff9b address.
If you create an ippool6 object in FortiOS with “startip 64:ff9b::” and “endip 64:ff9b::ffff:ffff“, (remember: the whole 32-bit IPv4 Internet!) and use this IPv6 range as a pool for the NAT46 (poolname6 in the policy for NAT46), the Fortigate will include the synthesized IPv4 source address as the IPv6 source address.
The IPv6-only service you operate behind your nat46 firewall can write this address to the log and may help you solve problems. This is the way IPv6-only servers can still provide the service for IPv4-only clients.
Configuration Example with NAT46 and Port-Forwarding
Here is my configuration example, with port-forwarding by a nat46 policy. First, we must create an IPv6 pool with a size of 32 bit to nat the (whole source) IPv4 address to a (destination) IPv6 address:
config firewall ippool6 edit "IPv6pool-for-nat46-with-full-32bit-IPv4-range" set startip 64:ff9b::0 set endip 64:ff9b::ffff:ffff set comments "common NAT46-NAT64 range" set nat46 enable next end
The next step is a service object. We need one virtual IP (vip) object for each port/service which has to be mapped from an IPv4 address to an IPv6 address. Here is the example for HTTP and HTTPS:
config firewall vip edit "nat46_testA:80" set extip 192.0.2.39 set nat44 disable set nat46 enable set extintf "v100" set portforward enable set ipv6-mappedip 2001:db8:de0:f270::200 set extport 80 set ipv6-mappedport 80 next edit "nat46_testA:443" set extip 192.0.2.39 set nat44 disable set nat46 enable set extintf "v100" set portforward enable set ipv6-mappedip 2001:db8:de0:f270::200 set extport 443 set ipv6-mappedport 443 next end
Now we create an object for port-forwarding within the NAT46 object. The IPv4-address is 192.0.2.39, IPv4 port is 22200 and the IPv6 port is 22 (default SSH):
config firewall vip edit "nat46_testA:22" set extip 192.0.2.39 set nat44 disable set nat46 enable set extintf "v100" set portforward enable set ipv6-mappedip 2001:db8:de0:f270::200 set extport 22200 set ipv6-mappedport 22 next end
For this type of port-forwarding an additional custom service object is required. This example is to forward port 22200, but not defined for which destination port:
config firewall service custom edit "nat46fwd22200tcp" set category "nat46forward" set comment "in use for nat46 port forward" set <strong>tcp-portrange 22200</strong> next end
Finally, we can combine all the created objects in policies:
config firewall policy edit 461 set srcintf "v100" set dstintf "v503" set action accept set nat46 enable set srcaddr "ExternalTestIPv4" set dstaddr "nat46_testA:80" "nat46_testA:443" set srcaddr6 "all" set dstaddr6 "all" set schedule "always" set service "HTTP" "HTTPS" set auto-asic-offload disable #just for better demonstration, set to enable for production set ippool enable set poolname6 "IPv6pool-for-nat46-with-full-32bit-IPv4-range" set comments "HTTP/HTTPS access policy" next edit 462 set srcintf "v100" set dstintf "v503" set action accept set nat46 enable set srcaddr "ExternalTestIPv4" set dstaddr "nat46_testA:22" set srcaddr6 "all" set dstaddr6 "all" set schedule "always" set service "nat46fwd22200tcp" "SSH" #we need both, the external tcp-port and the internal tcp-port set auto-asic-offload disable #just for better demonstration, set to enable for production set ippool enable set poolname6 "IPv6pool-for-nat46-with-full-32bit-IPv4-range" set comments "SSH access policy with port forwarding" next end
In policy 462 you can see the port-forwarding configuration. We need the service object nat46fwd22200tcp for the IPv4 mapped port and the SSH service object for the destination IPv6 service. Read the next section to see the internal process to understand, why the configuration needs this.
With this configuration, the IPv6-only daemon for SSH or HTTP/HTTPS can write the synthesized IPv4 address in the log and the marketing department can follow a client connection through the logs.
Until now I could not test the real effect of changing srcaddr6 and dstaddr6 from all to other values. Looking forward to get some feedback!
Bonus: Start of a Data Flow of a nat46 Connection
If you have a detailed look at the following lines (built-in tcpdump in FortiOS), you can see a new, hidden interface called naf.root (I expect root is the name of the vdom, it might be different if you have other vdoms) which is in use to help to translate IPv4 in IPv6 and back:
- In line 4 the test-ssh-client opens an IPv4 connection to port 22200 with a syn.
- In line 5 the syn packet is routed through the naf.root interface (still IPv4) and comes back in line 6 as an IPv6 packet to destination port 22 for SSH.
- The packet was sent to the vlan 503 (line 7) which uses a port channel (line 8) and leaves the unit on physical interface x3 (line 9).
All other lines are very similar and I am only publishing a part of the connection here. And I’m very sorry: I did not understand why there is a naf.root out in line 5 and why it is a naf.root in in line 6; same for all other lines with naf.root. Feedback is welcome!
fgtos7 # diagnose sniffer packet any 'host 192.0.2.46 or 64:ff9b::c000:22e' 4 500 l interfaces=[any] filters=[host 192.0.2.46 or 64:ff9b::c000:22e] 11:17:41.267612 v100 in 192.0.2.46.59172 → 192.0.2.39.22200: syn 160445192 11:17:41.267633 naf.root out 192.0.2.46.59172 → 192.0.2.39.22200: syn 160445192 11:17:41.267635 naf.root in 64:ff9b::c000:22e.59172 → 2001:db8:de0:f270::200.22: syn 160445192 11:17:41.267648 v503 out 64:ff9b::c000:22e.59172 → 2001:db8:de0:f270::200.22: syn 160445192 11:17:41.267649 PCdmz out 64:ff9b::c000:22e.59172 → 2001:db8:de0:f270::200.22: syn 160445192 11:17:41.267650 x3 out 64:ff9b::c000:22e.59172 → 2001:db8:de0:f270::200.22: syn 160445192 11:17:41.267846 v503 in 2001:db8:de0:f270::200.22 → 64:ff9b::c000:22e.59172: syn 4075988132 ack 160445193 11:17:41.267869 naf.root out 2001:db8:de0:f270::200.22 → 64:ff9b::c000:22e.59172: syn 4075988132 ack 160445193 11:17:41.267870 naf.root in 192.0.2.39.22200 → 192.0.2.46.59172: syn 4075988132 ack 160445193 11:17:41.267879 v100 out 192.0.2.39.22200 → 192.0.2.46.59172: syn 4075988132 ack 160445193 11:17:41.267880 PCwan out 192.0.2.39.22200 → 192.0.2.46.59172: syn 4075988132 ack 160445193 11:17:41.267880 x2 out 192.0.2.39.22200 → 192.0.2.46.59172: syn 4075988132 ack 160445193 11:17:41.279083 v100 in 192.0.2.46.59172 → 192.0.2.39.22200: ack 4075988133 11:17:41.279088 naf.root out 192.0.2.46.59172 → 192.0.2.39.22200: ack 4075988133 11:17:41.279089 naf.root in 64:ff9b::c000:22e.59172 → 2001:db8:de0:f270::200.22: ack 4075988133 11:17:41.279092 v503 out 64:ff9b::c000:22e.59172 → 2001:db8:de0:f270::200.22: ack 4075988133 11:17:41.279093 PCdmz out 64:ff9b::c000:22e.59172 → 2001:db8:de0:f270::200.22: ack 4075988133 11:17:41.279094 x3 out 64:ff9b::c000:22e.59172 → 2001:db8:de0:f270::200.22: ack 4075988133 11:17:41.284112 v100 in 192.0.2.46.59172 → 192.0.2.39.22200: psh 160445193 ack 4075988133 11:17:41.284116 naf.root out 192.0.2.46.59172 → 192.0.2.39.22200: psh 160445193 ack 4075988133 11:17:41.284117 naf.root in 64:ff9b::c000:22e.59172 → 2001:db8:de0:f270::200.22: psh 160445193 ack 4075988133 11:17:41.284120 v503 out 64:ff9b::c000:22e.59172 → 2001:db8:de0:f270::200.22: psh 160445193 ack 4075988133 11:17:41.284121 PCdmz out 64:ff9b::c000:22e.59172 → 2001:db8:de0:f270::200.22: psh 160445193 ack 4075988133 11:17:41.284122 x3 out 64:ff9b::c000:22e.59172 → 2001:db8:de0:f270::200.22: psh 160445193 ack 4075988133
Further sources: SIIT task of Tore Anderson on RIPE69 (Nov. 2014).
Photo by Ian Taylor on Unsplash.
How to install Palo Alto’s PAN-OS on a FortiGate
It happens occasionally that a customer has to choose between a Palo and a Forti. While I would always favour the Palo for good reasons, I can understand that the Forti is chosen for cost savings, for example.
Fortunately, there is a hidden way of installing PAN-OS, the operating system from Palo Alto Networks, on FortiGate hardware firewalls. Here’s how you can do it:
I’m using a Fortinet FortiGate FG-501E for this demo with (formerly) FortiOS v7.2.7. I’m upgrading it to PAN-OS 11.1.1.
The main step is to upload and reboot the FortiGate into an alternative image, that is: a PAN-OS image. For generic FortiGates, you must choose the KVM-based PAN-OS images. With the following CLI command on the FortiGate, you can download the image from an TFTP server and reboot into it:
execute restore image tftp PA-VM-KVM-11.1.1.qcow2 192.168.21.5
The whole process in my lab was as follows. Note that you have to acknowledge the upgrade to an “unsupported image”:
fg2 # execute restore image tftp PA-VM-KVM-11.1.1.qcow2 192.168.21.5 This operation will replace the current firmware version! Do you want to continue? (y/n)y Please wait... Connect to tftp server 192.168.21.5 ... ########################################################## Get image from tftp server OK. Warning: Upgrading to an unsupported image. Do you want to proceed? (y/n)y Checking new firmware integrity ... pass Please wait for system to restart.
After the reboot, you’re in the normal startup configuration of a Palo Alto firewall. –> Connect to it via the default IPv4 address of 192.168.1.1 with username:password of admin:admin.
In the dashboard, you can see the model and serial number, which are the ones from my FortiGate in this case:
Funnily enough, all those different interface names are used as well, that is:
Photo by Lindsay Henwood on Unsplash.