I got an email where someone asked whether I know how to change the link-local IPv6 addresses on a FortiGate similar to any other network/firewall devices. He could not find anything about this on the Fortinet documentation nor on Google.

Well, I could not find anything either. What’s up? It’s not new to me that you cannot really configure IPv6 on the FortiGate GUI, but even on the CLI I couldn’t find anything about changing this link-local IPv6 address from the default EUI-64 based one to a manually assigned one. Hence I opened a ticket at Fortinet. It turned out that you cannot *change* this address at all, but that you must *add* another LL address which will be used for the router advertisements (RA) after a reboot (!) of the firewall. Stupid design!

Again and again and again I am not happy at all with the IPv6 implementation on the FortiGates. Too many bugs and features missing, while everything is too complicated to configure. (Have a look at my Fortinet feature requests.) For the following tests I used a FortiGate FG-90D with firmware v5.6.5 build1600 (GA).

Before (Default Behaviour)

Before I touched the config the state of IPv6 was the following. Have a look at the “fg-trust” interface with its link-local address in line 12:

fg # diagnose ipv6 address list
dev=31 devname=vsys_fgfm flag=P scope=254 prefix=128 addr=::1
dev=29 devname=vsys_ha flag=P scope=254 prefix=128 addr=::1
dev=28 devname=fg-server flag=P scope=0 prefix=64 addr=2003:de:2016:220::1
dev=27 devname=fg-trust2 flag=P scope=0 prefix=64 addr=2003:de:2016:211::1
dev=26 devname=fg-trust flag=P scope=0 prefix=64 addr=2003:de:2016:210::1
dev=24 devname=root flag=P scope=254 prefix=128 addr=::1
dev=5 devname=wan1 flag=P scope=0 prefix=64 addr=2003:de:2016::2
dev=6 devname=wan2 flag=P scope=253 prefix=10 addr=fe80::a5b:eff:fea1:8360
dev=28 devname=fg-server flag=P scope=253 prefix=10 addr=fe80::a5b:eff:fea1:835e
dev=27 devname=fg-trust2 flag=P scope=253 prefix=10 addr=fe80::a5b:eff:fea1:835e
dev=26 devname=fg-trust flag=P scope=253 prefix=10 addr=fe80::a5b:eff:fea1:835e
dev=5 devname=wan1 flag=P scope=253 prefix=10 addr=fe80::a5b:eff:fea1:835f

The configuration at this point was:

config system interface
    edit "fg-trust"
        set vdom "root"
        set ip 192.168.210.1 255.255.255.0
        set allowaccess ping https ssh
        set role lan
        set snmp-index 5
        config ipv6
            set ip6-address 2003:de:2016:210::1/64
            set ip6-allowaccess ping https ssh
            set ip6-send-adv enable
            config ip6-prefix-list
                edit 2003:de:2016:210::/64
                    set autonomous-flag enable
                    set onlink-flag enable
                next
            end
        end
        set interface "internal1"
        set vlanid 210
    next
end

And a Linux machine got the following routing table, in which the default route had a gateway of

fe80::a5b:eff:fea1:835e

weberjoh@jw-vm05-Ubuntu-Test-3:~$ ip -6 r s
2003:de:2016:210::/64 dev ens32  proto kernel  metric 256  expires 2591699sec pref medium
fe80::/64 dev ens32  proto kernel  metric 256  pref medium
default via fe80::a5b:eff:fea1:835e dev ens32  proto ra  metric 1024  expires 1499sec pref medium

Configuration of the Link-Local Address

To add a link-local address you need the “config ip6-extra-addr” submenu. I added the quite simple

fe80::1/64

config system interface
edit fg-trust
config ipv6
config ip6-extra-addr
edit fe80::1/64
next
end
end
end

Now, in order to have the router advertisements sent from this newly created link-local address, you have to reboot the firewall! Come on Fortinet, you need a complete reboot for this?!? (Note that the support ticket told me to disable the “ip6-send-adv” before adding the LL address, and enabling it again after that. But this was not successful. At this point the RAs were still sent from the old EUI-64 based LL address.) Hence a reboot:

execute reboot

After

After this changes and the reboot the added link-local IPv6 was present (line 6):

fg # diagnose ipv6 address list
dev=31 devname=vsys_fgfm flag=P scope=254 prefix=128 addr=::1
dev=29 devname=vsys_ha flag=P scope=254 prefix=128 addr=::1
dev=28 devname=fg-server flag=P scope=0 prefix=64 addr=2003:de:2016:220::1
dev=27 devname=fg-trust2 flag=P scope=0 prefix=64 addr=2003:de:2016:211::1
dev=26 devname=fg-trust flag=SP scope=253 prefix=64 addr=fe80::1
dev=26 devname=fg-trust flag=P scope=0 prefix=64 addr=2003:de:2016:210::1
dev=24 devname=root flag=P scope=254 prefix=128 addr=::1
dev=5 devname=wan1 flag=P scope=0 prefix=64 addr=2003:de:2016::2
dev=6 devname=wan2 flag=P scope=253 prefix=10 addr=fe80::a5b:eff:fea1:8360
dev=28 devname=fg-server flag=P scope=253 prefix=10 addr=fe80::a5b:eff:fea1:835e
dev=27 devname=fg-trust2 flag=P scope=253 prefix=10 addr=fe80::a5b:eff:fea1:835e
dev=26 devname=fg-trust flag=P scope=253 prefix=10 addr=fe80::a5b:eff:fea1:835e
dev=5 devname=wan1 flag=P scope=253 prefix=10 addr=fe80::a5b:eff:fea1:835f

The complete configuration section for this interface looked like this:

config system interface
    edit "fg-trust"
        set vdom "root"
        set ip 192.168.210.1 255.255.255.0
        set allowaccess ping https ssh
        set role lan
        set snmp-index 5
        config ipv6
            set ip6-address 2003:de:2016:210::1/64
            set ip6-allowaccess ping https ssh
            config ip6-extra-addr
                edit fe80::1/64
                next
            end
            set ip6-send-adv enable
            config ip6-prefix-list
                edit 2003:de:2016:210::/64
                    set autonomous-flag enable
                    set onlink-flag enable
                next
            end
        end
        set interface "internal1"
        set vlanid 210
    next
end

And the Linux machine (after a reboot as well) got the correct next hop for its default route:

weberjoh@jw-vm05-Ubuntu-Test-3:~$ ip -6 r s
2003:de:2016:210::/64 dev ens32  proto kernel  metric 256  expires 2591853sec pref medium
fe80::/64 dev ens32  proto kernel  metric 256  pref medium
default via fe80::1 dev ens32  proto ra  metric 1024  expires 1653sec pref medium

Accordingly I could verify that the router advertisements were sent from my added link-local address

fe80::1

Competitors

That’s it. I am not happy with this approach from Fortinet in “changing” the link-local address. On other firewalls such as the Palo Alto Networks firewall you can clearly change the behaviour of the interface ID portion, and it even works without rebooting the firewall:

Cheers.

Featured image “Buy Local” by Mariano Mantel is licensed under CC BY-NC 2.0.

A security device such as a firewall should rely on NTP authentication to overcome NTP spoofing attacks. Therefore I am using NTP authentication on the FortiGate as well. As always, this so-called next-generation firewall has a very limited GUI while you need to configure all details through the CLI. I hate it, but that’s the way Fortinet is doing it. Furthermore the “set authentication” command is hidden unless you’re downgrading to NTPv3 (?!?) and it only supports MD5 rather than SHA-1. Not that “next-generation”!

Finally, you have no chance of knowing whether NTP authentication is working or not. I intentionally misconfigured some of my NTP keys which didn’t change anything in the NTP synchronization process while it should not work at all. Fail!

I am using a FortiGate FG-100D with FortiOS version v5.6.6 build1630 (GA). If you want to configure custom NTP servers you have to go through the CLI at all:

Then, configuring on the CLI, it took me quite some time to realize that the NTP authentication commands are completely hidden unless you are using NTPv3. Don’t know why this is a requirement at all since NTP authentication, of course, works with NTPv4 as well. And why isn’t this documented?

However, here are the commands I used to set up my three NTP servers with authentication:

config system ntp
    set ntpsync enable
    set type custom
    config ntpserver
        edit 1
            set server "ntp1.weberlab.de"
            set ntpv3 enable
            set authentication enable
            set key ENC 3xZj6FcN+Hg0ltR3BIQevJR3G+umyFrzN4mXeRRoxlTXM9HwKMMb1wo/t3AscNHjuuVkC58OTXP30U6rPce7RvGXfVfBA81s92JQ9duTKZv3be+N4KPiOM8EbTxYFN9irk/Kf8VuNDVZITsVGW+m6qaJewHycIk4wRypuHbA4s2/6GtL4ryYXHvksoB9bckwqOCqAw==
            set key-id 1
        next
        edit 2
            set server "ntp2.weberlab.de"
            set ntpv3 enable
            set authentication enable
            set key ENC wdqOtz4Q6HAe+RSzpGpx0nqZmRImT2gH3nwGStdDJn93EOLNv+kP5fxxjazyT+ArjRVWZVFYZnT/8fFqujwWP2GhyyALS4FdYPExaKTFAe/9m6DpIzTod1k8m8LbAJT0PnOG+8O3CgqLnhpnHm8v8Cp2oly/iORJ/ajVPQzvuvCuDzHX1fDQxsO4fJhFOVKlMgn/RQ==
            set key-id 2
        next
        edit 3
            set server "ntp3.weberlab.de"
            set ntpv3 enable
            set authentication enable
            set key ENC 0XXZMf6zshlsRxbElifoqXJXRxuM4Pti92wIYHq3pKKjvsHLuGPYx3wpqhylITZcabVS49X6EE6JwmHS22BTrCJLTVoO8TAvKaq/ZXHsawBLLme7WO7VQA5SumIx88q9VCj7Bd9aYKoevn4oBl5VRomY3I78DvoQ015nK8J+zReuWXWGL5LgL9qo3mM7j0YJTTGsgw==
            set key-id 3
        next
    end
end

In order to view any live values the

get system ntp

fg # get system ntp
ntpsync             : enable
type                : custom
syncinterval        : 1
ntpserver:
    == [ 1 ]
    id:     1
    == [ 2 ]
    id:     2
    == [ 3 ]
    id:     3
source-ip           : 0.0.0.0
server-mode         : disable

diagnose sys ntp status

fg # diagnose sys ntp status
synchronized: yes, ntpsync: enabled, server-mode: disabled

ipv6 server(ntp3.weberlab.de) 2003:de:2016:330::dcfb:123 -- reachable(0xff) S:1 T:8 selected
        server-version=3, stratum=1
        reference time is e03fd1c3.96a0308 -- UTC Fri Mar 22 21:27:31 2019
        clock offset is 0.019739 sec, root delay is 0.000000 sec
        root dispersion is 0.000153 sec, peer dispersion is 623 msec

ipv6 server(ntp2.weberlab.de) 2003:de:2016:330::6b5:123 -- reachable(0xff) S:0 T:7
        server-version=3, stratum=1
        reference time is e03fd1bb.d7cf8fae -- UTC Fri Mar 22 21:27:23 2019
        clock offset is 0.015482 sec, root delay is 0.000000 sec
        root dispersion is 0.001114 sec, peer dispersion is 504 msec

ipv6 server(ntp1.weberlab.de) 2003:de:2016:336::dcf7:123 -- reachable(0xff) S:0 T:7
        server-version=3, stratum=1
        reference time is e03fd18c.e184d3e8 -- UTC Fri Mar 22 21:26:36 2019
        clock offset is -0.023505 sec, root delay is 0.000000 sec
        root dispersion is 0.004059 sec, peer dispersion is 411 msec

Trivia: Failed Upgrade

Initially, I wanted to upgrade the FortiGate for this blogpost to its latest version from v5.6.6 to v5.6.8. Just a minor upgrade, right? However, this upgrade destroyed my VPN that was needed for the NTP servers. Even downgrading the version and restoring hasn’t worked. Just another example why I don’t really like those FortiGates. Details:

I just did a minor upgrade on a @Fortinet #FortiGate FG-100D (5.6.6 to 5.6.8) with the result that my configured IPsec VPN tunnel is completely lost. Fortinet at its best. #fail #Ihaveit pic.twitter.com/76pwBYyFMM

— Johannes Weber (@webernetz) March 22, 2019

Featured image “handypics August 2015 087” by PercyGermany is licensed under CC BY-NC-ND 2.0.

For some reason, I am currently using a FortiGate on a location that has no native IPv6 support. Uh, I don’t want to talk about that. ;) However, at least the FortiGate firewalls are capable of 6in4 tunnels. Hence I am using the IPv6 Tunnel Broker from Hurricane Electric again. Quite easy so far.

But note, as always: Though FortiGate supports these IPv6 features such as a 6in4 tunnel or stateful/-less DHCPv6 server, those features are NOT stable or well designed at all. I had many bugs and outages during my last years. Having “NAT enabled” on every new IPv6 policy is ridiculous. Furthermore, having independent security policies for legacy IP and IPv6 is obviously a really bad design. One single policy responsible for both Internet protocols is a MUST. Anyway, let’s look at the 6in4 tunnel:

Configuring this IPv6-in-IPv4 tunnel is quite easy since HE itself offers the configuration:

Of course, you need an internal layer 3 interface as well. That is, a complete configuration (6in4 tunnel, default route, inside interface with RDNSS) looks like that:

config system sit-tunnel
    edit "HE"
        set destination 216.66.80.30
        set ip6 2001:470:1f0a:16b0::2/64
        set source 194.247.4.10
    next
end
config router static6
    edit 1
        set device "HE"
    next
end
config system interface
    edit "internal"
        config ipv6
            set ip6-address 2001:470:1f0b:16b0::1/64
            set ip6-allowaccess ping https ssh
            set ip6-send-adv enable
            config ip6-prefix-list
                edit 2001:470:1f0b:16b0::/64
                    set autonomous-flag enable
                    set onlink-flag enable
                    set rdnss 2620:fe::fe
                    set dnssl "weberlab.de"
                next
            end
        end
    next
end

Finally, you need some IPv6 policy entries to permit traffic. Again, note that you MUST NOT select the NAT, which is stupidly pre-selected by Fortinet:

On every new IPv6 policy on a @Fortinet FortiGate firewall the default selection is "NAT". #fail #IPv6 pic.twitter.com/BfS4qKEfc6

— Johannes Weber (@webernetz) March 1, 2017

Stumbling Blocks

I am using a FortiGate FG-90D with FortiOS 5.6.8 build1672 (GA).

Note that this “HE” interface, as it is named in the example configuration above, is NOT visible in the interface section in the GUI:

while it IS visible in the routing section:

Honestly: Who is approving such decisions at Fortinet? This is not sound at all, isn’t it?

Verifying

You can have a look at the routing monitor to see the default route in place:

Some CLI commands are as follows. Getting information about the tunnel interface you can use this kind of hidden command:

fnsysctl ifconfig

fg2 # fnsysctl ifconfig HE
HE      Link encap:Unknown  HWaddr C2:F7:04:0A:00:00
        inet addr6: 2001:470:1f0a:16b0::2 prefixlen 64
        link-local6: fe80::c2f7:40a prefixlen 128
        UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1480  Metric:1
        RX packets:664858 errors:0 dropped:0 overruns:0 frame:0
        TX packets:1015185 errors:0 dropped:0 overruns:0 carrier:0
        collisions:0 txqueuelen:0
        RX bytes:179754139 (171.4 MB)  TX bytes:114587175 (109.3 MB)

IPv6 routing table:

fg2 # get router info6 routing-table
IPv6 Routing Table
Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF,
       IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       I - IS-IS, B - BGP
       * - candidate default

Timers: Uptime

S*      ::/0 [10/0] via ::, HE, 01w2d12h
C       ::1/128 via ::, root, 04w2d19h
C       2001:470:1f0a:16b0::/64 via ::, HE, 01w2d12h
C       2001:470:1f0b:16b0::/64 via ::, internal, 01w2d11h
C       fe80::/10 via ::, internal, 01w2d11h
C       fe80::c2f7:40a/128 via ::, HE, 01w2d12h

And some basic network connectivity test, aka ping:

fg2 # execute ping6-options reset

fg2 # execute ping6 weberblog.net
PING weberblog.net(2a01:488:42:1000:50ed:8588:8a:c570) 56 data bytes
64 bytes from 2a01:488:42:1000:50ed:8588:8a:c570: icmp_seq=1 ttl=56 time=9.13 ms
64 bytes from 2a01:488:42:1000:50ed:8588:8a:c570: icmp_seq=2 ttl=56 time=11.4 ms
64 bytes from 2a01:488:42:1000:50ed:8588:8a:c570: icmp_seq=3 ttl=56 time=9.57 ms
64 bytes from 2a01:488:42:1000:50ed:8588:8a:c570: icmp_seq=4 ttl=56 time=10.3 ms
64 bytes from 2a01:488:42:1000:50ed:8588:8a:c570: icmp_seq=5 ttl=56 time=10.1 ms

--- weberblog.net ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss, time 4045ms
rtt min/avg/max/mdev = 9.136/10.145/11.487/0.806 ms

That’s it. Thanks for watching. ;) Don’t forget to hit the subscribe button.

Featured image “Make It Count” by Mr. Nixter is licensed under CC BY-NC 2.0.

This is a really nice feature: you can run iperf3 directly on a FortiGate to speed-test your network connections. It’s basically an iperf3 client. Using some public iperf servers you can test your Internet bandwidth; using some internal servers you can test your own routed/switched networks, VPNs, etc. However, the maximum throughput for the test is CPU dependent. So please be careful when interpreting the results. Here we go:

I am using a FortiGate FG-90D with FortiOS v6.0.10. I don’t know whether this iperf implementation is present on all FortiOS releases on all FortiGates. On mine, it is. ;) Here is more information about iperf3.

You have to set at least the iperf client and server interface on the FortiGate in order to run it. The server interface is NOT used when testing the bandwidth to an external server. However, you have to specify it, otherwise, you’re getting an error. (You can test internal paths within the FortiGate <- that’s why you have to set the client and server interface. However, I don’t know whether these tests will have any value.) To test your ISP connection, you have to find a public iperf server, e.g., here: https://iperf.cc/. The FortiGate implementation of iperf does not accept hostnames, but only IP addresses.

Test, Test, Test

A basic run looks like this. Using port 5200 (in my example) and testing in both directions:

diagnose traffictest client-intf wan1
diagnose traffictest server-intf wan1
diagnose traffictest port 5200
diagnose traffictest run -c 213.209.106.95
diagnose traffictest run -R -c 213.209.106.95

That is:

fg2 # diagnose traffictest client-intf wan1
client-intf:    wan1

fg2 # diagnose traffictest server-intf wan1
server-intf:    wan1

fg2 # diagnose traffictest port 5200
port:   5200

fg2 # diagnose traffictest show
server-intf:    wan1
client-intf:    wan1
port:   5200
proto:  TCP

fg2 # diagnose traffictest run -c 213.209.106.95
Connecting to host 213.209.106.95, port 5200
[  8] local 194.247.4.10 port 1489 connected to 213.209.106.95 port 5200
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  8]   0.00-1.03   sec  15.6 MBytes   127 Mbits/sec    0    359 KBytes
[  8]   1.03-2.00   sec  16.2 MBytes   140 Mbits/sec    0    410 KBytes
[  8]   2.00-3.05   sec  18.8 MBytes   150 Mbits/sec    0    385 KBytes
[  8]   3.05-4.01   sec  16.2 MBytes   143 Mbits/sec    0    392 KBytes
[  8]   4.01-5.06   sec  18.8 MBytes   149 Mbits/sec    0    380 KBytes
[  8]   5.06-6.04   sec  16.2 MBytes   140 Mbits/sec    0    389 KBytes
[  8]   6.04-7.04   sec  17.5 MBytes   146 Mbits/sec    0    387 KBytes
[  8]   7.04-8.05   sec  16.2 MBytes   135 Mbits/sec    0    404 KBytes
[  8]   8.05-9.06   sec  17.5 MBytes   145 Mbits/sec    0    386 KBytes
[  8]   9.06-10.06  sec  17.5 MBytes   148 Mbits/sec    0    386 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  8]   0.00-10.06  sec   171 MBytes   142 Mbits/sec    0             sender
[  8]   0.00-10.06  sec   171 MBytes   142 Mbits/sec                  receiver

iperf Done.
iperf3: interrupt - the server has terminated

fg2 # diagnose traffictest run -R -c 213.209.106.95
Connecting to host 213.209.106.95, port 5200
Reverse mode, remote host 213.209.106.95 is sending
[  8] local 194.247.4.10 port 1491 connected to 213.209.106.95 port 5200
[ ID] Interval           Transfer     Bandwidth
[  8]   0.00-1.00   sec  8.02 MBytes  67.0 Mbits/sec
[  8]   1.00-2.00   sec  8.13 MBytes  68.4 Mbits/sec
[  8]   2.00-3.00   sec  8.27 MBytes  69.5 Mbits/sec
[  8]   3.00-4.00   sec  8.19 MBytes  68.7 Mbits/sec
[  8]   4.00-5.00   sec  8.51 MBytes  71.2 Mbits/sec
[  8]   5.00-6.00   sec  8.46 MBytes  71.1 Mbits/sec
[  8]   6.00-7.00   sec  8.08 MBytes  67.7 Mbits/sec
[  8]   7.00-8.02   sec  8.32 MBytes  68.7 Mbits/sec
[  8]   8.02-9.03   sec  8.32 MBytes  69.1 Mbits/sec
[  8]   9.03-10.01  sec  7.96 MBytes  68.1 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  8]   0.00-10.01  sec  83.7 MBytes  70.1 Mbits/sec    0             sender
[  8]   0.00-10.01  sec  82.4 MBytes  69.0 Mbits/sec                  receiver

iperf Done.
iperf3: interrupt - the server has terminated

Other useful commands are:

diagnose traffictest show
diagnose traffictest run -v
diagnose traffictest run -h

They show the current configuration on the FortiGate (first one) and some more details about iperf itself:

fg2 # diagnose traffictest show
server-intf:    wan1
client-intf:    wan1
port:   5200
proto:  TCP

fg2 # diagnose traffictest run -v
iperf 3.0.9

fg2 # diagnose traffictest run -h
  -f, --format    [kmgKMG]  format to report: Kbits, Mbits, KBytes, MBytes
  -i, --interval  #         seconds between periodic bandwidth reports
  -F, --file name           xmit/recv the specified file
  -A, --affinity n/n,m      set CPU affinity
  -V, --verbose             more detailed output
  -J, --json                output in JSON format
  -d, --debug               emit debugging output
  -v, --version             show version information and quit
  -h, --help                show this message and quit
  -b, --bandwidth #[KMG][/#] target bandwidth in bits/sec (0 for unlimited)
                            (default 1 Mbit/sec for UDP, unlimited for TCP)
                            (optional slash and packet count for burst mode)
  -t, --time      #         time in seconds to transmit for (default 10 secs)
  -n, --bytes     #[KMG]    number of bytes to transmit (instead of -t)
  -k, --blockcount #[KMG]   number of blocks (packets) to transmit (instead of -t or -n)
  -l, --len       #[KMG]    length of buffer to read or write
                            (default 128 KB for TCP, 8 KB for UDP)
  -P, --parallel  #         number of parallel client streams to run
  -R, --reverse             run in reverse mode (server sends, client receives)
  -w, --window    #[KMG]    TCP window size (socket buffer size)
  -C, --linux-congestion <algo>  set TCP congestion control algorithm (Linux only)
  -M, --set-mss   #         set TCP maximum segment size (MTU - 40 bytes)
  -N, --nodelay             set TCP no delay, disabling Nagle's Algorithm
  -4, --version4            only use IPv4
  -6, --version6            only use IPv6
  -S, --tos N               set the IP 'type of service'
  -L, --flowlabel N         set the IPv6 flow label (only supported on Linux)
  -Z, --zerocopy            use a 'zero copy' method of sending data
  -O, --omit N              omit the first n seconds
  -T, --title str           prefix every output line with this string
  --get-server-output       get results from server

[KMG] indicates options that support a K/M/G suffix for kilo-, mega-, or giga-

Caveats, Caveats, Caveats

Unfortunately, here are some (major!) caveats: At first, the iperf implementation on the FortiGate is heavily CPU related. My FG-90D has a 1 Gbps uplink to the Internet. Running iperf3 on the Forti reveals only about 150 Mbps (see above), while the CPU usage immediately peaked at 100 %. Ouch:

Testing my ISP speed *through* the FortiGate from a Linux system behind it, iperf3 showed about 900 Mbps, while the CPU usage on the Forti stayed by about 3-5 %. Following is the bandwidth widget from the Forti during my tests:

Certainly this behavior is different on other FortiGates hardware. To be fair, my FG-90D is not the newest nor the biggest model. I have tested the traffictest feature on a FG-501E with FortiOS v6.2.5 which was able to receive 900 Mbps while only one out of eight cores peaked at about 25 %.

Second caveat: it’s not working with IPv6, but only with legacy IP. :(

fg2 # diagnose traffictest run -c 2a02:2028:ff00::f9:2
iperf3: error - unable to connect to server: Invalid argument
iperf3: interrupt - the server has terminated

fg2 # diagnose traffictest run -6 -c 2a02:2028:ff00::f9:2
iperf3: error - unable to connect to server:
iperf3: interrupt - the server has terminated

Conclusion

Uh, that’s hard. In theory, this is a cool hidden feature. If you’re keeping track of your CPU usage you can probably use it for getting realistic results. Especially on links with small bandwidth.

However, if you really want to test your big ISP connection, you shouldn’t rely on it. Or to say it differently: If you’re getting the expected results with iperf on the Forti, you’re ok. If not, you don’t know why. ;(

PS: Happy Birthday Nicolai!

Photo by Harley-Davidson on Unsplash.

The NTP Pool is a volunteer organization that provides time synchronization service to hundreds of millions of computers worldwide. A typical client might query a particular NTP Pool server ~10-60 times/hour. Wikipedia lists some abusive clients that far exceeded the normal rate. This wastes NTP server resources, may interfere with other clients, and can trigger DDoS protections. In late 2019, a software update made some FortiGate firewalls very unfriendly to the NTP Pool.

Fortinet is a multinational corporation that produces computer security devices, including the FortiGate firewall. The firewall product documentation describes how to use the NTP Pool for time synchronization, e.g., by using:

0.europe.pool.ntp.org
1.europe.pool.ntp.org
2.europe.pool.ntp.org

DNS will resolve 0.europe.pool.ntp.org into one of the hundreds of NTP servers in the NTP Pool. Let’s say that at firewall startup 1.2.3.4 was the NTP server selected by DNS. What happens if sometime later the administrator of 1.2.3.4 decides to stop participating in the NTP Pool and shuts down the public NTP service? We were told that older FortiGate NTP software did not handle that condition gracefully. A software update, FortiOS 6.2.3, was issued in December 2019 that periodically reissued the DNS lookup. In our example, if 0.europe.pool.ntp.org no longer resolved into 1.2.3.4 some re-initialization was done by the new software. That new code had a serious flaw. Bursts of NTP requests with rates that sometimes exceeded 20,000/sec were sent for up to 10 seconds duration. As DNS resolution changed, which is common for the NTP pools, a single FortiGate firewall would send bursts to different NTP pool servers.

A burst from a single FortiGate firewall is shown below

In early 2020, NTP pool operators noticed these distinctive bursts. They sometimes represented over 20 % of the total NTP traffic on a single server! Diagnostic work by several NTP Pool volunteers identified FortiGate firewalls as the source of these bursts. In April 2020, we contacted Fortinet support who acknowledged the bug and said that the next software release containing a fix would soon be deployed. However, the new software had unrelated bugs that prevented it from being widely deployed. On August 22, 2020, FortiOS 6.2.5 was released to eliminate the bursts.

The fix for FortiGate administrators is simple:

• Do not use FortiOS release 6.2.3, or
• if using release 6.2.3, do not use the NTP Pool.

FortiGate notified a handful of customers of the problem. However, as of this writing (November 2020), the NTP Pool is still receiving frequent NTP bursts from hundreds (at least) of FortiGate firewalls. While Fortinet was responsive in email and phone discussions, on the whole, their corrective actions were disappointing. Hopefully, FortiGate administrators will read this note and take corrective action as appropriate.

The diagnostic work was done by Miroslav Lichvar, Hal Murray, and Steven Sommars with contributions from several other NTP Pool volunteers.

Photo by Heather Zabriskie on Unsplash.

More than 6 years ago (!) I published a tutorial on how to set up an IPsec VPN tunnel between a FortiGate firewall and a Cisco ASA. As time flies by, ASA is now able to terminate route-based VPN tunnels (which is great!), we have IKEv2 running everywhere and enhanced security proposals. Hence, it’s time for an update:

My Setup

This is my setup for this tutorial: (Yes, public IPv4 addresses behind the Forti.)

I am using a Fortinet FortiWiFi FWF-61E with FortiOS v6.2.5 build1142 (GA) and a Cisco ASA 5515 with version 9.12(3)12 and ASDM 7.14(1). These are the VPN parameters:

• Route-based VPN, that is: numbered tunnel interface and real route entries for the network(s) to the other side. But no proxy-IDs aka traffic selection aka crypto map. Thank goodness for that.
• The tunnel interface on the Forti is added during the VPN setup automatically. However, you have to set the IP address on the tunnel interface manually after that. The static route on the ASA needs an IP address as the gateway.
• IKEv2 (no distinction anymore between main or aggressive mode as with IKEv1)
• PSK: 30 chars alphanumeric, generated with a password generator! (ref)
• IKE crypto/policies:
  • Diffie-Hellman group 21
  • AES-256-GCM
  • SHA-512 (you could use SHA-256 if you like)
  • 8 hours
• IPsec crypto/proposals/transform sets:
  • AES-256-GCM
  • SHA-512 (again, you can use SHA-256 as well)
  • Diffie-Hellman group 21
  • 1 hour
• No NAT between the internal networks (of course not ;))!

FortiGate

You can do the configuration through the GUI:

or through the CLI: (incl. the zone commands <- can be omitted if you aren’t using zones)

config system interface
    edit "asa"
        set vdom "root"
        set ip 10.1.37.1 255.255.255.255
        set allowaccess ping
        set type tunnel
        set remote-ip 10.1.37.2 255.255.255.252
        set interface "wan1"
    next
end
config system zone
    edit "s2s-vpns"
        set interface "asa"
    next
end
config vpn ipsec phase1-interface
    edit "asa"
        set interface "wan1"
        set ike-version 2
        set keylife 28800
        set peertype any
        set net-device enable
        set proposal aes256gcm-prfsha512
        set dhgrp 21
        set nattraversal disable
        set remote-gw 185.23.77.7
        set psksecret ThisIsThePreSharedKey
    next
end
config vpn ipsec phase2-interface
    edit "asa"
        set phase1name "asa"
        set proposal aes256gcm
        set dhgrp 21
        set keylifeseconds 3600
    next
end
config router static
    edit 5
        set dst 172.16.37.0 255.255.255.0
        set device "asa"
    next
end

Cisco ASA

Same on the ASA, either via the “GUI”:

or via classical CLI commands: (The ACL is omitted.)

interface Tunnel1
 nameif fg2
 ip address 10.1.37.2 255.255.255.252
 tunnel source interface outside
 tunnel destination 194.247.4.10
 tunnel mode ipsec ipv4
 tunnel protection ipsec profile aes256gcm-sha512-dh21-3600s
!
route fg2 194.247.5.0 255.255.255.224 10.1.37.1 1
!
crypto ipsec ikev2 ipsec-proposal aes256gcm-sha512
 protocol esp encryption aes-gcm-256
 protocol esp integrity sha-512
crypto ipsec profile aes256gcm-sha512-dh21-3600s
 set ikev2 ipsec-proposal aes256gcm-sha512
 set pfs group21
 set security-association lifetime seconds 3600
crypto ikev2 policy 1
 encryption aes-gcm-256
 integrity null
 group 21
 prf sha512
 lifetime seconds 28800
!
group-policy 194.247.4.10 internal
group-policy 194.247.4.10 attributes
 vpn-tunnel-protocol ikev2
tunnel-group 194.247.4.10 type ipsec-l2l
tunnel-group 194.247.4.10 general-attributes
 default-group-policy 194.247.4.10
tunnel-group 194.247.4.10 ipsec-attributes
 ikev2 remote-authentication pre-shared-key ThisIsThePreSharedKey
 ikev2 local-authentication pre-shared-key ThisIsThePreSharedKey

Monitoring

Some screenshots from the FortiGate:

as well as CLI outputs:

fg2 # get vpn ike gateway asa

vd: root/0
name: asa
version: 2
interface: wan1 6
addr: 194.247.4.10:500 -> 185.23.77.7:500
created: 3158587s ago
IKE SA  created: 1/111  established: 1/111  time: 0/3/100 ms
IPsec SA  created: 1/973  established: 1/973  time: 0/0/100 ms

  id/spi: 2040 7be16624b6a980a3/b107958ab150a4fb
  direction: initiator
  status: established 23585-23585s ago = 10ms
  proposal: unknown-256-unknown
  SK_ei: c301af190feb89e7-e89076489227f77e-73a80ecd3692c0c7-925c73a84a30c063-618eb9af
  SK_er: 5362b4bc6103b45f-776a3e817a61026f-75b7cd0220fb8d70-05f32a71240799e6-f1441bb6
  SK_ai:
  SK_ar:
  lifetime/rekey: 28800/4914
  DPD sent/recv: 00000000/00000000

fg2 #
fg2 #
fg2 # get vpn ipsec tunnel name asa

gateway
  name: 'asa'
  type: route-based
  local-gateway: 194.247.4.10:0 (static)
  remote-gateway: 185.23.77.7:0 (static)
  mode: ike-v2
  interface: 'wan1' (6)
  rx  packets: 110976  bytes: 145943836  errors: 0
  tx  packets: 64092  bytes: 3004962  errors: 0
  dpd: on-demand/negotiated  idle: 20000ms  retry: 3  count: 0
  selectors
    name: 'asa'
    auto-negotiate: disable
    mode: tunnel
    src: 0:0.0.0.0/0.0.0.0:0
    dst: 0:0.0.0.0/0.0.0.0:0
    SA
      lifetime/rekey: 3600/2171
      mtu: 1446
      tx-esp-seq: 27
      replay: enabled
      qat: 0
      inbound
        spi: 15ad154f
        enc:  aes-gc  e1831416107c6ca5c1d6da624269ba4e21b7d45c95d5a16da8c0f9200b598ebbab76f5b9
        auth:   null
      outbound
        spi: 9573f1de
        enc:  aes-gc  3d6e5ab8c1ac1de02a230095d76778dd5b88aeeff7dfae8b25df26c265bdec56710d040e
        auth:   null
      NPU acceleration: none

fg2 #
fg2 #
fg2 # diagnose vpn tunnel list name asa
list ipsec tunnel by names in vd 0
------------------------------------------------------
name=asa ver=2 serial=3 194.247.4.10:0->185.23.77.7:0 dst_mtu=1500
bound_if=6 lgwy=static/1 tun=intf/0 mode=auto/1 encap=none/536 options[0218]=npu create_dev frag-rfc  accept_traffic=1 overlay_id=0

proxyid_num=1 child_num=0 refcnt=14 ilast=12 olast=12 ad=/0
stat: rxp=110977 txp=64094 rxb=145943972 txb=3005118
dpd: mode=on-demand on=1 idle=20000ms retry=3 count=0 seqno=440
natt: mode=none draft=0 interval=0 remote_port=0
proxyid=asa proto=0 sa=1 ref=3 serial=2
  src: 0:0.0.0.0/0.0.0.0:0
  dst: 0:0.0.0.0/0.0.0.0:0
  SA:  ref=3 options=10226 type=00 soft=0 mtu=1446 expire=2114/0B replaywin=1024
       seqno=29 esn=0 replaywin_lastseq=00000014 itn=0 qat=0 hash_search_len=1
  life: type=01 bytes=0/0 timeout=3298/3600
  dec: spi=15ad154f esp=aes-gcm key=36 e1831416107c6ca5c1d6da624269ba4e21b7d45c95d5a16da8c0f9200b598ebbab76f5b9
       ah=null key=0
  enc: spi=9573f1de esp=aes-gcm key=36 3d6e5ab8c1ac1de02a230095d76778dd5b88aeeff7dfae8b25df26c265bdec56710d040e
       ah=null key=0
  dec:pkts/bytes=20/1600, enc:pkts/bytes=40/5360
  npu_flag=20 npu_rgwy=185.23.77.7 npu_lgwy=194.247.4.10 npu_selid=5 dec_npuid=0 enc_npuid=0

fg2 #
fg2 #
fg2 # get router info routing-table all

Routing table for VRF=0
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
       O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
       * - candidate default

S*      0.0.0.0/0 [10/0] via 194.247.4.1, wan1
C       10.1.37.0/30 is directly connected, asa
C       10.1.37.1/32 is directly connected, asa
S       172.16.37.0/24 [10/0] via 10.1.37.2, asa
S       192.168.11.0/24 [10/0] is directly connected, ssg5-weberhom
S       193.24.227.224/27 [10/0] is directly connected, pa
C       194.247.4.0/27 is directly connected, wan1
C       194.247.5.0/27 is directly connected, internal


fg2 #

And some screenshots from the ASA: (the third one showing the logs after a manual “logout”)

as well as CLI outputs:

asa# show crypto ikev2 sa detail

IKEv2 SAs:

Session-id:16, Status:UP-ACTIVE, IKE count:1, CHILD count:1

Tunnel-id Local                                               Remote                                                  Status         Role
1219040189 185.23.77.7/500                                     194.247.4.10/500                                         READY    INITIATOR
      Encr: AES-GCM, keysize: 256, Hash: N/A, DH Grp:21, Auth sign: PSK, Auth verify: PSK
      Life/Active Time: 28800/298 sec
      Session-id: 16
      Status Description: Negotiation done
      Local spi: E82116F37CF38D12       Remote spi: 3D48FE4CB448BA6B
      Local id: 185.23.77.7
      Remote id: 194.247.4.10
      Local req mess id: 26             Remote req mess id: 0
      Local next mess id: 26            Remote next mess id: 0
      Local req queued: 26              Remote req queued: 0
      Local window: 1                   Remote window: 1
      DPD configured for 10 seconds, retry 2
      NAT-T is not detected
      IKEv2 Fragmentation Configured MTU: 576 bytes, Overhead: 28 bytes, Effective MTU: 548 bytes
Child sa: local selector  0.0.0.0/0 - 255.255.255.255/65535
          remote selector 0.0.0.0/0 - 255.255.255.255/65535
          ESP spi in/out: 0x5f713ed2/0x15ad1552
          AH spi in/out: 0x0/0x0
          CPI in/out: 0x0/0x0
          Encr: AES-GCM, keysize: 256, esp_hmac: N/A
          ah_hmac: None, comp: IPCOMP_NONE, mode tunnel
Parent SA Extended Status:
      Delete in progress: FALSE
      Marked for delete: FALSE
asa#
asa#
asa# show crypto ipsec sa peer 194.247.4.10 detail
peer address: 194.247.4.10
    Crypto map tag: __vti-crypto-map-5-0-1, seq num: 65280, local addr: 185.23.77.7

      local ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
      remote ident (addr/mask/prot/port): (0.0.0.0/0.0.0.0/0/0)
      current_peer: 194.247.4.10


      #pkts encaps: 29, #pkts encrypt: 29, #pkts digest: 29
      #pkts decaps: 45, #pkts decrypt: 45, #pkts verify: 45
      #pkts compressed: 0, #pkts decompressed: 0
      #pkts not compressed: 29, #pkts comp failed: 0, #pkts decomp failed: 0
      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
      #TFC rcvd: 0, #TFC sent: 0
      #Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
      #pkts no sa (send): 0, #pkts invalid sa (rcv): 0
      #pkts encaps failed (send): 0, #pkts decaps failed (rcv): 0
      #pkts invalid prot (rcv): 0, #pkts verify failed: 0
      #pkts invalid identity (rcv): 0, #pkts invalid len (rcv): 4009213712
      #pkts invalid pad (rcv): 0,
      #pkts invalid ip version (send): 0, #pkts invalid ip version (rcv): 0
      #pkts invalid len (send): 0, #pkts invalid len (rcv): 0
      #pkts invalid ctx (send): 0, #pkts invalid ctx (rcv): 0
      #pkts invalid ifc (send): 0, #pkts invalid ifc (rcv): 0
      #pkts failed (send): 0, #pkts failed (rcv): 0
      #pkts replay rollover (send): 0, #pkts replay rollover (rcv): 0
      #pkts replay failed (rcv): 0
      #pkts min mtu frag failed (send): 0, #pkts bad frag offset (rcv): 0
      #pkts internal err (send): 0, #pkts internal err (rcv): 0

      local crypto endpt.: 185.23.77.7/500, remote crypto endpt.: 194.247.4.10/500
      path mtu 1500, ipsec overhead 55(36), media mtu 1500
      PMTU time remaining (sec): 0, DF policy: copy-df
      ICMP error validation: disabled, TFC packets: disabled
      current outbound spi: 15AD1552
      current inbound spi : 5F713ED2

    inbound esp sas:
      spi: 0x5F713ED2 (1601257170)
         SA State: active
         transform: esp-aes-gcm-256 esp-null-hmac no compression
         in use settings ={L2L, Tunnel, PFS Group 21, IKEv2, VTI, }
         slot: 0, conn_id: 51, crypto-map: __vti-crypto-map-5-0-1
         sa timing: remaining key lifetime (kB/sec): (3962873/3231)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
          0xAAAAAAAA 0xAAAAB8AA
    outbound esp sas:
      spi: 0x15AD1552 (363664722)
         SA State: active
         transform: esp-aes-gcm-256 esp-null-hmac no compression
         in use settings ={L2L, Tunnel, PFS Group 21, IKEv2, VTI, }
         slot: 0, conn_id: 51, crypto-map: __vti-crypto-map-5-0-1
         sa timing: remaining key lifetime (kB/sec): (4193275/3231)
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap:
          0x00000000 0x00000001

asa#
asa#
asa# show route static

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2, V - VPN
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route, + - replicated route
Gateway of last resort is 185.23.77.1 to network 0.0.0.0

S*       0.0.0.0 0.0.0.0 [1/0] via 185.23.77.1, outside
S        194.247.5.0 255.255.255.224 [1/0] via 10.1.37.1, fg2

asa#

PS: Sorry for being legacy IP only this time. ;(

Photo by Casey Horner on Unsplash.

As we have just set up a TLS capable syslog server, let’s configure a Fortinet FortiGate firewall to send syslog messages via an encrypted channel (TLS). Let’s go:

I am using a Fortinet FortiGate (FortiWiFi) FWF-61E with FortiOS v6.4.7 build1911 (GA) for this tutorial. My syslog-ng server with version 3.13.2 is running on Ubuntu 18.04.6 LTS.

You cannot configure any syslog server details (rather than the address itself) via the GUI on this so-called “Next Generation Firewall”. Log & Report -> Log Settings -> Remote Logging and Archiving:

Hence we have to use the CLI:

config log syslogd setting
    set status enable
    set server "syslog2.weberlab.de"
    set mode reliable
    set port 6514
    set enc-algorithm high
end

Some notes:

• You can’t “set enc-algorithm high” unless you have “set mode reliable”.
• You don’t have to “set port 6514” manually because it is set automatically by doing the “set enc-algorithm high” command.
• Using an FQDN as the server (as I did in the listing), the FortiGate will use legacy IP though an AAAA record is present. If you want to use IPv6 you must use an IPv6 address here.
• ;)
• Official Fortinet CLI reference

If your syslog server uses a self-signed or untrusted certificate it won’t work right now. This is good from a security point of view. You’ll see the following syslog messages on your syslog server: “tlsv1 alert unknown ca”:

weberjoh@nb17-lx2:~$ tail -f /var/log/syslog
Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: Syslog connection accepted; fd='96', client='AF_INET6([2001:470:1f0b:16b0::1]:15831)', local='AF_INET6([::]:6514)'
Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: Syslog connection accepted; fd='99', client='AF_INET6([2001:470:1f0b:16b0::1]:15830)', local='AF_INET6([::]:6514)'
Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: SSL error while reading stream; tls_error='SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca'
Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: I/O error occurred while reading; fd='96', error='Connection reset by peer (104)'
Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: Syslog connection closed; fd='96', client='AF_INET6([2001:470:1f0b:16b0::1]:15831)', local='AF_INET6([::]:6514)'
Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: SSL error while reading stream; tls_error='SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca'
Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: I/O error occurred while reading; fd='99', error='Connection reset by peer (104)'
Oct 26 09:53:49 nb17-lx2 syslog-ng[901]: Syslog connection closed; fd='99', client='AF_INET6([2001:470:1f0b:16b0::1]:15830)', local='AF_INET6([::]:6514)'

Or looking at it with Wireshark you’ll see this “Alert Message”:

Import the syslog x.509 certificate at System -> Certificates -> Import -> CA Certificate:

Logging via TLS will immediately start after that. No further configuration is needed.

Confirmed with Wireshark again: “Change Cipher Spec” followed by Application Data. Nice!

Very good. Keep in mind that you should not use unencrypted management protocols at all. Hence this one here is a step in the right direction.

Photo by Robin Spielmann on Unsplash.

During the last weeks, I had an interesting request to publish NTP servers to client systems by using DHCPv6 in an IPv6 only network. Our Fortigate (or me?) had to learn how to publish the information. Hence this post is not only about NTP and IPv6, but a small guide on how to walk through RFCs and how to get out the relevant information. I’m very happy I got the possibility to share my experience here. Thank you, Johannes!

It started with a small question from my co-worker: “Can you send our NTP servers with DHCP in the IPv6 only network?” And my first idea was: “I’m sure, I can configure this within of minutes.” But 10 minutes later I had to change my mind: it takes me more time to understand how I have to configure this. The simple reason for this is the missing option in the system configuration, the vendor didn’t implement (yet)…

I will show how to manage this with a FortiGate (FortiOS 6.0.14, there is no newer version available for this model) operating as a DHCPv6 server. At the time of beginning, the DHCP6 server configuration looks like this:

config system dhcp6 server
    edit 641
        set rapid-commit enable
        set lease-time 3602
        set domain "example.com"IP
        set subnet 2001:db8:c:641::/64
        set interface "IPv6only-01"
        config ip-range
            edit 1
                set start-ip 2001:db8:c:641::6401
                set end-ip 2001:db8:c:641::64ff
            next
        end
        set dns-server1 2001:db8:c:a53::53
        set dns-server2 2001:db8:c:b53::53
    next
end

As you can see, the DHCPv6 server is configured to publish a domain name and two DNS servers for recursive name resolution. Further on there’s an IPv6 range for stateful DHCPv6. But FortiOS didn’t allow me to configure the NTP server directly. But it allows me to send three self-defined options in DHCPv6:

config system dhcp6 server
    edit < id >
        set option1 {string}   Option 1.
        set option2 {string}   Option 2.
        set option3 {string}   Option 3.
    next
end

Could this be the way to fulfil my co-worker’s request? And how to create the {string} to send an IPv6 address or hostname to the client system? Having a look at the CLI of the FortiGate:

config system dhcp6 server
edit < id >
(server) # set option1 ?
< option-code >  [< options >] options must be even number of hexadecimal characters(optional)

This information didn’t help me. Now it looks like nothing else than reading RFCs helps. :-(

Digging through RFCs

In the list of FortiOS 6.0 supported RFCs, RFC 3315 is mentioned. But in RFC 3315 is no useful information about NTP… Reading the new RFC for DHCPv6, RFC 8415, you can find in section 24 a list of DHCPv6 options and the link to the complete list of DHCPv6 parameters assigned by IANA. Looks like that’s the information I need.

There is an option with the name “NTP Server”, number 56 (decimal), but the information about the order and possible values are somewhere else: in RFC 5908.

In RFC typical style there (RFC 5908 on page 4) you can find the required format of the option to send NTP-server information to the client system:

The format of the NTP Server Option is:
      0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |      OPTION_NTP_SERVER        |          option-len           |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |                         suboption-1                           |
     :                                                               :
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |                         suboption-2                           |
     :                                                               :
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     :                                                               :
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |                         suboption-n                           |
     :                                                               :
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

       option-code: OPTION_NTP_SERVER (56),
       option-len:  Total length of the included suboptions.

For the ones who never had the need to read something like this, a few hints which helped me:

• The headline of the “table” are the bits, starting from 00 to 31 = 32bit.
• The field OPTION_NTP_SERVER is a 16-bit value and the field option-len is a 16bit value, too.
• The next fields suboption-1, suboption-2 to suboption-n has, at this point, no fixed length; but at least 32bit. This is because there is a pipe character on the left and right side, that indicates: this line is required. The next line, which has a colon at both sides, tells us it has a variable length, 0 or more lines.

To find the next piece of the puzzle, we continue reading the next page in RFC 5908 (page 5). Here we find how to format the sub-options:

The format of the NTP Server Address Suboption is:

0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |    NTP_SUBOPTION_SRV_ADDR     |        suboption-len = 16     |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |                                                               |
     |                                                               |
     |                   IPv6 address of NTP server                  |
     |                                                               |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

       IPv6 address of the NTP server: An IPv6 address,
       suboption-code: NTP_SUBOPTION_SRV_ADDR (1),
       suboption-len:  16.

Next hint:

• The field NTP_SUBOPTION_SRV_ADDR is a 16-bit value and the field suboption-len is a 16bit value, too.
• The next field (IPv6 address of NTP server) has a fixed length of 128 bit (4 lines with 32 bit). I hope you already expected this length. :-)

OK, we try to understand the information we have until now: We have to set the option number for the NTP server, which is 56 (dec) and 0x38 (hex). The NTP_SUBOPTION_SRV_ADDR has a sub-option code of 1 (0x0001 as 16bit hex value) and a suboption-length of 16 octets (if it is easier for you: 16 bytes). Then we have to put in the IPv6 (unicast-) address of the server. We take 2001:0db8:2800:0000:0000:0000:0ac3:0123 as an example. In the end, we calculate the complete length of the option and put the value in the field option-len:

• 2 octets for the NTP suboption field
• 2 octets for the NTP suboption length
• 16 octets for the NTP IPv6 server address

In summary 20 octets (0x14 in hex) is the calculated value for the field option-len. If we put in all the data, the fields look like this:

0                   1                   2                   3
      0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     | OPTION_NTP_SERVER = 0x0038    |      option-len = 0x0014      |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     |SUBOPTION_NTP_SRV_ADDR = 0x0001| suboption-len = 16dec = 0x0010|
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     | 2001:0db8:                                                    |
     | 2800:0000:                                                    |
     | 0000:0000:        IPv6 address of NTP server                  |
     | 0ac3:0123                                                     |
     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

But in FortiOS this is different than expected (by me?): we don’t have to calculate and put in the option-length here! We just need the option code (in decimal = 56) and the option value in hexadecimal characters as in the following line:

(server) # set option1 56 '0001001020010db828000000000000000ac30123'
       NTP_SUBOPTION_SRV_ADDR |   |
                 suboption-lenght | IPv6 unicast address of NTP server

It took me a few hours with Wireshark to realize this…

Publishing FQDNs

My co-worker was quite happy about my phone call with the good news. But: is publishing an IPv6 unicast address a good solution for the future? I decided no, I want to publish an FQDN.

Now, as I did understand the format of the NTP-server option, I had a look for how to publish the name of the NTP server. For this we have to read section 4.3 on page 6 of RFC 5908. The format of the NTP Server FQDN Suboption is:

0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |    NTP_SUBOPTION_SRV_FQDN     |         suboption-len         |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |                                                               |
   |                      FQDN of NTP server                       |
   :                                                               :
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
     suboption-code: NTP_SUBOPTION_SRV_FQDN (3),
     suboption-len: Length of the included FQDN field,
     FQDN: Fully-Qualified Domain Name of the NTP server or SNTP server.
           This field MUST be encoded as described in [RFC3315],
           Section 8.  Internationalized domain names are not allowed
           in this field.

Let’s have a try to send the hostname 3.de.pool.ntp.org to the client systems.

The FQDN must be encoded as described in section 8 of RFC 3315. The few lines there tell us to read section 3.1 of RFC 1035 but MUST NOT use the compressed form as described in section 4.1.4 of RFC 1035. And we MUST NOT use an IDN (internationalized domain name)!

This is how encode the hostname 3.de.pool.ntp.org based on section 3.1 in RFC 1035: First of all, we have to specify the length of the next part of the hostname (also called: label):

• one character which results in value 1 (0x01) followed by the ascii-code of the character 3: 0x33.
• Repeat this for the next label: length of 0x02 followed by 0x6465 for de
• For the next label: length (4 octets) and the word pool results in data 0x04706f6f6c
• And so on: length (3 octets) and data are 0x036e7470 for ntp
• One time similar again: length (3 octets) and data are 0x036f7267 for org
• And now, very important: notification for end of hostname: 0x00

I repeat the whole hexadecimal characters (separated with space characters for a better understanding):

01 33 02 6465 04 706f6f6c 03 6e7470 03 6f7267 00
    3  .  d e  .  p o o l  .  n t p  .  o r g

This is the FQDN for the NTP suboption. Now we have to calculate the length of the suboption: 19 octets, 0x13 in hex.

My line for configuring the hostname 3.de.pool.ntp.org in FortiOS looks is this one:

(server) # set option2 56 '00030013013302646504706f6f6c036e7470036f726700'
        suboption type = FQDN |   |
suboption length = 19dec = 0x0013 | FQDN as described a few lines above

Do you want to publish two FQDNs? For example 2.de.pool.ntp.org and 3.de.pool.ntp.org? No problem, just do it:

(server) # set option2 56 '00030026013202646504706f6f6c036e7470036f726700013302646504706f6f6c036e7470036f726700'

But this wasn’t a good solution for my co-worker. He was not able to configure his client with the information I sent by DHCPv6. (Please don’t ask me for details!) Next, he asked me to send the SNTP address.

What about SNTP?

A fast search and I found the next interesting RFC with the name Simple Network Time Protocol (SNTP) Configuration Option for DHCPv6: RFC 4075. Looking on page 2:

The format of the Simple Network Time Protocol servers option is as
   shown below:
       0                   1                   2                   3
       0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |      OPTION_SNTP_SERVERS       |        option-len            |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                                                               |
      |                  SNTP server (IPv6 address)                   |
      |                                                               |
      |                                                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      |                                                               |
      |                  SNTP server (IPv6 address)                   |
      |                                                               |
      |                                                               |
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      :                              ...                              :
      +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
      option-code: OPTION_SNTP_SERVERS (31)
      option-len:  Length of the 'SNTP server' fields, in octets;
                   it must be a multiple of 16
      SNTP server: IPv6 address of SNTP server

SNTP is option code 31, FortiOS is calculating the option length itself, we just need the IPv6 unicast (only unicast?) address of the SNTP server

(server) # set option3 31 '20010db828000000000000000ac30123'

Do you want to publish two SNTP servers? No problem, do it like this:

(server) # set option3 31 '20010db828000000000000000ac2012320010db828000000000000000ac30123'

FortiOS and RouterOS Configuration

Today, the DHCP6 server configuration of the FortiGate looks like this:

config system dhcp6 server
    edit 641
        set rapid-commit enable
        set lease-time 3602
        set domain "example.com"
        set subnet 2001:db8:c:641::/64
        set interface "IPv6only-01"
        set option1 56 '0001001020010db828000000000000000ac30123'
        set option2 56 '00030013013302646504706f6f6c036e7470036f726700'
        set option3 21 '20010db828000000000000000ac30123'
        config ip-range
            edit 1
                set start-ip 2001:db8:c:641::6401
                set end-ip 2001:db8:c:641::64ff
            next
        end
        set dns-server1 2001:db8:c:a53::53
        set dns-server2 2001:db8:c:b53::53
    next
end

FortiOS in 6.0.14 allows only 3 custom values, I can’t publish more custom options. One more feature request to Fortinet: please allow more (maybe 9?) custom options here. :-)

In the same way, it’s possible to publish other information with DHCPv6, if the operating systems of the server allow the administrator to publish custom values in DHCPv6. In some operating systems, at the time of writing these lines, it’s the only way to send even basic information to a client system. For example, if you have to do a similar configuration in MikroTik RouterOS (tested with 6.48.5), it looks like this:

/ipv6 dhcp-server option
add code=23 name=DNSa value="'2001:db8:c:a53::53'"
add code=23 name=DNSb value="'2001:db8:c:b53::53'"
add code=24 name=DNSsearch value="0x07'example'0x03'com'0x00"
add code=31 name=SNTPboth value="'2001:db8:2800::ac2:123''2001:db8:2800::ac3:123'"
add code=56 name=NTPaddr   value="'000120010db8000c0b53000000000053'"
/ipv6 dhcp-server
add dhcp-option= DNSa,DNSb,DNSsearch,SNTPboth,NTPaddr interface=eth3 lease-time=29m name=IPv6test

In my experience, RouterOS is sending only requested options, other than FortiOS. In RouterOS, we don’t have to calculate the length of the DHCPv6 option, too.

Please always keep in mind: if you don’t send at least the other-flag in RA packets, most client systems don’t try to get DHCPv6 information from the network.

Photo by Bank Phrom on Unsplash.

In general, Network Address Translation (NAT) solves some problems but should be avoided wherever possible. It has nothing to do with security and is only a short-term solution on the way to IPv6. (Yes, I know, the last 20 years have proven that NAT is used everywhere every time. 😉) This applies to all kinds of NATs for IPv4 (SNAT, DNAT, PAT) as well as for NPTv6 and NAT66.

However, there are two types of NATs that do not only change the network addresses but do a translation between the two Internet Protocols, that is IPv4 <-> IPv6 and vice versa. Let’s focus on NAT46 this time. In which situations is it used and why? Supplemented by a configuration guide for the FortiGates, a downloadable PCAP and Wireshark screenshots.

NAT46 Concept

Basically, if you have IPv6-only servers (to avoid the unnecessary dual-stack burden for at least some of your infrastructure), but still want to have those servers accessible for IPv4-only clients, you have to use some kind of protocol translation somewhere. Either through reverse proxies or load balancers or through a NAT46 gateway like this:

Note that the NAT46 proxy, pronounced NAT-four-six by the way, does not necessarily have to be in the direct path between the client and server – it only has to be accessible by appropriate routes. Nevertheless, having one central firewall in place which does the job fits perfectly for small installations.

Furthermore, note that you need a hostname in the public DNS with an A record for the IPv4 address on your NAT46 gateway. But you don’t need a special DNS device such as a DNS64 box for NAT64 to work.

The basic concept of translating IP and ICMP between IPv4 and IPv6 aka “Stateless IP/ICMP Translation Algorithm (SIIT)” is described in RFC 7915. Funnily enough, the keyword “NAT46” is not said in the document at all.

Since I have at least one “server” (it’s a Raspi) running IPv6-only, I was able to test this NAT46 gateway. My True Random PSK Generator at https://random.weberlab.de/ has only an AAAA record, while I used https://random46.weberlab.de/ to access it via legacy IP:

NAT46 on a FortiGate

I’m not a big fan of FortiGate firewalls because they are neither reliable nor sound in many situations. However, they offer many cool and new features that other vendors don’t have. So let’s configure NAT46. I’m using a FortiWiFi-61E with FortiOS v7.0.9 for this setup.

At first, it requires a NAT object aka “Virtual IP” of type IPv4 which maps the (public) IPv4 address to the (internal) IPv6 address:

Second, you need an IPv6 Pool for the IPv6 source from which the firewall will initiate the internal IPv6 connections. Note the NAT46 checkbox. Also note that the “pool” is not really a pool but only capable of one or two IPv6 addresses. It took me a while to figure it out. All ranges I tested weren’t valid until I reduced the pool to one single IPv6 address. As you can see, I chose a very special-looking one: an IPv6 address with 4646 at the very end to spot it easily:

Finally, you need an appropriate Firewall Policy. Note that you must select the “NAT46” feature before you can select the destination which is the virtual IP object:

I’m a little scared when looking at this policy in the overview since the destination IPv6 address object says “any” though nothing was selected when creating the policy. AHH. It looks like this policy now allows all IPv6 destinations. Hopefully, it doesn’t. (That’s what I mean when stating that FortiGate firewalls are not that sound.)

Having a look at the Forward Traffic Log you can indeed see both Internet Protocols:

The appropriate CLI commands for this feature are: (You have to adjust some of them according to your needs/setup, e.g., the profile-group, the inspection-mode, and the like.)

config firewall vip
    edit "random46"
        set comment "Test NAT46"
        set extip 194.247.5.23
        set nat44 disable
        set nat46 enable
        set extintf "wan1"
        set ipv6-mappedip 2001:470:1f0b:16b0:6986:b8d4:3649:9cbe
    next
end
config firewall ippool6
    edit "SNAT46"
        set startip 2001:470:1f0b:16b0::4646
        set endip 2001:470:1f0b:16b0::4646
        set nat46 enable
    next
end
config firewall policy
    edit 41
        set name "random46"
        set srcintf "wan1"
        set dstintf "internal"
        set action accept
        set nat46 enable
        set srcaddr "all"
        set dstaddr "random46"
        set srcaddr6 "all"
        set dstaddr6 "all"
        set schedule "always"
        set service "HTTP" "HTTPS" "PING" "PING6"
        set utm-status enable
        set inspection-mode proxy
        set profile-type group
        set profile-group "app-only"
        set logtraffic all
        set ippool enable
        set poolname6 "SNAT46"
    next
end

Having a look at the sessions via CLI, you can see both ones, legacy IP and IPv6. Note the “peer” line for each IP in which the other IP is referenced. Nice! You have to use two different commands to show those sessions, though. (That’s what I mean when stating that FortiGate firewalls are not that sound.)

fg2 # diagnose sys session list

session info: proto=6 proto_state=05 duration=1 expire=0 timeout=3600 flags=00000000 socktype=0 sockport=0 av_idx=0 use=3
origin-shaper=
reply-shaper=
per_ip_shaper=
class_id=0 ha_id=0 policy_dir=0 tunnel=/ vlan_cos=0/255
state=log may_dirty npu f00
statistic(bytes/packets/allow_err): org=1343/11/1 reply=7375/9/1 tuples=2
tx speed(Bps/kbps): 1316/10 rx speed(Bps/kbps): 7230/57
orgin->sink: org pre->post, reply pre->post dev=6->20/20->6 gwy=194.247.5.23/194.247.4.1
hook=pre dir=org act=noop 85.215.94.29:53928->194.247.5.23:443(0.0.0.0:0)
hook=post dir=reply act=noop 194.247.5.23:443->85.215.94.29:53928(0.0.0.0:0)
peer=2001:470:1f0b:16b0::4646:53928->2001:470:1f0b:16b0:6986:b8d4:3649:9cbe:443 naf=1
hook=pre dir=org act=noop 2001:470:1f0b:16b0::4646:53928->2001:470:1f0b:16b0:6986:b8d4:3649:9cbe:443(:::0)
hook=post dir=reply act=noop 2001:470:1f0b:16b0:6986:b8d4:3649:9cbe:443->2001:470:1f0b:16b0::4646:53928(:::0)
pos/(before,after) 0/(0,0), 0/(0,0)
misc=0 policy_id=41 pol_uuid_idx=606 auth_info=0 chk_client_info=0 vd=0
serial=018cd679 tos=ff/ff app_list=0 app=0 url_cat=0
rpdb_link_id=00000000 ngfwid=n/a
npu_state=0x4040400 ofld-O
npu info: flag=0x00/0x00, offload=0/0, ips_offload=0/0, epid=0/0, ipid=0/0, vlan=0x0000/0x0000
vlifid=0/0, vtag_in=0x0000/0x0000 in_npu=0/0, out_npu=0/0, fwd_en=0/0, qid=0/0
no_ofld_reason:
ofld_fail_reason(kernel, drv): none/not-established, none(0)/none(0)
npu_state_err=00/04
total session 1


fg2 # diagnose sys session6 list

session6 info: proto=6 proto_state=15 duration=1 expire=0 timeout=3600 flags=00000000 sockport=0 socktype=0 use=3
origin-shaper=
reply-shaper=
per_ip_shaper=
class_id=0 ha_id=0 policy_dir=0 tunnel=/ vlan_cos=0/0
state=log may_dirty npu app_valid
statistic(bytes/packets/allow_err): org=1563/11/0 reply=7802/12/0 tuples=2
tx speed(Bps/kbps): 893/7 rx speed(Bps/kbps): 4458/35
orgin->sink: org pre->post, reply pre->post dev=20->23/23->20
hook=pre dir=org act=noop 2001:470:1f0b:16b0::4646:53928->2001:470:1f0b:16b0:6986:b8d4:3649:9cbe:443(:::0)
hook=post dir=reply act=noop 2001:470:1f0b:16b0:6986:b8d4:3649:9cbe:443->2001:470:1f0b:16b0::4646:53928(:::0)
peer=194.247.5.23:443->85.215.94.29:53928 naf=2
dst_mac=b8:27:eb:03:a0:ac
misc=0 policy_id=41 auth_info=0 chk_client_info=0 vd=0
serial=00312c35 tos=ff/ff ips_view=0 app_list=2000 app=40568 url_cat=0
rpdb_link_id = 00000000 ngfwid=n/a
npu_state=0x4041808 ofld-R
npu info: flag=0x00/0x81, offload=0/0, ips_offload=0/0, epid=0/64, ipid=0/76, vlan=0x0000/0x0000
vlifid=0/76, vtag_in=0x0000/0x0000 in_npu=0/1, out_npu=0/1, fwd_en=0/0, qid=0/3
no_ofld_reason:
ofld_fail_reason(kernel, drv): none/not-established, none(0)/none(0)
npu_state_err=00/04

Finally, note that this setup required several other things around the mere network config which I have not shown here. That is:

• a hostname for random46.weberlab.de with only at least an A record
• ServerAliases on the apache2 config for the virtual host
• an adjusted rewrite condition to forward HTTP -> HTTPS for this ServerAlias
• running certbot again to have a valid X.509 certificate with this hostname in the subject alternative name field

Deeper Look on the Wire

I’ve captured some basic runs, that is: doing an HTTP request, getting redirected to HTTPS, as well as a ping aka echo-request. I captured on the client as well as on the server simultaneously and merged them later on. This capture is within the Ultimate PCAP already, but you can download it solely as well:

You can easily filter for ip or ipv6 to see only one of those Internet Protocols. Here they are side-by-side, looking at the SYN for the HTTP session:

As expected, the upper-layer protocol stuff is exactly the same after the NAT46 proxy, such as the TLS handshake with its ECDH client key exchange:

However, looking at ICMPv4 vs. ICMPv6 messages for echo-requests/-replies, the data portion looks a little different. ICMPv4 in this example used a timestamp which should be silently dropped according to RFC 7915, section 4.2. Looks like the FortiGate isn’t doing it that way but keeps the timestamp information within the data portion:

However, it’s working quite good. Nice! If you can spot any other differences between those translated protocols, please write a comment!

Fun Fact: NAT646

The other day I was on a german train using my T-Mobile tethering on my iPhone which gives perfect IPv6-native access, incl. DNS64/NAT64. Now, when surfing to this IPv4-only NAT46 domain, it eventually does a 646 translation. ;)

Of course, that would not have happened if the hostname would have had an AAAA record as well, which would be the case for real-world purposes in which your server hostname has an AAAA record (since it is IPv6-only) *and* the additional A record for the NAT46 translation.

Photo by Joshua Sortino on Unsplash.

Johannes published a basic NAT46 configuration for a Fortigate firewall with FortiOS 7.0 some time ago. I run such a service (legacy IPv4 access to IPv6-only resources) since FortiOS 5.6, which means more than six years; lastly with FortiOS 6.4. It’s running for more than 100 servers without any other problems as we see them with IPv4 only or dual stack services.

But we weren’t happy with the basic configuration example by Fortinet. We wanted some NAT46 sample configuration with more details, that is: including the original source IPv4 address within the synthesized/SNATted IPv6 address. More in this post, after a short story about my way to a running nat46 configuration with port forwarding in FortiOS 7.2.x.

(Please note: this post is about NAT46, not NAT64, which is more common. :))

We operate a firewall which is doing NAT46 for many of our hosting services. All servers have IPv6 GUA (Global Unicast Addresses) and we are happy with the configuration. It looks like this:

Bear the Challenge and Update the FortiGate

Now, Fortinet support asked me, because of an issue we opened a ticket, to update my productive firewall (with the nat46 configuration!) to FortiOS 7.0 or 7.2 soon. I didn’t want to do this step since I read the first 7.0.1 release notes in summer 2021. This is because Fortinet changed the whole NAT64 and NAT46 configuration with policy46, vip46 and vip64 and so on. You will lose all of your configuration for IPv4-IPv6 translation features, not written there word by word, but as a result. They call this “simplify policy and routing configurations” at Fortinet in their Release Notes. And of course, as you already expect: the transfer of the nat46/nat64 configuration from a 6.x operating system to the new 7.x configuration is not a simple clicking around.

During the update process, there is no automatic transfer of the ip pools for nat64 and nat46, and no transfer of the nat64 and nat46 policies. After the first boot with 7.0.x, you only get a hint like this on the console:

The config file may contain errors,
Please see details by the command 'diagnose debug config-error-log read'

As I always strongly recommend when you get this line: run the debug command before the next reboot! There you find a short hint about the ignored transfer of some configuration sections; in this case, as announced in the release notes:

fgtos70 # diagnose debug config-error-log read
→ → "set" "gui-nat46-64" "enable" @ root.system.settings:command parse error (error -61)
→ → "config" "firewall" "vip46" @ root:command parse error (error -61)
→ → "config" "firewall" "policy64" @ root:command parse error (error -61)
→ → "config" "firewall" "policy46" @ root:command parse error (error -61)
→ → "config" "system" "nat64" @ root:command parse error (error -61)
fgtos70 #

In my case, the configuration had around 800 lines less… Ouch!

I didn’t want to wait and keep many services of our hosting offline for a longer time, so an additional reboot and an update from 7.0.13 to 7.2.6 was the next task.

With the help of a self-written search-and-replace script and some additional, manual changes, I had a text file with my whole nat46 configuration which I could import without any warnings or errors. BE HAPPY.

Optimized nat46 Configuration

Unfortunately, there are no examples with the well-known NAT46/SIIT IPv6-prefix 64:ff9b::/96 in many documentations. This 64:ff9b::/96 range is 32-bit wide (96 + 32 = 128 bit) and represents the whole (32-bit wide) IPv4 range in IPv6, used for NAT46 (and NAT64) and SIIT (see RFC 6052 and RFC 7755 and many others).

The IPv4 address is shown as a hexadecimal address in the last two hextets of the IPv6 address. This means: if you have the (example) IPv4 address 9.9.9.9 it converts to 64:ff9b::909:909 in IPv6 if you use it for NAT46/NAT64; IPv4 address 10.11.12.13 is translated to 64:ff9b::a0b:c0d.  And one more example: IPv4 192.0.2.240 is 64:ff9b::c000:2f0. This is also called a synthesized IPv4 address when the IPv4 address is included in the 64:ff9b address.

If you create an ippool6 object in FortiOS with “startip 64:ff9b::” and “endip 64:ff9b::ffff:ffff“, (remember: the whole 32-bit IPv4 Internet!) and use this IPv6 range as a pool for the NAT46 (poolname6 in the policy for NAT46), the Fortigate will include the synthesized IPv4 source address as the IPv6 source address.

The IPv6-only service you operate behind your nat46 firewall can write this address to the log and may help you solve problems. This is the way IPv6-only servers can still provide the service for IPv4-only clients.

Configuration Example with NAT46 and Port-Forwarding

Here is my configuration example, with port-forwarding by a nat46 policy. First, we must create an IPv6 pool with a size of 32 bit to nat the (whole source) IPv4 address to a (destination) IPv6 address:

config firewall ippool6
    edit "IPv6pool-for-nat46-with-full-32bit-IPv4-range"
        set startip 64:ff9b::0
        set endip 64:ff9b::ffff:ffff
        set comments "common NAT46-NAT64 range"
        set nat46 enable
    next
end

The next step is a service object. We need one virtual IP (vip) object for each port/service which has to be mapped from an IPv4 address to an IPv6 address. Here is the example for HTTP and HTTPS:

config firewall vip
    edit "nat46_testA:80"
        set extip 192.0.2.39
        set nat44 disable
        set nat46 enable
        set extintf "v100"
        set portforward enable
        set ipv6-mappedip 2001:db8:de0:f270::200
        set extport 80
        set ipv6-mappedport 80
    next
    edit "nat46_testA:443"
        set extip 192.0.2.39
        set nat44 disable
        set nat46 enable
        set extintf "v100"
        set portforward enable
        set ipv6-mappedip 2001:db8:de0:f270::200
        set extport 443
        set ipv6-mappedport 443
    next
end

Now we create an object for port-forwarding within the NAT46 object. The IPv4-address is 192.0.2.39, IPv4 port is 22200 and the IPv6 port is 22 (default SSH):

config firewall vip
    edit "nat46_testA:22"
        set extip 192.0.2.39
        set nat44 disable
        set nat46 enable
        set extintf "v100"
        set portforward enable
        set ipv6-mappedip 2001:db8:de0:f270::200
        set extport 22200
        set ipv6-mappedport 22
    next
end

For this type of port-forwarding an additional custom service object is required. This example is to forward port 22200, but not defined for which destination port:

config firewall service custom
    edit "nat46fwd22200tcp"
        set category "nat46forward"
        set comment "in use for nat46 port forward"
        set <strong>tcp-portrange 22200</strong>
    next
end

Finally, we can combine all the created objects in policies:

config firewall policy
    edit 461
        set srcintf "v100"
        set dstintf "v503"
        set action accept
        set nat46 enable
        set srcaddr "ExternalTestIPv4"
        set dstaddr "nat46_testA:80" "nat46_testA:443"
        set srcaddr6 "all"
        set dstaddr6 "all"
        set schedule "always"
        set service "HTTP" "HTTPS"
        set auto-asic-offload disable     #just for better demonstration, set to enable for production
        set ippool enable
        set poolname6 "IPv6pool-for-nat46-with-full-32bit-IPv4-range"
        set comments "HTTP/HTTPS access policy"
    next
    edit 462
        set srcintf "v100"
        set dstintf "v503"
        set action accept
        set nat46 enable
        set srcaddr "ExternalTestIPv4"
        set dstaddr "nat46_testA:22" 
        set srcaddr6 "all"
        set dstaddr6 "all"
        set schedule "always"
        set service "nat46fwd22200tcp" "SSH"   #we need both, the external tcp-port and the internal tcp-port
        set auto-asic-offload disable          #just for better demonstration, set to enable for production
        set ippool enable
        set poolname6 "IPv6pool-for-nat46-with-full-32bit-IPv4-range"
        set comments "SSH access policy with port forwarding"
    next
end

In policy 462 you can see the port-forwarding configuration. We need the service object nat46fwd22200tcp for the IPv4 mapped port and the SSH service object for the destination IPv6 service. Read the next section to see the internal process to understand, why the configuration needs this.

With this configuration, the IPv6-only daemon for SSH or HTTP/HTTPS can write the synthesized IPv4 address in the log and the marketing department can follow a client connection through the logs.

Until now I could not test the real effect of changing srcaddr6 and dstaddr6 from all to other values. Looking forward to get some feedback!

Bonus: Start of a Data Flow of a nat46 Connection

If you have a detailed look at the following lines (built-in tcpdump in FortiOS), you can see a new, hidden interface called naf.root (I expect root is the name of the vdom, it might be different if you have other vdoms) which is in use to help to translate IPv4 in IPv6 and back:

• In line 4 the test-ssh-client opens an IPv4 connection to port 22200 with a syn.
• In line 5 the syn packet is routed through the naf.root interface (still IPv4) and comes back in line 6 as an IPv6 packet to destination port 22 for SSH.
• The packet was sent to the vlan 503 (line 7) which uses a port channel (line 8) and leaves the unit on physical interface x3 (line 9).

All other lines are very similar and I am only publishing a part of the connection here. And I’m very sorry: I did not understand why there is a naf.root out in line 5 and why it is a naf.root in in line 6; same for all other lines with naf.root. Feedback is welcome!

fgtos7 # diagnose sniffer packet any 'host 192.0.2.46 or 64:ff9b::c000:22e' 4 500 l
interfaces=[any]
filters=[host 192.0.2.46 or 64:ff9b::c000:22e]
11:17:41.267612 v100 in 192.0.2.46.59172 → 192.0.2.39.22200: syn 160445192 
11:17:41.267633 naf.root out 192.0.2.46.59172 → 192.0.2.39.22200: syn 160445192 
11:17:41.267635 naf.root in 64:ff9b::c000:22e.59172 → 2001:db8:de0:f270::200.22: syn 160445192 
11:17:41.267648 v503 out 64:ff9b::c000:22e.59172 → 2001:db8:de0:f270::200.22: syn 160445192 
11:17:41.267649 PCdmz out 64:ff9b::c000:22e.59172 → 2001:db8:de0:f270::200.22: syn 160445192 
11:17:41.267650 x3 out 64:ff9b::c000:22e.59172 → 2001:db8:de0:f270::200.22: syn 160445192 
11:17:41.267846 v503 in 2001:db8:de0:f270::200.22 → 64:ff9b::c000:22e.59172: syn 4075988132 ack 160445193 
11:17:41.267869 naf.root out 2001:db8:de0:f270::200.22 → 64:ff9b::c000:22e.59172: syn 4075988132 ack 160445193 
11:17:41.267870 naf.root in 192.0.2.39.22200 → 192.0.2.46.59172: syn 4075988132 ack 160445193 
11:17:41.267879 v100 out 192.0.2.39.22200 → 192.0.2.46.59172: syn 4075988132 ack 160445193 
11:17:41.267880 PCwan out 192.0.2.39.22200 → 192.0.2.46.59172: syn 4075988132 ack 160445193 
11:17:41.267880 x2 out 192.0.2.39.22200 → 192.0.2.46.59172: syn 4075988132 ack 160445193 
11:17:41.279083 v100 in 192.0.2.46.59172 → 192.0.2.39.22200: ack 4075988133 
11:17:41.279088 naf.root out 192.0.2.46.59172 → 192.0.2.39.22200: ack 4075988133 
11:17:41.279089 naf.root in 64:ff9b::c000:22e.59172 → 2001:db8:de0:f270::200.22: ack 4075988133 
11:17:41.279092 v503 out 64:ff9b::c000:22e.59172 → 2001:db8:de0:f270::200.22: ack 4075988133 
11:17:41.279093 PCdmz out 64:ff9b::c000:22e.59172 → 2001:db8:de0:f270::200.22: ack 4075988133 
11:17:41.279094 x3 out 64:ff9b::c000:22e.59172 → 2001:db8:de0:f270::200.22: ack 4075988133 
11:17:41.284112 v100 in 192.0.2.46.59172 → 192.0.2.39.22200: psh 160445193 ack 4075988133 
11:17:41.284116 naf.root out 192.0.2.46.59172 → 192.0.2.39.22200: psh 160445193 ack 4075988133 
11:17:41.284117 naf.root in 64:ff9b::c000:22e.59172 → 2001:db8:de0:f270::200.22: psh 160445193 ack 4075988133 
11:17:41.284120 v503 out 64:ff9b::c000:22e.59172 → 2001:db8:de0:f270::200.22: psh 160445193 ack 4075988133 
11:17:41.284121 PCdmz out 64:ff9b::c000:22e.59172 → 2001:db8:de0:f270::200.22: psh 160445193 ack 4075988133 
11:17:41.284122 x3 out 64:ff9b::c000:22e.59172 → 2001:db8:de0:f270::200.22: psh 160445193 ack 4075988133

Further sources: SIIT task of Tore Anderson on RIPE69 (Nov. 2014).

Photo by Ian Taylor on Unsplash.

It happens occasionally that a customer has to choose between a Palo and a Forti. While I would always favour the Palo for good reasons, I can understand that the Forti is chosen for cost savings, for example.

Fortunately, there is a hidden way of installing PAN-OS, the operating system from Palo Alto Networks, on FortiGate hardware firewalls. Here’s how you can do it:

I’m using a Fortinet FortiGate FG-501E for this demo with (formerly) FortiOS v7.2.7. I’m upgrading it to PAN-OS 11.1.1.

The main step is to upload and reboot the FortiGate into an alternative image, that is: a PAN-OS image. For generic FortiGates, you must choose the KVM-based PAN-OS images. With the following CLI command on the FortiGate, you can download the image from an TFTP server and reboot into it:

execute restore image tftp PA-VM-KVM-11.1.1.qcow2 192.168.21.5

The whole process in my lab was as follows. Note that you have to acknowledge the upgrade to an “unsupported image”:

fg2 # execute restore image tftp PA-VM-KVM-11.1.1.qcow2 192.168.21.5
This operation will replace the current firmware version!
Do you want to continue? (y/n)y

Please wait... 

Connect to tftp server 192.168.21.5 ... 
##########################################################

Get image from tftp server OK.
Warning: Upgrading to an unsupported image. Do you want to proceed? (y/n)y

Checking new firmware integrity ... pass

Please wait for system to restart.

After the reboot, you’re in the normal startup configuration of a Palo Alto firewall. –> Connect to it via the default IPv4 address of 192.168.1.1 with username:password of admin:admin.

In the dashboard, you can see the model and serial number, which are the ones from my FortiGate in this case:

Funnily enough, all those different interface names are used as well, that is:

Photo by Lindsay Henwood on Unsplash.

I got DHCPv6-PD aka prefix delegation up and running on a FortiGate. Yes! ✅ Configuring it is tricky since it’s not always clear, which options to use. You cannot see everything in the GUI (it even changes depending on other options made later on or selects hidden and wrong default values), hence you must set specific options via the CLI. I navigated around some bugs and finally got it running. Here we go:

Please refer to my previous blog post about DHCPv6 prefix delegation in detail.

Setup Notes

This is the basic lab, showing the ISP connection (green box, though not of interest for the following configs), as well as the Internet connection from the Forti (red box) and the internal clients (blue box):

• I’m using a FG-60F with FortiOS 7.6.1.
• The wan1 interface is attached to an AVM FRITZ!BOX 7560 with FRITZ!OS 07.30, acting as the DHCPv6 server for addresses (IA_NA) as well as for prefixes (IA_PD, handing out /62 prefixes).
• Have a look at RFC 8415 “Dynamic Host Configuration Protocol for IPv6 (DHCPv6)” respectively the IANA “Dynamic Host Configuration Protocol for IPv6 (DHCPv6)” for more details concerning the DHCPv6 option types.
• The internal interface shall use a /64 out of the delegated prefix.
• With FortiOS 7.4.6, the FortiGate did not succeed in requesting anything over DHCPv6 at all. I’ve no idea, why. Some notes here. Eventually, I tried an upgrade to FortiOS 7.6.1, in which it worked. The same problem still exists with FortiOS 7.6.1. In the end, disabling/enabling the DHCPv6 client on the wan1 interface made it work.
• The FortiGate receives its default route through the RA from the Fritzbox. (Remember that DHCPv6 does NOT send a default route at all. Everyone must rely on RAs in any case!)
• Though possible, I did not succeed in configuring the RA for the internal interface, serving a /64 out of the delegated range for the SLAAC method. Within the RA, the prefix did not show up at all. :( Most likely I forgot to set the IAPD value with “set delegated-prefix-iaid 5” within the “config ip6-delegated-prefix-list” section since I was able to get this running on some later scenarios.
• Nevertheless, I configured the internal interface to send the RA with the M-flag while setting up a stateful DHCPv6 server on the FortiGate itself. With this option, I was able to hand out a /64 to the internal clients. ✅
• On several configuration options, you can choose a “subnet” which defaults to “::/0” in the GUI as well as in the CLI. This is like an identifier to set the n-th network within the delegated prefix. Always change this to something with a /64 (rather than the default /0) to get the job done correctly. Otherwise, the FortiGate chooses the full delegated prefix such as a /62 (rather than a /64) and uses it on its interface, which stands against any best practice, will kill SLAAC, makes me horribly sad, and so on.
• Within the configuration, you have to choose an “IAPD” value which defaults to 5. This value is randomly chosen by Fortinet and used later on to identify the delegated prefix. I left it by this value of 5. The ugly thing: You have to set this value for the stateful DHCPv6 server as well, but this cannot be done through the GUI. If you activate the DHCPv6 server with the “Delegated” IP mode, you’ll get an error in the GUI. That is: At least for this single entry you must use the CLI.
• At least in FortiOS 7.6.1 there is a bug in which the FortiGate does not reply to RSs with RAs. That is: Your clients won’t get the default route until the first RA that is sent on a regular basis by the Forti is received by those clients. As a workaround, I’ve set the min- and max-interval values to 10 respectively 30 seconds. Since the defaults are way higher (600 seconds = 10 minutes), clients have to wait really long until IPv6 is up and running.
• Another bug is related to the output of the routing table for IPv6, since it does not show the default route that is gathered from the RA on the wan1 interface. Funnily, this is only true for the get router info6 routing-table but not for the diagnose ipv6 route list output.
• Other quite useful information about FortiGate’s DHCPv6 configuration options can be found here and there.

Configuration of the wan1 Interface

Through the GUI (kudos to Fortinet, in that we can now configure more IPv6 stuff through the GUI ;)). Note the “IAPD 5” without a prefix hint.

CLI:

config system interface
    edit "wan1"
        set vdom "root"
        set mode dhcp
        set allowaccess ping
        config ipv6
            set ip6-mode dhcp
            set ip6-allowaccess ping
            set dhcp6-prefix-delegation enable
            config dhcp6-iapd-list
                edit 5
                next
            end
        end
    next
end

Configuration of the internal Interface

Basics are possible through the GUI, while you MUST set some values through the CLI later.

Here is everything from the CLI in which you can set the “delegated-prefix-iaid 5” for the DHCPv6 server. Note that I changed the “ip6-other-flag” to disable since for stateful DHCPv6 you only need the M-flag:

config system interface
    edit "internal"
        set vdom "root"
        config ipv6
            set ip6-mode delegated
            set ip6-allowaccess ping https ssh
            set ip6-send-adv enable
            set ip6-manage-flag enable
            set ip6-other-flag disable
            set ip6-max-interval 30
            set ip6-min-interval 10
            set ip6-delegated-prefix-iaid 5
            set ip6-upstream-interface "wan1"
            set ip6-subnet ::1/64
        end
    next
end
config system dhcp6 server
    edit 1
        set dns-service delegated
        set subnet ::/64
        set interface "internal"
        set upstream-interface "wan1"
        set delegated-prefix-iaid 5
        set ip-mode delegated
    next
end

Information

Some information can be gathered through the GUI and the CLI. Note the missing IPv6 default route in the GUI as well as in the first CLI command, while the second one lists the “gwy”:

FortiGate-60F # get router info6 routing-table
IPv6 Routing Table
Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF,
       IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, B - BGP, V - BGP VPNv6
       * - candidate default

Timers: Uptime

Routing table for VRF=0
C       ::1/128 via ::, root, 00:19:47
C       2003:c6:af32:8400:6d5:90ff:fe42:8612/128 via ::, wan1, 00:07:38
C       2003:c6:af32:84fc::/64 via ::, internal, 00:03:47


FortiGate-60F # 
FortiGate-60F # 
FortiGate-60F # 
FortiGate-60F # diagnose ipv6 route list
vf=0 vrf=0 tbl=65534 type=02(local) protocol=0(unspec) flag=80200000 prio=0 dst:::1/128 dev=18(root) pmtu=16436
vf=0 vrf=0 tbl=65534 type=02(local) protocol=0(unspec) flag=80200000 prio=0 dst:2003:c6:af32:8400:6d5:90ff:fe42:8612/128 dev=18(root) pmtu=16436
vf=0 vrf=0 tbl=65534 type=01(unicast) protocol=2(kernel) flag=00000000 prio=256 dst:2003:c6:af32:8400:6d5:90ff:fe42:8612/128 dev=5(wan1) pmtu=1492
vf=0 vrf=0 tbl=65534 type=02(local) protocol=0(unspec) flag=00300000 prio=0 dst:2003:c6:af32:84fc::/128 dev=18(root) pmtu=16436
vf=0 vrf=0 tbl=65534 type=02(local) protocol=0(unspec) flag=80200000 prio=0 dst:2003:c6:af32:84fc::1/128 dev=18(root) pmtu=16436
vf=0 vrf=0 tbl=65534 type=01(unicast) protocol=0(unspec) flag=01000200 prio=0 dst:2003:c6:af32:84fc::2/128 dev=23(internal) pmtu=1500
vf=0 vrf=0 tbl=65534 type=01(unicast) protocol=2(kernel) flag=00400000 prio=256 dst:2003:c6:af32:84fc::/64 dev=23(internal) pmtu=1500
vf=0 vrf=0 tbl=65534 type=02(local) protocol=0(unspec) flag=00300000 prio=0 dst:fe80::/128 dev=18(root) pmtu=16436
vf=0 vrf=0 tbl=65534 type=02(local) protocol=0(unspec) flag=00300000 prio=0 dst:fe80::/128 dev=18(root) pmtu=16436
[...]
vf=0 vrf=0 tbl=65534 type=01(unicast) protocol=3(boot) flag=00000000 prio=256 dst:ff00::/8 dev=5(wan1) pmtu=1492
vf=0 vrf=0 tbl=65534 type=01(unicast) protocol=3(boot) flag=00000000 prio=256 dst:ff00::/8 dev=6(wan2) pmtu=1500
vf=0 vrf=0 tbl=65534 type=01(unicast) protocol=2(kernel) flag=00450000 prio=1024 gwy:fe80::7eff:4dff:fe93:9b5 dev=5(wan1) pmtu=1492
vf=0 vrf=0 tbl=0 type=07(unreachable) protocol=2(kernel) flag=00200000 prio=-1 dev=18(root) pmtu=16436

FortiGate-60F #

The following Wireshark screenshot shows the DHCPv6 messages between the FortiGate (wan1) and the Fritzbox. The first reply (message nr. 6) is one that wasn’t working correctly (status code 13 = no binding), for whatever reason. I disabled/enabled DHCPv6 on wan1 again. Messages 18-21 are the correct ones, handing out the /62 prefix:

This is the DHCPv6 debug log during the complete startup of the wan1 interface. Maybe someone finds hints about the mentioned DHCPv6 “no binding” issues:

=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2024.12.19 09:13:30 =~=~=~=~=~=~=~=~=~=~=~=
login as: admin
admin@192.168.1.99's password: 
FortiGate-60F # diagnose debug reset

FortiGate-60F # diagnose debug console timestamp enable 

FortiGate-60F # diagnose debug application dhcp6c 255
Debug messages will be on for 30 minutes.
FortiGate-60F # diagnose debug enable

FortiGate-60F # 2024-12-19 00:14:10 [debug]dhcp6_check_timer() called
2024-12-19 00:14:10 [info]client6_mainloop() timeout=10 sec, cfd=5, kfd=6
2024-12-19 00:14:21 [debug]dhcp6_check_timer() called
2024-12-19 00:14:21 [info]client6_mainloop() timeout=10 sec, cfd=5, kfd=6
2024-12-19 00:14:32 [debug]dhcp6_check_timer() called
2024-12-19 00:14:32 [info]client6_mainloop() timeout=10 sec, cfd=5, kfd=6
2024-12-19 00:14:35 [info]dhcp6c_kernel_event_handler() dhcp6c_kernel_event_handler called
2024-12-19 00:14:35 [info]dhcp6c_intf_changed() intf changed. caller=0x92bae8
2024-12-19 00:14:35 [debug]dhcp6_check_timer() called
2024-12-19 00:14:35 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:14:36 [info]dhcp6c_netlink_event_handler() netlink event is received.
2024-12-19 00:14:36 [debug]dhcp6_check_timer() called
2024-12-19 00:14:36 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:14:36 [info]dhcp6c_netlink_event_handler() netlink event is received.
2024-12-19 00:14:36 [debug]dhcp6_check_timer() called
2024-12-19 00:14:36 [debug]dhcp6_check_timer() timer func=0x926788
2024-12-19 00:14:36 [debug]dhcp6c_resource_update() dhcp6client: checking if we need to reinterfaceure
2024-12-19 00:14:36 [debug]dhcp6c_resource_update() modem is down
2024-12-19 00:14:36 [debug]interface_has_changed() intf wan1 vdom root HA status old:1 new:1 

2024-12-19 00:14:36 [debug]dhcp6_check_timer() called
2024-12-19 00:14:36 [info]client6_mainloop() timeout=10 sec, cfd=5, kfd=6
2024-12-19 00:14:36 [info]dhcp6c_netlink_event_handler() netlink event is received.
2024-12-19 00:14:36 [debug]dhcp6_check_timer() called
2024-12-19 00:14:36 [info]client6_mainloop() timeout=10 sec, cfd=5, kfd=6
2024-12-19 00:14:37 [info]dhcp6c_kernel_event_handler() dhcp6c_kernel_event_handler called
2024-12-19 00:14:37 [info]dhcp6c_intf_changed() intf changed. caller=0x92bae8
2024-12-19 00:14:37 [debug]dhcp6_check_timer() called
2024-12-19 00:14:37 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:14:39 [info]dhcp6c_kernel_event_handler() dhcp6c_kernel_event_handler called
2024-12-19 00:14:39 [info]dhcp6c_intf_changed() intf changed. caller=0x92bae8
2024-12-19 00:14:39 [debug]dhcp6_check_timer() called
2024-12-19 00:14:39 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:14:40 [info]dhcp6c_netlink_event_handler() netlink event is received.
2024-12-19 00:14:40 [info]dhcp6c_nlm_handler() Restart and renew interface wan1.
2024-12-19 00:14:40 [debug]dhcp6_remove_event() removing an event on wan1, state=SOLICIT
2024-12-19 00:14:40 [warning]uncache_intf6_info() delete interface  wan1 cache
2024-12-19 00:14:40 [debug]dhcp6c_intf_get_duid() called
2024-12-19 00:14:40 [debug]dhcp6c_intf_get_duid() Generated a new DUID: 00:03:00:01:04:d5:90:42:86:12.
2024-12-19 00:14:40 [debug]dhcp6_reset_timer() reset a timer on wan1, state=INIT, timeo=0, retrans=76
2024-12-19 00:14:40 [info]dhcp6c_nlm_handler() Restart and renew interface wan1.
2024-12-19 00:14:40 [debug]dhcp6_remove_event() removing an event on wan1, state=INIT
2024-12-19 00:14:40 [warning]uncache_intf6_info() delete interface  wan1 cache
2024-12-19 00:14:40 [debug]dhcp6c_intf_get_duid() called
2024-12-19 00:14:40 [debug]dhcp6c_intf_get_duid() Generated a new DUID: 00:03:00:01:04:d5:90:42:86:12.
2024-12-19 00:14:40 [debug]dhcp6_reset_timer() reset a timer on wan1, state=INIT, timeo=0, retrans=188
2024-12-19 00:14:40 [info]dhcp6c_nlm_handler() Restart and renew interface wan1.
2024-12-19 00:14:40 [debug]dhcp6_remove_event() removing an event on wan1, state=INIT
2024-12-19 00:14:40 [warning]uncache_intf6_info() delete interface  wan1 cache
2024-12-19 00:14:40 [debug]dhcp6c_intf_get_duid() called
2024-12-19 00:14:40 [debug]dhcp6c_intf_get_duid() Generated a new DUID: 00:03:00:01:04:d5:90:42:86:12.
2024-12-19 00:14:40 [debug]dhcp6_reset_timer() reset a timer on wan1, state=INIT, timeo=0, retrans=561
2024-12-19 00:14:40 [debug]dhcp6_check_timer() called
2024-12-19 00:14:40 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:14:40 [info]dhcp6c_netlink_event_handler() netlink event is received.
2024-12-19 00:14:40 [info]dhcp6c_intf_changed() intf changed. caller=0x92be04
2024-12-19 00:14:40 [info]dhcp6c_intf_changed() intf changed. caller=0x92be04
2024-12-19 00:14:40 [info]client6_mainloop() cmdb msg received, sock =5
2024-12-19 00:14:40 [info]dhcp6c_intf_changed() intf changed. caller=0x23f9be8
2024-12-19 00:14:40 [debug]dhcp6_check_timer() called
2024-12-19 00:14:40 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:14:40 [info]dhcp6c_netlink_event_handler() netlink event is received.
2024-12-19 00:14:40 [debug]dhcp6_check_timer() called
2024-12-19 00:14:40 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:14:43 [debug]dhcp6_check_timer() called
2024-12-19 00:14:43 [debug]dhcp6_check_timer() timer func=0x9274f0
2024-12-19 00:14:43 [debug]client6_send() a new XID (503090) is generated
2024-12-19 00:14:43 [debug]copy_option() set client ID (len 10)
2024-12-19 00:14:43 [debug]copyout_option() set identity association
2024-12-19 00:14:43 [debug]copy_option() set elapsed time (len 2)
2024-12-19 00:14:43 [debug]copy_option() set option request (len 4)
2024-12-19 00:14:43 [debug]copyout_option() set IA_PD
2024-12-19 00:14:43 [debug]client6_send() send solicit to ff02::1:2%wan1
2024-12-19 00:14:43 [debug]dhcp6_reset_timer() reset a timer on wan1, state=SOLICIT, timeo=0, retrans=1049
2024-12-19 00:14:43 [debug]dhcp6_check_timer() timer func=0x926788
2024-12-19 00:14:43 [debug]dhcp6c_resource_update() dhcp6client: checking if we need to reinterfaceure
2024-12-19 00:14:43 [debug]dhcp6c_resource_update() modem is down
2024-12-19 00:14:43 [debug]interface_has_changed() intf wan1 vdom root HA status old:1 new:1 

2024-12-19 00:14:43 [debug]dhcp6_check_timer() called
2024-12-19 00:14:43 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:14:43 [info]client6_mainloop() msg received, sock =11
2024-12-19 00:14:43 [debug]client6_recv() receive advertise from fe80::7eff:4dff:fe93:9b5%wan1 on wan1
2024-12-19 00:14:43 [debug]dhcp6_get_options() get DHCP option client ID, len 10
2024-12-19 00:14:43 [debug]   DUID: 00:03:00:01:04:d5:90:42:86:12
2024-12-19 00:14:43 [debug]dhcp6_get_options() get DHCP option server ID, len 10
2024-12-19 00:14:43 [debug]   DUID: 00:03:00:01:7c:ff:4d:93:09:b5
2024-12-19 00:14:43 [debug]dhcp6_get_options() get DHCP option preference, len 1
2024-12-19 00:14:43 [debug]   preference: 0
2024-12-19 00:14:43 [debug]dhcp6_get_options() get DHCP option DNS, len 16
2024-12-19 00:14:43 [debug]dhcp6_get_options() get DHCP option opt_86, len 16
2024-12-19 00:14:43 [info]dhcp6_get_options() unknown or unexpected DHCP6 option opt_86, len 16
2024-12-19 00:14:43 [debug]dhcp6_get_options() get DHCP option identity association, len 40
2024-12-19 00:14:43 [debug]   IA_NA: ID=5, T1=1800, T2=2880
2024-12-19 00:14:43 [debug]copyin_option() get DHCP option IA address, len 24
2024-12-19 00:14:43 [debug]copyin_option()   IA_NA address: 2003:c6:af32:8400:6d5:90ff:fe42:8612 pltime=3600 vltime=7200
2024-12-19 00:14:43 [debug]dhcp6_get_options() get DHCP option IA_PD, len 41
2024-12-19 00:14:43 [debug]   IA_PD: ID=5, T1=1800, T2=2880
2024-12-19 00:14:43 [debug]copyin_option() get DHCP option IA_PD prefix, len 25
2024-12-19 00:14:43 [debug]copyin_option()   IA_PD prefix: 2003:c6:af32:84fc::/62 pltime=3600 vltime=7200
2024-12-19 00:14:43 [debug]client6_recvadvert() server ID: 00:03:00:01:7c:ff:4d:93:09:b5, pref=0
2024-12-19 00:14:43 [debug]client6_recvadvert() reset timer for wan1 to 0.990649
2024-12-19 00:14:43 [debug]dhcp6_check_timer() called
2024-12-19 00:14:43 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:14:45 [info]client6_mainloop() cmdb msg received, sock =5
2024-12-19 00:14:45 [info]dhcp6c_intf_changed() intf changed. caller=0x23f9be8
2024-12-19 00:14:45 [debug]dhcp6_check_timer() called
2024-12-19 00:14:45 [debug]dhcp6_check_timer() timer func=0x9274f0
2024-12-19 00:14:45 [debug]select_server() picked a server (ID: 00:03:00:01:7c:ff:4d:93:09:b5)
2024-12-19 00:14:45 [debug]client6_send() a new XID (1cf373) is generated
2024-12-19 00:14:45 [debug]copy_option() set client ID (len 10)
2024-12-19 00:14:45 [debug]copy_option() set server ID (len 10)
2024-12-19 00:14:45 [debug]copyout_option() set IA address
2024-12-19 00:14:45 [debug]copyout_option() set identity association
2024-12-19 00:14:45 [debug]copy_option() set elapsed time (len 2)
2024-12-19 00:14:45 [debug]copy_option() set option request (len 4)
2024-12-19 00:14:45 [debug]copyout_option() set IA_PD prefix
2024-12-19 00:14:45 [debug]copyout_option() set IA_PD
2024-12-19 00:14:45 [debug]client6_send() send request to ff02::1:2%wan1
2024-12-19 00:14:45 [debug]dhcp6_reset_timer() reset a timer on wan1, state=REQUEST, timeo=0, retrans=1080
2024-12-19 00:14:45 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:14:45 [info]client6_mainloop() msg received, sock =11
2024-12-19 00:14:45 [debug]client6_recv() receive reply from fe80::7eff:4dff:fe93:9b5%wan1 on wan1
2024-12-19 00:14:45 [debug]dhcp6_get_options() get DHCP option client ID, len 10
2024-12-19 00:14:45 [debug]   DUID: 00:03:00:01:04:d5:90:42:86:12
2024-12-19 00:14:45 [debug]dhcp6_get_options() get DHCP option server ID, len 10
2024-12-19 00:14:45 [debug]   DUID: 00:03:00:01:7c:ff:4d:93:09:b5
2024-12-19 00:14:45 [debug]dhcp6_get_options() get DHCP option status code, len 16
2024-12-19 00:14:45 [debug]   status code: no binding
2024-12-19 00:14:45 [info]client6_recvreply() status code: no binding
2024-12-19 00:14:45 [debug]dhcp6c_ha_sync_send_lease_to() [dhcp6c_ha_sync_send_lease_to] serialno='          (null)', intf='wan1'

2024-12-19 00:14:45 [debug]dhcp6c_ha_sync_send_lease_to() send DHCP6_HA_SYNC_T_LEASE

2024-12-19 00:14:45 [debug]dhcp6_remove_event() removing an event on wan1, state=REQUEST
2024-12-19 00:14:45 [debug]dhcp6_remove_event() removing server (ID: 00:03:00:01:7c:ff:4d:93:09:b5)
2024-12-19 00:14:45 [debug]client6_recvreply() got an expected reply, sleeping.
2024-12-19 00:14:45 [debug]dhcp6_check_timer() called
2024-12-19 00:14:45 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:14:48 [debug]dhcp6_check_timer() called
2024-12-19 00:14:48 [debug]dhcp6_check_timer() timer func=0x926788
2024-12-19 00:14:48 [debug]dhcp6c_resource_update() dhcp6client: checking if we need to reinterfaceure
2024-12-19 00:14:48 [debug]dhcp6c_resource_update() modem is down
2024-12-19 00:14:48 [debug]interface_has_changed() intf wan1 vdom root HA status old:1 new:1 

2024-12-19 00:14:48 [debug]dhcp6_check_timer() called
2024-12-19 00:14:48 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:14:50 [debug]dhcp6_check_timer() called
2024-12-19 00:14:50 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:14:52 [debug]dhcp6_check_timer() called
2024-12-19 00:14:52 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:14:54 [debug]dhcp6_check_timer() called
2024-12-19 00:14:54 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:14:56 [info]dhcp6c_netlink_event_handler() netlink event is received.
2024-12-19 00:14:56 [info]dhcp6c_intf_changed() intf changed. caller=0x92be04
2024-12-19 00:14:56 [info]dhcp6c_intf_changed() intf changed. caller=0x92be04
2024-12-19 00:14:56 [debug]dhcp6_check_timer() called
2024-12-19 00:14:56 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:14:59 [debug]dhcp6_check_timer() called
2024-12-19 00:14:59 [debug]dhcp6_check_timer() timer func=0x926788
2024-12-19 00:14:59 [debug]dhcp6c_resource_update() dhcp6client: checking if we need to reinterfaceure
2024-12-19 00:14:59 [debug]dhcp6c_resource_update() modem is down
2024-12-19 00:14:59 [debug]interface_has_changed() intf wan1 vdom root HA status old:1 new:1 

2024-12-19 00:14:59 [debug]dhcp6_check_timer() called
2024-12-19 00:14:59 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:15:01 [debug]dhcp6_check_timer() called
2024-12-19 00:15:01 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:15:03 [debug]dhcp6_check_timer() called
2024-12-19 00:15:03 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:15:05 [debug]dhcp6_check_timer() called
2024-12-19 00:15:05 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:15:07 [debug]dhcp6_check_timer() called
2024-12-19 00:15:07 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:15:09 [debug]dhcp6_check_timer() called
2024-12-19 00:15:09 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:15:11 [debug]dhcp6_check_timer() called
2024-12-19 00:15:11 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:15:13 [debug]dhcp6_check_timer() called
2024-12-19 00:15:13 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:15:15 [debug]dhcp6_check_timer() called
2024-12-19 00:15:15 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:15:17 [debug]dhcp6_check_timer() called
2024-12-19 00:15:17 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:15:19 [debug]dhcp6_check_timer() called
2024-12-19 00:15:19 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:15:21 [debug]dhcp6_check_timer() called
2024-12-19 00:15:21 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:15:23 [debug]dhcp6_check_timer() called
2024-12-19 00:15:23 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:15:25 [debug]dhcp6_check_timer() called
2024-12-19 00:15:25 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:15:26 [info]dhcp6c_netlink_event_handler() netlink event is received.
2024-12-19 00:15:26 [info]dhcp6c_intf_changed() intf changed. caller=0x92be04
2024-12-19 00:15:26 [info]dhcp6c_intf_changed() intf changed. caller=0x92be04
2024-12-19 00:15:26 [info]dhcp6c_intf_changed() intf changed. caller=0x92be04
2024-12-19 00:15:26 [info]dhcp6c_intf_changed() intf changed. caller=0x92be04
2024-12-19 00:15:26 [info]dhcp6c_intf_changed() intf changed. caller=0x92be04
2024-12-19 00:15:26 [info]dhcp6c_intf_changed() intf changed. caller=0x92be04
2024-12-19 00:15:26 [debug]dhcp6_check_timer() called
2024-12-19 00:15:26 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:15:29 [debug]dhcp6_check_timer() called
2024-12-19 00:15:29 [debug]dhcp6_check_timer() timer func=0x926788
2024-12-19 00:15:29 [debug]dhcp6c_resource_update() dhcp6client: checking if we need to reinterfaceure
2024-12-19 00:15:29 [debug]dhcp6c_resource_update() modem is down
2024-12-19 00:15:29 [debug]interface_has_changed() intf wan1 vdom root HA status old:1 new:1 

2024-12-19 00:15:29 [debug]dhcp6_check_timer() called
2024-12-19 00:15:29 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:15:29 [info]dhcp6c_netlink_event_handler() netlink event is received.
2024-12-19 00:15:29 [debug]dhcp6_check_timer() called
2024-12-19 00:15:29 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:15:31 [debug]dhcp6_check_timer() called
2024-12-19 00:15:31 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:15:33 [debug]dhcp6_check_timer() called
2024-12-19 00:15:33 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:15:35 [debug]dhcp6_check_timer() called
2024-12-19 00:15:35 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:15:37 [debug]dhcp6_check_timer() called
2024-12-19 00:15:37 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:15:39 [debug]dhcp6_check_timer() called
2024-12-19 00:15:39 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:15:41 [debug]dhcp6_check_timer() called
2024-12-19 00:15:41 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:15:43 [debug]dhcp6_check_timer() called
2024-12-19 00:15:43 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:15:45 [debug]dhcp6_check_timer() called
2024-12-19 00:15:45 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:15:47 [debug]dhcp6_check_timer() called
2024-12-19 00:15:47 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:15:49 [debug]dhcp6_check_timer() called
2024-12-19 00:15:49 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:15:51 [debug]dhcp6_check_timer() called
2024-12-19 00:15:51 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:15:53 [debug]dhcp6_check_timer() called
2024-12-19 00:15:53 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:15:55 [debug]dhcp6_check_timer() called
2024-12-19 00:15:55 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:15:57 [debug]dhcp6_check_timer() called
2024-12-19 00:15:57 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:15:59 [debug]dhcp6_check_timer() called
2024-12-19 00:15:59 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:16:01 [debug]dhcp6_check_timer() called
2024-12-19 00:16:01 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:16:03 [debug]dhcp6_check_timer() called
2024-12-19 00:16:03 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:16:05 [debug]dhcp6_check_timer() called
2024-12-19 00:16:05 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:16:07 [debug]dhcp6_check_timer() called
2024-12-19 00:16:07 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:16:09 [debug]dhcp6_check_timer() called
2024-12-19 00:16:09 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:16:11 [debug]dhcp6_check_timer() called
2024-12-19 00:16:11 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:16:14 [debug]dhcp6_check_timer() called
2024-12-19 00:16:14 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:16:16 [debug]dhcp6_check_timer() called
2024-12-19 00:16:16 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:16:18 [debug]dhcp6_check_timer() called
2024-12-19 00:16:18 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:16:20 [debug]dhcp6_check_timer() called
2024-12-19 00:16:20 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:16:22 [debug]dhcp6_check_timer() called
2024-12-19 00:16:22 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:16:24 [debug]dhcp6_check_timer() called
2024-12-19 00:16:24 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:16:26 [debug]dhcp6_check_timer() called
2024-12-19 00:16:26 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:16:26 [info]dhcp6c_netlink_event_handler() netlink event is received.
2024-12-19 00:16:26 [info]dhcp6c_intf_changed() intf changed. caller=0x92be04
2024-12-19 00:16:26 [debug]dhcp6_check_timer() called
2024-12-19 00:16:26 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:16:29 [debug]dhcp6_check_timer() called
2024-12-19 00:16:29 [debug]dhcp6_check_timer() timer func=0x926788
2024-12-19 00:16:29 [debug]dhcp6c_resource_update() dhcp6client: checking if we need to reinterfaceure
2024-12-19 00:16:29 [debug]dhcp6c_resource_update() modem is down
2024-12-19 00:16:29 [debug]interface_has_changed() intf wan1 vdom root HA status old:1 new:1 

2024-12-19 00:16:29 [debug]dhcp6_check_timer() called
2024-12-19 00:16:29 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:16:31 [debug]dhcp6_check_timer() called
2024-12-19 00:16:31 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:16:33 [debug]dhcp6_check_timer() called
2024-12-19 00:16:33 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:16:35 [debug]dhcp6_check_timer() called
2024-12-19 00:16:35 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:16:37 [debug]dhcp6_check_timer() called
2024-12-19 00:16:37 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:16:39 [debug]dhcp6_check_timer() called
2024-12-19 00:16:39 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:16:41 [debug]dhcp6_check_timer() called
2024-12-19 00:16:41 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:16:43 [debug]dhcp6_check_timer() called
2024-12-19 00:16:43 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:16:45 [debug]dhcp6_check_timer() called
2024-12-19 00:16:45 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:16:47 [debug]dhcp6_check_timer() called
2024-12-19 00:16:47 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:16:49 [debug]dhcp6_check_timer() called
2024-12-19 00:16:49 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:16:51 [debug]dhcp6_check_timer() called
2024-12-19 00:16:51 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:16:53 [debug]dhcp6_check_timer() called
2024-12-19 00:16:53 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:16:53 [debug]dhcp6_check_timer() called
2024-12-19 00:16:53 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:16:55 [debug]dhcp6_check_timer() called
2024-12-19 00:16:55 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:16:55 [debug]dhcp6_check_timer() called
2024-12-19 00:16:55 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:16:57 [debug]dhcp6_check_timer() called
2024-12-19 00:16:57 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:17:00 [debug]dhcp6_check_timer() called
2024-12-19 00:17:00 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:17:00 [debug]dhcp6_check_timer() called
2024-12-19 00:17:00 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:17:02 [debug]dhcp6_check_timer() called
2024-12-19 00:17:02 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:17:04 [debug]dhcp6_check_timer() called
2024-12-19 00:17:04 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:17:07 [info]dhcp6c_intf_changed() intf changed. caller=0x9273d0
2024-12-19 00:17:07 [debug]dhcp6_check_timer() called
2024-12-19 00:17:07 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:17:07 [info]client6_mainloop() cmdb msg received, sock =5
2024-12-19 00:17:07 [info]dhcp6c_intf_changed() intf changed. caller=0x23f9be8
2024-12-19 00:17:07 [debug]dhcp6_check_timer() called
2024-12-19 00:17:07 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:17:10 [debug]dhcp6_check_timer() called
2024-12-19 00:17:10 [debug]dhcp6_check_timer() timer func=0x926788
2024-12-19 00:17:10 [debug]dhcp6c_resource_update() dhcp6client: checking if we need to reinterfaceure
2024-12-19 00:17:10 [debug]dhcp6c_resource_update() modem is down
2024-12-19 00:17:10 [debug]interface_has_changed() intf wan1 vdom root HA status old:1 new:1 

2024-12-19 00:17:10 [debug]interface_has_changed() new dhcp6c options detected

2024-12-19 00:17:10 [debug]dhcp6c_resource_update() dhcp6client: detected changed interface, reinterfaceuring
2024-12-19 00:17:10 [warning]uncache_intf6_info() delete interface  wan1 cache
2024-12-19 00:17:10 [debug]dhcp6c_intf_get_duid() called
2024-12-19 00:17:10 [debug]dhcp6c_intf_get_duid() Generated a new DUID: 00:03:00:01:04:d5:90:42:86:12.
2024-12-19 00:17:10 [debug]dhcp6_reset_timer() reset a timer on wan1, state=INIT, timeo=0, retrans=459
2024-12-19 00:17:10 [debug]dhcp6_check_timer() called
2024-12-19 00:17:10 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:17:12 [debug]dhcp6_check_timer() called
2024-12-19 00:17:12 [debug]dhcp6_check_timer() timer func=0x9274f0
2024-12-19 00:17:12 [debug]client6_send() a new XID (b4c06e) is generated
2024-12-19 00:17:12 [debug]copy_option() set client ID (len 10)
2024-12-19 00:17:12 [debug]copy_option() set elapsed time (len 2)
2024-12-19 00:17:12 [debug]copy_option() set option request (len 4)
2024-12-19 00:17:12 [debug]copyout_option() set IA_PD
2024-12-19 00:17:12 [debug]client6_send() send solicit to ff02::1:2%wan1
2024-12-19 00:17:12 [debug]dhcp6_reset_timer() reset a timer on wan1, state=SOLICIT, timeo=0, retrans=1046
2024-12-19 00:17:12 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:17:12 [info]dhcp6c_netlink_event_handler() netlink event is received.
2024-12-19 00:17:12 [debug]dhcp6_check_timer() called
2024-12-19 00:17:12 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:17:12 [info]client6_mainloop() msg received, sock =11
2024-12-19 00:17:12 [debug]client6_recv() receive advertise from fe80::7eff:4dff:fe93:9b5%wan1 on wan1
2024-12-19 00:17:12 [debug]dhcp6_get_options() get DHCP option client ID, len 10
2024-12-19 00:17:12 [debug]   DUID: 00:03:00:01:04:d5:90:42:86:12
2024-12-19 00:17:12 [debug]dhcp6_get_options() get DHCP option server ID, len 10
2024-12-19 00:17:12 [debug]   DUID: 00:03:00:01:7c:ff:4d:93:09:b5
2024-12-19 00:17:12 [debug]dhcp6_get_options() get DHCP option preference, len 1
2024-12-19 00:17:12 [debug]   preference: 0
2024-12-19 00:17:12 [debug]dhcp6_get_options() get DHCP option DNS, len 16
2024-12-19 00:17:12 [debug]dhcp6_get_options() get DHCP option opt_86, len 16
2024-12-19 00:17:12 [info]dhcp6_get_options() unknown or unexpected DHCP6 option opt_86, len 16
2024-12-19 00:17:12 [debug]dhcp6_get_options() get DHCP option IA_PD, len 41
2024-12-19 00:17:12 [debug]   IA_PD: ID=5, T1=1800, T2=2880
2024-12-19 00:17:12 [debug]copyin_option() get DHCP option IA_PD prefix, len 25
2024-12-19 00:17:12 [debug]copyin_option()   IA_PD prefix: 2003:c6:af32:84fc::/62 pltime=3600 vltime=7200
2024-12-19 00:17:12 [debug]client6_recvadvert() server ID: 00:03:00:01:7c:ff:4d:93:09:b5, pref=0
2024-12-19 00:17:12 [debug]client6_recvadvert() reset timer for wan1 to 0.991872
2024-12-19 00:17:12 [debug]dhcp6_check_timer() called
2024-12-19 00:17:12 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:17:14 [info]dhcp6c_intf_changed() intf changed. caller=0x9273d0
2024-12-19 00:17:14 [debug]dhcp6_check_timer() called
2024-12-19 00:17:14 [debug]dhcp6_check_timer() timer func=0x9274f0
2024-12-19 00:17:14 [debug]select_server() picked a server (ID: 00:03:00:01:7c:ff:4d:93:09:b5)
2024-12-19 00:17:14 [debug]client6_send() a new XID (f583c6) is generated
2024-12-19 00:17:14 [debug]copy_option() set client ID (len 10)
2024-12-19 00:17:14 [debug]copy_option() set server ID (len 10)
2024-12-19 00:17:14 [debug]copy_option() set elapsed time (len 2)
2024-12-19 00:17:14 [debug]copy_option() set option request (len 4)
2024-12-19 00:17:14 [debug]copyout_option() set IA_PD prefix
2024-12-19 00:17:14 [debug]copyout_option() set IA_PD
2024-12-19 00:17:14 [debug]client6_send() send request to ff02::1:2%wan1
2024-12-19 00:17:14 [debug]dhcp6_reset_timer() reset a timer on wan1, state=REQUEST, timeo=0, retrans=949
2024-12-19 00:17:14 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:17:14 [info]client6_mainloop() cmdb msg received, sock =5
2024-12-19 00:17:14 [info]dhcp6c_intf_changed() intf changed. caller=0x23f9be8
2024-12-19 00:17:14 [debug]dhcp6_check_timer() called
2024-12-19 00:17:14 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:17:14 [info]client6_mainloop() msg received, sock =11
2024-12-19 00:17:14 [debug]client6_recv() receive reply from fe80::7eff:4dff:fe93:9b5%wan1 on wan1
2024-12-19 00:17:14 [debug]dhcp6_get_options() get DHCP option client ID, len 10
2024-12-19 00:17:14 [debug]   DUID: 00:03:00:01:04:d5:90:42:86:12
2024-12-19 00:17:14 [debug]dhcp6_get_options() get DHCP option server ID, len 10
2024-12-19 00:17:14 [debug]   DUID: 00:03:00:01:7c:ff:4d:93:09:b5
2024-12-19 00:17:14 [debug]dhcp6_get_options() get DHCP option preference, len 1
2024-12-19 00:17:14 [debug]   preference: 0
2024-12-19 00:17:14 [debug]dhcp6_get_options() get DHCP option DNS, len 16
2024-12-19 00:17:14 [debug]dhcp6_get_options() get DHCP option opt_86, len 16
2024-12-19 00:17:14 [info]dhcp6_get_options() unknown or unexpected DHCP6 option opt_86, len 16
2024-12-19 00:17:14 [debug]dhcp6_get_options() get DHCP option IA_PD, len 41
2024-12-19 00:17:14 [debug]   IA_PD: ID=5, T1=1800, T2=2880
2024-12-19 00:17:14 [debug]copyin_option() get DHCP option IA_PD prefix, len 25
2024-12-19 00:17:14 [debug]copyin_option()   IA_PD prefix: 2003:c6:af32:84fc::/62 pltime=3600 vltime=7200
2024-12-19 00:17:14 [debug]info_printf() nameserver[0] 2620:fe::fe
2024-12-19 00:17:14 [debug]client6_process() dynamic dns1=[2620:fe::fe]

2024-12-19 00:17:14 [debug]get_ia() make an IA: PD-5
2024-12-19 00:17:14 [debug]update_prefix() create a prefix 2003:c6:af32:84fc::/62 pltime=3600, vltime=7200
2024-12-19 00:17:14 [debug]dhcp6c_ha_sync_send_lease_to() [dhcp6c_ha_sync_send_lease_to] serialno='          (null)', intf='wan1'

2024-12-19 00:17:14 [debug]dhcp6c_ha_sync_send_lease_to() send DHCP6_HA_SYNC_T_LEASE

2024-12-19 00:17:14 [debug]dhcp6_remove_event() removing an event on wan1, state=REQUEST
2024-12-19 00:17:14 [debug]dhcp6_remove_event() removing server (ID: 00:03:00:01:7c:ff:4d:93:09:b5)
2024-12-19 00:17:14 [debug]client6_recvreply() got an expected reply, sleeping.
2024-12-19 00:17:14 [debug]dhcp6_check_timer() called
2024-12-19 00:17:14 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:17:14 [info]dhcp6c_netlink_event_handler() netlink event is received.
2024-12-19 00:17:14 [info]client6_mainloop() cmdb msg received, sock =5
2024-12-19 00:17:14 [info]dhcp6c_intf_changed() intf changed. caller=0x23f9be8
2024-12-19 00:17:14 [debug]dhcp6_check_timer() called
2024-12-19 00:17:14 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:17:14 [info]client6_mainloop() cmdb msg received, sock =5
2024-12-19 00:17:14 [info]dhcp6c_intf_changed() intf changed. caller=0x23f9be8
2024-12-19 00:17:14 [debug]dhcp6_check_timer() called
2024-12-19 00:17:14 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:17:14 [info]dhcp6c_netlink_event_handler() netlink event is received.
2024-12-19 00:17:14 [debug]dhcp6_check_timer() called
2024-12-19 00:17:14 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:17:14 [info]dhcp6c_netlink_event_handler() netlink event is received.
2024-12-19 00:17:14 [info]dhcp6c_nlm_handler() Restart and renew interface wan1.
2024-12-19 00:17:14 [debug]remove_ia() remove an IA: PD-5
2024-12-19 00:17:14 [debug]remove_siteprefix() remove a site prefix 2003:c6:af32:84fc::/62
2024-12-19 00:17:14 [warning]uncache_intf6_info() delete interface  wan1 cache
2024-12-19 00:17:14 [debug]dhcp6c_intf_get_duid() called
2024-12-19 00:17:14 [debug]dhcp6c_intf_get_duid() Generated a new DUID: 00:03:00:01:04:d5:90:42:86:12.
2024-12-19 00:17:14 [debug]dhcp6_reset_timer() reset a timer on wan1, state=INIT, timeo=0, retrans=601
2024-12-19 00:17:14 [info]dhcp6c_nlm_handler() Restart and renew interface wan1.
2024-12-19 00:17:14 [debug]dhcp6_remove_event() removing an event on wan1, state=INIT
2024-12-19 00:17:14 [warning]uncache_intf6_info() delete interface  wan1 cache
2024-12-19 00:17:14 [debug]dhcp6c_intf_get_duid() called
2024-12-19 00:17:14 [debug]dhcp6c_intf_get_duid() Generated a new DUID: 00:03:00:01:04:d5:90:42:86:12.
2024-12-19 00:17:14 [debug]dhcp6_reset_timer() reset a timer on wan1, state=INIT, timeo=0, retrans=749
2024-12-19 00:17:14 [info]dhcp6c_intf_changed() intf changed. caller=0x92be04
2024-12-19 00:17:14 [info]dhcp6c_intf_changed() intf changed. caller=0x92be04
2024-12-19 00:17:14 [info]dhcp6c_intf_changed() intf changed. caller=0x92be04
2024-12-19 00:17:14 [debug]dhcp6_check_timer() called
2024-12-19 00:17:14 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:17:14 [info]client6_mainloop() cmdb msg received, sock =5
2024-12-19 00:17:14 [info]dhcp6c_intf_changed() intf changed. caller=0x23f9be8
2024-12-19 00:17:14 [debug]dhcp6_check_timer() called
2024-12-19 00:17:14 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:17:16 [info]dhcp6c_netlink_event_handler() netlink event is received.
2024-12-19 00:17:16 [info]dhcp6c_intf_changed() intf changed. caller=0x92be04
2024-12-19 00:17:16 [debug]dhcp6_check_timer() called
2024-12-19 00:17:16 [debug]dhcp6_check_timer() timer func=0x9274f0
2024-12-19 00:17:16 [debug]client6_send() a new XID (620a9f) is generated
2024-12-19 00:17:16 [debug]copy_option() set client ID (len 10)
2024-12-19 00:17:16 [debug]copyout_option() set identity association
2024-12-19 00:17:16 [debug]copy_option() set elapsed time (len 2)
2024-12-19 00:17:16 [debug]copy_option() set option request (len 4)
2024-12-19 00:17:16 [debug]copyout_option() set IA_PD
2024-12-19 00:17:16 [debug]client6_send() send solicit to ff02::1:2%wan1
2024-12-19 00:17:16 [debug]dhcp6_reset_timer() reset a timer on wan1, state=SOLICIT, timeo=0, retrans=1085
2024-12-19 00:17:16 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:17:16 [info]client6_mainloop() msg received, sock =11
2024-12-19 00:17:16 [debug]client6_recv() receive advertise from fe80::7eff:4dff:fe93:9b5%wan1 on wan1
2024-12-19 00:17:16 [debug]dhcp6_get_options() get DHCP option client ID, len 10
2024-12-19 00:17:16 [debug]   DUID: 00:03:00:01:04:d5:90:42:86:12
2024-12-19 00:17:16 [debug]dhcp6_get_options() get DHCP option server ID, len 10
2024-12-19 00:17:16 [debug]   DUID: 00:03:00:01:7c:ff:4d:93:09:b5
2024-12-19 00:17:16 [debug]dhcp6_get_options() get DHCP option preference, len 1
2024-12-19 00:17:16 [debug]   preference: 0
2024-12-19 00:17:16 [debug]dhcp6_get_options() get DHCP option DNS, len 16
2024-12-19 00:17:16 [debug]dhcp6_get_options() get DHCP option opt_86, len 16
2024-12-19 00:17:16 [info]dhcp6_get_options() unknown or unexpected DHCP6 option opt_86, len 16
2024-12-19 00:17:16 [debug]dhcp6_get_options() get DHCP option identity association, len 40
2024-12-19 00:17:16 [debug]   IA_NA: ID=5, T1=1800, T2=2880
2024-12-19 00:17:16 [debug]copyin_option() get DHCP option IA address, len 24
2024-12-19 00:17:16 [debug]copyin_option()   IA_NA address: 2003:c6:af32:8400:6d5:90ff:fe42:8612 pltime=3600 vltime=7200
2024-12-19 00:17:16 [debug]dhcp6_get_options() get DHCP option IA_PD, len 41
2024-12-19 00:17:16 [debug]   IA_PD: ID=5, T1=1800, T2=2880
2024-12-19 00:17:16 [debug]copyin_option() get DHCP option IA_PD prefix, len 25
2024-12-19 00:17:16 [debug]copyin_option()   IA_PD prefix: 2003:c6:af32:84fc::/62 pltime=3600 vltime=7200
2024-12-19 00:17:16 [debug]client6_recvadvert() server ID: 00:03:00:01:7c:ff:4d:93:09:b5, pref=0
2024-12-19 00:17:16 [debug]client6_recvadvert() reset timer for wan1 to 0.988651
2024-12-19 00:17:16 [debug]dhcp6_check_timer() called
2024-12-19 00:17:16 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:17:19 [debug]dhcp6_check_timer() called
2024-12-19 00:17:19 [debug]dhcp6_check_timer() timer func=0x9274f0
2024-12-19 00:17:19 [debug]select_server() picked a server (ID: 00:03:00:01:7c:ff:4d:93:09:b5)
2024-12-19 00:17:19 [debug]client6_send() a new XID (ad2c8a) is generated
2024-12-19 00:17:19 [debug]copy_option() set client ID (len 10)
2024-12-19 00:17:19 [debug]copy_option() set server ID (len 10)
2024-12-19 00:17:19 [debug]copyout_option() set IA address
2024-12-19 00:17:19 [debug]copyout_option() set identity association
2024-12-19 00:17:19 [debug]copy_option() set elapsed time (len 2)
2024-12-19 00:17:19 [debug]copy_option() set option request (len 4)
2024-12-19 00:17:19 [debug]copyout_option() set IA_PD prefix
2024-12-19 00:17:19 [debug]copyout_option() set IA_PD
2024-12-19 00:17:19 [debug]client6_send() send request to ff02::1:2%wan1
2024-12-19 00:17:19 [debug]dhcp6_reset_timer() reset a timer on wan1, state=REQUEST, timeo=0, retrans=1022
2024-12-19 00:17:19 [debug]dhcp6_check_timer() timer func=0x926788
2024-12-19 00:17:19 [debug]dhcp6c_resource_update() dhcp6client: checking if we need to reinterfaceure
2024-12-19 00:17:19 [debug]dhcp6c_resource_update() modem is down
2024-12-19 00:17:19 [debug]interface_has_changed() intf wan1 vdom root HA status old:1 new:1 

2024-12-19 00:17:19 [debug]dhcp6_check_timer() called
2024-12-19 00:17:19 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:17:19 [info]client6_mainloop() msg received, sock =11
2024-12-19 00:17:19 [debug]client6_recv() receive reply from fe80::7eff:4dff:fe93:9b5%wan1 on wan1
2024-12-19 00:17:19 [debug]dhcp6_get_options() get DHCP option client ID, len 10
2024-12-19 00:17:19 [debug]   DUID: 00:03:00:01:04:d5:90:42:86:12
2024-12-19 00:17:19 [debug]dhcp6_get_options() get DHCP option server ID, len 10
2024-12-19 00:17:19 [debug]   DUID: 00:03:00:01:7c:ff:4d:93:09:b5
2024-12-19 00:17:19 [debug]dhcp6_get_options() get DHCP option preference, len 1
2024-12-19 00:17:19 [debug]   preference: 0
2024-12-19 00:17:19 [debug]dhcp6_get_options() get DHCP option DNS, len 16
2024-12-19 00:17:19 [debug]dhcp6_get_options() get DHCP option opt_86, len 16
2024-12-19 00:17:19 [info]dhcp6_get_options() unknown or unexpected DHCP6 option opt_86, len 16
2024-12-19 00:17:19 [debug]dhcp6_get_options() get DHCP option identity association, len 40
2024-12-19 00:17:19 [debug]   IA_NA: ID=5, T1=1800, T2=2880
2024-12-19 00:17:19 [debug]copyin_option() get DHCP option IA address, len 24
2024-12-19 00:17:19 [debug]copyin_option()   IA_NA address: 2003:c6:af32:8400:6d5:90ff:fe42:8612 pltime=3600 vltime=7200
2024-12-19 00:17:19 [debug]dhcp6_get_options() get DHCP option IA_PD, len 41
2024-12-19 00:17:19 [debug]   IA_PD: ID=5, T1=1800, T2=2880
2024-12-19 00:17:19 [debug]copyin_option() get DHCP option IA_PD prefix, len 25
2024-12-19 00:17:19 [debug]copyin_option()   IA_PD prefix: 2003:c6:af32:84fc::/62 pltime=3600 vltime=7200
2024-12-19 00:17:19 [debug]info_printf() nameserver[0] 2620:fe::fe
2024-12-19 00:17:19 [debug]client6_process() dynamic dns1=[2620:fe::fe]

2024-12-19 00:17:19 [debug]get_ia() make an IA: PD-5
2024-12-19 00:17:19 [debug]update_prefix() create a prefix 2003:c6:af32:84fc::/62 pltime=3600, vltime=7200
2024-12-19 00:17:19 [debug]get_ia() make an IA: NA-5
2024-12-19 00:17:19 [debug]update_address() create an address 2003:c6:af32:8400:6d5:90ff:fe42:8612 pltime=3600, vltime=7200
2024-12-19 00:17:19 [debug]dhcp6c_ha_sync_send_lease_to() [dhcp6c_ha_sync_send_lease_to] serialno='          (null)', intf='wan1'

2024-12-19 00:17:19 [debug]dhcp6c_ha_sync_send_lease_to() send DHCP6_HA_SYNC_T_LEASE

2024-12-19 00:17:19 [debug]dhcp6_remove_event() removing an event on wan1, state=REQUEST
2024-12-19 00:17:19 [debug]dhcp6_remove_event() removing server (ID: 00:03:00:01:7c:ff:4d:93:09:b5)
2024-12-19 00:17:19 [debug]client6_recvreply() got an expected reply, sleeping.
2024-12-19 00:17:19 [debug]dhcp6_check_timer() called
2024-12-19 00:17:19 [info]client6_mainloop() timeout=10 sec, cfd=5, kfd=6
2024-12-19 00:17:19 [info]dhcp6c_netlink_event_handler() netlink event is received.
2024-12-19 00:17:19 [info]client6_mainloop() cmdb msg received, sock =5
2024-12-19 00:17:19 [info]dhcp6c_intf_changed() intf changed. caller=0x23f9be8
2024-12-19 00:17:19 [debug]dhcp6_check_timer() called
2024-12-19 00:17:19 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:17:19 [info]client6_mainloop() cmdb msg received, sock =5
2024-12-19 00:17:19 [info]dhcp6c_intf_changed() intf changed. caller=0x23f9be8
2024-12-19 00:17:19 [debug]dhcp6_check_timer() called
2024-12-19 00:17:19 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:17:19 [info]client6_mainloop() cmdb msg received, sock =5
2024-12-19 00:17:19 [info]dhcp6c_intf_changed() intf changed. caller=0x23f9be8
2024-12-19 00:17:19 [debug]dhcp6_check_timer() called
2024-12-19 00:17:19 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:17:19 [info]dhcp6c_netlink_event_handler() netlink event is received.
2024-12-19 00:17:19 [debug]dhcp6_check_timer() called
2024-12-19 00:17:19 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:17:19 [info]dhcp6c_netlink_event_handler() netlink event is received.
2024-12-19 00:17:19 [debug]dhcp6_check_timer() called
2024-12-19 00:17:19 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:17:19 [info]dhcp6c_netlink_event_handler() netlink event is received.
2024-12-19 00:17:19 [debug]dhcp6_check_timer() called
2024-12-19 00:17:19 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:17:20 [info]dhcp6c_netlink_event_handler() netlink event is received.
2024-12-19 00:17:20 [info]dhcp6c_intf_changed() intf changed. caller=0x92be04
2024-12-19 00:17:20 [debug]dhcp6_check_timer() called
2024-12-19 00:17:20 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:17:20 [info]dhcp6c_netlink_event_handler() netlink event is received.
2024-12-19 00:17:20 [info]dhcp6c_intf_changed() intf changed. caller=0x92be04
2024-12-19 00:17:20 [debug]dhcp6_check_timer() called
2024-12-19 00:17:20 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:17:23 [debug]dhcp6_check_timer() called
2024-12-19 00:17:23 [debug]dhcp6_check_timer() timer func=0x926788
2024-12-19 00:17:23 [debug]dhcp6c_resource_update() dhcp6client: checking if we need to reinterfaceure
2024-12-19 00:17:23 [debug]dhcp6c_resource_update() modem is down
2024-12-19 00:17:23 [debug]interface_has_changed() intf wan1 vdom root HA status old:1 new:1 

2024-12-19 00:17:23 [debug]dhcp6_check_timer() called
2024-12-19 00:17:23 [info]client6_mainloop() timeout=10 sec, cfd=5, kfd=6
2024-12-19 00:17:30 [info]dhcp6c_netlink_event_handler() netlink event is received.
2024-12-19 00:17:30 [debug]dhcp6_check_timer() called
2024-12-19 00:17:30 [info]client6_mainloop() timeout=10 sec, cfd=5, kfd=6
2024-12-19 00:17:41 [debug]dhcp6_check_timer() called
2024-12-19 00:17:41 [info]client6_mainloop() timeout=10 sec, cfd=5, kfd=6
2024-12-19 00:17:52 [debug]dhcp6_check_timer() called
2024-12-19 00:17:52 [info]client6_mainloop() timeout=10 sec, cfd=5, kfd=6
2024-12-19 00:17:56 [info]dhcp6c_netlink_event_handler() netlink event is received.
2024-12-19 00:17:56 [info]dhcp6c_intf_changed() intf changed. caller=0x92be04
2024-12-19 00:17:56 [info]dhcp6c_intf_changed() intf changed. caller=0x92be04
2024-12-19 00:17:56 [info]dhcp6c_intf_changed() intf changed. caller=0x92be04
2024-12-19 00:17:56 [info]dhcp6c_intf_changed() intf changed. caller=0x92be04
2024-12-19 00:17:56 [debug]dhcp6_check_timer() called
2024-12-19 00:17:56 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:17:59 [debug]dhcp6_check_timer() called
2024-12-19 00:17:59 [debug]dhcp6_check_timer() timer func=0x926788
2024-12-19 00:17:59 [debug]dhcp6c_resource_update() dhcp6client: checking if we need to reinterfaceure
2024-12-19 00:17:59 [debug]dhcp6c_resource_update() modem is down
2024-12-19 00:17:59 [debug]interface_has_changed() intf wan1 vdom root HA status old:1 new:1 

2024-12-19 00:17:59 [debug]dhcp6_check_timer() called
2024-12-19 00:17:59 [info]client6_mainloop() timeout=10 sec, cfd=5, kfd=6
2024-12-19 00:18:10 [debug]dhcp6_check_timer() called
2024-12-19 00:18:10 [info]client6_mainloop() timeout=10 sec, cfd=5, kfd=6
2024-12-19 00:18:15 [info]dhcp6c_netlink_event_handler() netlink event is received.
2024-12-19 00:18:15 [debug]dhcp6_check_timer() called
2024-12-19 00:18:15 [info]client6_mainloop() timeout=10 sec, cfd=5, kfd=6
2024-12-19 00:18:15 [info]dhcp6c_netlink_event_handler() netlink event is received.
2024-12-19 00:18:15 [debug]dhcp6_check_timer() called
2024-12-19 00:18:15 [info]client6_mainloop() timeout=10 sec, cfd=5, kfd=6
2024-12-19 00:18:25 [debug]dhcp6_check_timer() called
2024-12-19 00:18:25 [info]client6_mainloop() timeout=10 sec, cfd=5, kfd=6
2024-12-19 00:18:26 [info]dhcp6c_netlink_event_handler() netlink event is received.
2024-12-19 00:18:26 [info]dhcp6c_intf_changed() intf changed. caller=0x92be04
2024-12-19 00:18:26 [info]dhcp6c_intf_changed() intf changed. caller=0x92be04
2024-12-19 00:18:26 [debug]dhcp6_check_timer() called
2024-12-19 00:18:26 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:18:29 [debug]dhcp6_check_timer() called
2024-12-19 00:18:29 [debug]dhcp6_check_timer() timer func=0x926788
2024-12-19 00:18:29 [debug]dhcp6c_resource_update() dhcp6client: checking if we need to reinterfaceure
2024-12-19 00:18:29 [debug]dhcp6c_resource_update() modem is down
2024-12-19 00:18:29 [debug]interface_has_changed() intf wan1 vdom root HA status old:1 new:1 

2024-12-19 00:18:29 [debug]dhcp6_check_timer() called
2024-12-19 00:18:29 [info]client6_mainloop() timeout=10 sec, cfd=5, kfd=6
2024-12-19 00:18:40 [debug]dhcp6_check_timer() called
2024-12-19 00:18:40 [info]client6_mainloop() timeout=10 sec, cfd=5, kfd=6
2024-12-19 00:18:51 [debug]dhcp6_check_timer() called
2024-12-19 00:18:51 [info]client6_mainloop() timeout=10 sec, cfd=5, kfd=6
2024-12-19 00:18:56 [info]dhcp6c_netlink_event_handler() netlink event is received.
2024-12-19 00:18:56 [info]dhcp6c_intf_changed() intf changed. caller=0x92be04
2024-12-19 00:18:56 [info]dhcp6c_intf_changed() intf changed. caller=0x92be04
2024-12-19 00:18:56 [debug]dhcp6_check_timer() called
2024-12-19 00:18:56 [info]client6_mainloop() timeout=2 sec, cfd=5, kfd=6
2024-12-19 00:18:59 [debug]dhcp6_check_timer() called
2024-12-19 00:18:59 [debug]dhcp6_check_timer() timer func=0x926788
2024-12-19 00:18:59 [debug]dhcp6c_resource_update() dhcp6client: checking if we need to reinterfaceure
2024-12-19 00:18:59 [debug]dhcp6c_resource_update() modem is down
2024-12-19 00:18:59 [debug]interface_has_changed() intf wan1 vdom root HA status old:1 new:1 

2024-12-19 00:18:59 [debug]dhcp6_check_timer() called
2024-12-19 00:18:59 [info]client6_mainloop() timeout=10 sec, cfd=5, kfd=6
2024-12-19 00:19:10 [debug]dhcp6_check_timer() called
2024-12-19 00:19:10 [info]client6_mainloop() timeout=10 sec, cfd=5, kfd=6
diagnose debug disable

FortiGate-60F #

Finally, this Wireshark trace shows the client’s view. The FortiGate does not respond to router solicitations (RS, packets nr. 5, 12, 19), hence the real IPv6 process kicks in after it has received the first regular router advertisement (RA, nr. 43): [This trace was made while I had the O-flag still set to enable. Sorry for that.]

That’s it. Happy networking. ;)

Soli Deo Gloria!

Photo by Nick Fewings on Unsplash.

You can use a FortiGate to connect to the Internet (that is: Dual-Stack!) directly in various ways. In my current setup, I’m using a PPPoE residential xDLS connection. It’s not that easy to configure everything correctly since it requires the use of many different protocols such as PPPoE and PPPoEv6 (PPP IPV6CP) along with DHCPv6-PD. But here it is:

Please refer to this post about DHCPv6-PD in general and this one about configuring DHCPv6-PD on a FortiGate. I’m using a FortiGate FG-60F with FortiOS 7.6.1, which is connected through a DrayTek Vigor167 modem to the German ISP “Deutsche Telekom” on wan1. No VLAN configuration is needed since the DSL modem already encapsulates the traffic within VLAN 7 on the ISP side.

The technical steps work as follows:

• the FortiGate connects/authenticates via PPPoE stuff to the ISP’s router
• it gets the single IPv4 address from the ISP via PPP IPCP
• for IPv6, the FortiGate
  1. generates its own “wan1” IPv6 address via SLAAC (autoconf),
  2. uses DHCPv6-PD to get an IPv6 prefix from the ISP,
  3. and shall propose SLAAC on the “internal” interface for its clients with one /64 out of this delegated prefix. <- This is the tricky part because you have to use the correct “delegated-prefix-iaid” IDs along with the correct flags for the RA (no M- nor O-flag within the RA, but the L- and A-flags for the prefix option).
• At least in FortiOS 7.6.1, there is a bug in which the FortiGate does not reply to RSs with RAs. That is: Your clients won’t get any IPv6 GUA address until the first RA that is sent regularly by the Forti is received by those clients. As a workaround, I’ve set the min- and max-interval values to 10, respectively 30 seconds. Staying with the defaults that are way higher (600 seconds = 10 minutes), clients would have to wait really long until IPv6 is up and running.
• Another bug is related to the output of the routing table for IPv6 since it does not show the default route that is gathered from the RA on the wan1 interface. Funnily, this is only true for the “get router info6 routing-table” but not for the “diagnose ipv6 route list” output.

Side note: Unfortunately, we don’t have static IP addresses nor static IPv6 subnets on most German residential ISPs. Hence, after every DSL reconnect or firewall reboot, we’ll get new public IPv4/IPv6 addresses along with a new IPv6 prefix. 🤦

A little GUI, please

As always when using IPv6 on the FortiGates, very little can be done through the GUI. (This is a major drawback compared to a Palo Alto Networks firewall, where you can configure everything through the GUI perfectly.) At least you can configure the PPPoE username/password, which is used for legacy IP and for IPv6 as well. Nice. Note the “IAPD 5 prefix hint” which is used by default on this FortiOS version. I’ve no idea why Fortinet has chosen the “5” as the default ID, but never mind.

Some more CLI, unfortunately

The big magic is done via these CLI commands. Good luck. ;) It was really hard to find out the correct values for all those commands, especially those “set ip6-subnet ::1/64” for the internal interface while the “set subnet ::/64” for the ip6-delegated-prefix-list, and so on.

config system interface
    edit "wan1"
        set vdom "root"
        set mode pppoe
        set allowaccess ping
        set role wan
        config ipv6
            set ip6-mode pppoe
            set ip6-allowaccess ping
            set dhcp6-prefix-delegation enable
            set autoconf enable
            config dhcp6-iapd-list
                edit 5
                next
            end
        end
        set username "001234567890#098765432123#0001@t-online.de"
        set password ENC ThisIsNotMyRealPasswordOfCourse//
    next
    edit "internal"
        set vdom "root"
        set ip 192.168.1.99 255.255.255.0
        set allowaccess ping https ssh fabric
        set role lan
        config ipv6
            set ip6-mode delegated
            set ip6-allowaccess ping https ssh
            set ip6-send-adv enable
            set ip6-max-interval 30
            set ip6-min-interval 10
            set ip6-delegated-prefix-iaid 5
            set ip6-upstream-interface "wan1"
            set ip6-subnet ::1/64
            config ip6-delegated-prefix-list
                edit 1
                    set upstream-interface "wan1"
                    set delegated-prefix-iaid 5
                    set subnet ::/64
                    set rdnss-service delegated
                next
            end
        end
    next
end

Runtime IP Addresses

The Interfaces Overview shows the public IPv4 and IPv6 addresses on the wan1 interface as well as the derived IPv6 address on the internal subnet:

Funnily, the Routing pane lists the default IPv4 route but not the IPv6 one. 🤦‍♂️

Packets Don’t Lie: Wireshark

Looking on the wan1 cable (captured with a real network TAP, the ProfiShark 1G), you can see the PPPoE authentication process along with the legacy IP address through PPP IPCP:

Followed by RAs from the ISP and the DHCPv6-PD process, which hands out the delegated /56 prefix along with two recursive DNS servers:

Captured on the internal interface (to be precise, on a Windows machine connected to the FortiGate), you can see the RSes sent from that client that remain unanswered (packets 14, 35, 114), followed by an RA from the FortiGate (nr. 517) that corretly contains the first /64 prefix out of the delegated /56 prefix along with the recursive DNS servers as well:

This is what an “ipconfig/all” looked like on that Windows PC during that time. You can spot all three IPv6 addresses (1x LL, 2x GUA) that are visible as DAD messages in the Wireshark screenshot as well:

Ethernet-Adapter Ethernet 5:

   Verbindungsspezifisches DNS-Suffix:
   Beschreibung. . . . . . . . . . . : Realtek USB GbE Family Controller #2
   Physische Adresse . . . . . . . . : 00-E0-4C-68-66-C1
   DHCP aktiviert. . . . . . . . . . : Ja
   Autokonfiguration aktiviert . . . : Ja
   IPv6-Adresse. . . . . . . . . . . : 2003:c6:af35:ed00:a1c1:25c8:2189:8512(Bevorzugt)
   Temporäre IPv6-Adresse. . . . . . : 2003:c6:af35:ed00:c51:106e:6fa1:5075(Bevorzugt)
   Verbindungslokale IPv6-Adresse  . : fe80::7df5:d240:f60d:4ae6%15(Bevorzugt)
   IPv4-Adresse  . . . . . . . . . . : 192.168.1.111(Bevorzugt)
   Subnetzmaske  . . . . . . . . . . : 255.255.255.0
   Lease erhalten. . . . . . . . . . : Donnerstag, 19. Dezember 2024 15:10:41
   Lease läuft ab. . . . . . . . . . : Donnerstag, 26. Dezember 2024 15:10:41
   Standardgateway . . . . . . . . . : fe80::6d5:90ff:fe42:8615%15
                                       192.168.1.99
   DHCP-Server . . . . . . . . . . . : 192.168.1.99
   DHCPv6-IAID . . . . . . . . . . . : 704700492
   DHCPv6-Client-DUID. . . . . . . . : 00-01-00-01-2A-C5-E5-11-9C-B6-D0-3E-7D-72
   DNS-Server  . . . . . . . . . . . : 192.168.178.1
                                       2003:180:2:8000::53
                                       2003:180:2:8100::53
   NetBIOS über TCP/IP . . . . . . . : Aktiviert

Soli Deo Gloria!

Photo by James Balensiefen on Unsplash.