Triggered by a customer who had problems getting enough speed through an IPsec site-to-site VPN tunnel between FortiGate firewalls I decided to test different encryption/hashing algorithms to verify the network throughput. I used two FortiWiFi 90D firewalls that have an official IPsec VPN throughput of 1 Gbps. Using Iperf I measured the transfer rates with no VPN tunnel as well as with different IPsec proposals.
I first ran into really slow performances which were related to the default “Software Switch” on the FortiGate. After deleting this type of logical switch, the VPN throughput was almost as expected.
Lab
My lab consists of the following components:
Both FortiWiFi 90D firewalls had the firmware version v5.2.5, build701. The two notebooks were booted with Knoppix 7.6.1 and used Iperf version 2.0.5. The “left” machine ran as the server with either:
iperf -s iperf -s -u
while the “right” machine started Iperf with the following commands for different TCP and UDP tests:
iperf -c 192.168.10.10 -r iperf -c 192.168.10.10 -r -P 8 iperf -c 192.168.10.10 -r -u -b 1000M
I tested the throughput without a VPN at all (only routing) and with a few different proposals (see table below). The Diffie-Hellman group for PFS was always set to 14. This is not related to the test results because it is only used for the key establishment and not for the actual symmetric encryption of the traffic.
I also switched the offloading of encryption to “enable” (refer to the Hardware Acceleration Guide), which did not change anything, either.
config system npu
set enc-offload-antireplay enable
end
Furthermore, I tested the differences between a normal TCP test and the manual set of the TCP window size and buffer length with “-w 512k -l 512k”, such as shown here or here. But this made no differences, too, since Knoppix Linux seems to auto set the window size pretty optimal.
Results
These are the results. The first four tests are without a VPN. While the first two are without routing (simply plugged in both clients into the same software switch on the FortiGate), tests 3 & 4 are routed through the FortiGates. This was the first time at which I was really shocked about the bad performance of only 180 Mbit/s routing speed. Furthermore, almost all IPsec proposals ran at a speed of 86 MBit/s, which is only 9 % of the IPsec throughput listed in the data sheet.
| Proposals | TCP Tx/Rx [MBit/s] | TCP Tx/Rx [MBit/s] | UDP Tx/Rx [MBit/s] |
|---|---|---|---|
| IPerf Options | -r | -r -P 8 | -u -r -b 1000M |
| Same Software Switch H - FGSW - H | 942/937 | 941/936 | 807/805 |
| Same Software Switch + Hardware Switch H - FGSW - SW - H | 942/936 | 941/936 | 807/804 |
| No VPN, only Routing FortiGate directly H - FG - FG - H | 155/177 | 151/168 | 211/206 |
| No VPN, only Routing H - FG - SW - FG - H | 155/177 | 152/168 | 211/210 |
| DES-MD5 | 86/86 | 83/82 | 93/94 |
| 3DES-MD5 | 86/86 | 83/83 | 93/94 |
| 3DES-SHA1 | 86/86 | 83/83 | 95/94 |
| AES128-SHA1 | 86/86 | 83/83 | 88/87 |
| AES256-SHA256 | 86/86 | 122/133 | 93/93 |
| AES256-SHA512 | 85/85 | 80/80 | 84/92 |
The software switch was the problem!
After hours of investigating the slow VPN speed results, I tested the VPN without the software switch on the network ports side, which led to the following results (first column with a “Hardware Switch”, second column with a single interface):
| Proposals | Hardware Switch TCP Tx/Rx [MBit/s] | Single Interface TCP Tx/Rx [MBit/s] |
|---|---|---|
| Iperf Options | -r | -r |
| No VPN, only Routing H - FG - SW - FG - H | 937/937 | 933/932 |
| DES-MD5 | 852/840 | 845/839 |
| 3DES-SHA1 | 707/642 | 701/634 |
| AES128-SHA1 | 825/835 | 826/830 |
| AES256-SHA1 | 820/830 | 816/825 |
| AES256-SHA256 | 723/819 | 814/825 |
| AES256-SHA512 | 637/808 | 812/810 |
Now the speed was quite acceptable, for the mere routing as well as for the VPN throughput. 940 MBit/s for routing through both FortiGate is almost realistic for TCP, and about 830 MBit/s for VPN encryption/decryption is realistic, too.
Here are the “single interface” results in a graph. Only the 3DES tests are a bit slower than all the other ones:
Conclusion
Well, it was my fault that I left the default software switch in place. I should have know better. However, it was the default setting on this FortiWiFi devices.
At the end, the VPN throughput between those FortiGates was really acceptable.