In-depth look at Fortigate's ADVPN - Part II
In part I, we have configured dial-up IPSec tunnel at the Hub1 and eliminated any configuration change required at the Hub/HQ site when a new Spoke/Branch is added to the network. But there is a limitation. That is even though we have achieved configuration flexibility, our underlying topology is still hub-and-spoke. All spoke-to-spoke communication goes through the hub first.
With ADVPN we can remove that limitation. We can achieve a fully meshed network by using ADVPN (Auto Discovery VPN). How ???
If someone is familiar with Cisco's DMVPN, the concepts are same here. Instead Fortinet's calls their implementation ADVPN.
 |
| ADVPN Physical Topology |
 |
| ADVPN Routing Topology |
For example, when Spoke2 tries to communicate with Spoke3 - as usual traffic goes to the Hub1. Hub1 knows the whole network topology. When Hub1 sees that two spokes are trying to communicate with each other, it will intervene. By using ADVPN - it will signal Spoke2 that there is a better path and that is - Spoke2 communicates with Spoke3 directly. Hub1 passes all the required informations to create a new IPSec tunnel between Spoke2 and Spoke3. Then Spoke2 creates a new IPSec site-to-site tunnel with Spoke3 and dynamic routing protocol (OSPF) takes care of the routing. All this signaling happens with fortinet's proprietary extensions to IPSec protocol. So, with ADVPN we can convert a hub-and-spoke topology to fully-meshed topology with minimum configuration effort. After the shortcut IPSec tunnel creation the routing topology looks like below -
 | |||||||||||||||||||||||||
| ADVPN Routing Topology With Shortcut Tunnels
The IP adressering is given in the table below -
|
Hub1 Configuration
Let's look at Hub1's physical interface and IP address configuration -
config system interface
!!!Lan facing interface
edit "port7"
set vdom "root"
set ip 192.168.1.1 255.255.255.0
set allowaccess ping
set type physical
set description "Lan"
set alias "Lan"
next
!!!Internet facing interface
edit "port8"
set vdom "root"
set ip 172.16.0.2 255.255.255.252
set allowaccess ping
set type physical
set description "Internet"
set alias "Internet"
next
!!!Loopback interface, used for router-id in OSPF
edit "Loopback"
set vdom "root"
set ip 192.168.0.1 255.255.255.255
set allowaccess ping
set type loopback
set description "Loopback"
set alias "Loopback"
set role lan
next
end
Now we will configure the IPSec phase-1 at the Hub1 with ADVPN.
config vpn ipsec phase1-interface
edit "Hub1-Phase1"
set ike-version 2
set proposal des-md5
set dhgrp 5
set authmethod psk
set psksecret test123
set nattraversal disable
set keylife 86400
set dpd on-idle
set dpd-retrycount 3
set dpd-retryinterval 20
!!!Internet facing interface
set interface "port8"
!!!It is a dynamic (dial-up) tunnel which accepts negotiation from any spokes.
set type dynamic
set peertype any
!!!Activating fortigate's new dial-up logic
set net-device disable
!!!To find which tunnel to use when connecting to a spoke
set tunnel-search nexthop
!!!No reverse route injection, routing table will be built by routing protocol
set add-route disable
!!!Now we will activate ADVPN with one single command.
!!!As the Hub1 knows full topology it will be always sender in ADVPN
!!!Hub1 sends all the necessary informations to Spokes to that they can create point-to-point IPSec tunnel between them.
set auto-discovery-sender enable
next
end
Next we will configure IPSec phase-2 at the Hub1.
config vpn ipsec phase2-interface
edit "Hub1-Phase2"
set encapsulation tunnel-mode
set proposal des-md5
set pfs disable
set keylife-type seconds
set keylifeseconds 43200
set keepalive disable
!!!Connecting phase-2 with phase-1
set phase1name "Hub1-Phase1"
next
end
As we will be running OSPF, we need to assign an IP adress to Hub1's IPSec tunnel interface. We have reserved 192.168.10.0/24 network as our overlay network. Every device participating in dial-up IPSec VPN will get an IP adress from this network for their tunnel interface.
config system interface
edit "Hub1-Phase1"
set vdom "root"
set ip 192.168.10.1 255.255.255.255
set allowaccess ping
set type tunnel
!!!In fortigate remote IP needs to an IP adress from 192.168.10.0/24 network.
!!!For clarity, in Hub we will use 192.168.10.254/24, an unused IP adress from overlay network.
!!!And in Spokes we will use Hub's tunnel IP as remote IP.
set remote-ip 192.168.10.254 255.255.255.0
set snmp-index 10
set interface "port8"
next
end
Now we will configure the firewall policies. Our policy is very simple - we will allow everything from Hub1's LAN to Spoke's LAN and vice versa. Also, we will allow traffic between Spokes which goes through the Hub.
For configuration flexibility we have assigned our interfaces to different zones which are pretty self-explanatory.
config system zone
edit "Lan-Zone"
set interface "port7"
next
edit "Vpn-Zone"
set interface "Hub1-Phase1"
next
edit "Internet-Zone"
set interface "port8"
next
end
config firewall policy
edit 1
set srcintf "Lan-Zone"
set dstintf "Vpn-Zone"
set srcaddr "all"
set dstaddr "all"
set action accept
set schedule "always"
set service "ALL"
next
edit 2
set srcintf "Vpn-Zone"
set dstintf "Lan-Zone"
set srcaddr "all"
set dstaddr "all"
set action accept
set schedule "always"
set service "ALL"
next
edit 3
!!!Allows traffic between spokes which goes through Hub1
set srcintf "Vpn-Zone"
set dstintf "Vpn-Zone"
set srcaddr "all"
set dstaddr "all"
set action accept
set schedule "always"
set service "ALL"
next
end
Now we will configure OSPF routing as below (remember Hub's OSPF interface is point-to-multipoint type - Hub1 connects to different Spokes).
config router ospf
!!!Loopback address is router-id
set router-id 192.168.0.1
config area
edit 0.0.0.0
next
end
config ospf-interface
edit "Hub1-Phase1"
!!!Tunnel interface will have some OSPF settings applied to it
set interface "Hub1-Phase1"
!!!OSPF network type will be point-to-multipoint
set network-type point-to-multipoint
!!!P-to-MP interface has a default hello interval of 30 sec and dead interval of 120 sec
!!!We will change it to 10 and 40 which is default equivalent of P-to-P interface.
!!!So, that hello and dead timers are consistent between Hub and Spoke
set dead-interval 40
set hello-interval 10
set mtu-ignore enable
next
end
config network
edit 1
!!!Advertising the loopback network
set prefix 192.168.0.1 255.255.255.255
next
edit 2
!!!Advertising the LAN/inside network
set prefix 192.168.1.0 255.255.255.0
next
edit 3
!!!Advertising the IPSec VPN/Overlay network
set prefix 192.168.10.0 255.255.255.0
next
end
end
Spoke2 configuration
As all the spokes has similar configuration, we will only show the configuration at Spoke2. For other spokes only the IP addresses changes.
Let's look at Spoke2's physical interface and IP address configuration -
config system interface
!!!Lan facing interface
edit "port7"
set vdom "root"
set ip 192.168.2.1 255.255.255.0
set allowaccess ping
set type physical
set description "Lan"
set alias "Lan"
next
!!!Internet facing interface
edit "port8"
set vdom "root"
set ip 172.16.0.6 255.255.255.252
set allowaccess ping
set type physical
set description "Internet"
set alias "Internet"
next
!!!Loopback interface, used for router-id in OSPF
edit "Loopback"
set vdom "root"
set ip 192.168.0.2 255.255.255.255
set allowaccess ping
set type loopback
set description "Loopback"
set alias "Loopback"
set role lan
next
end
Now we will configure the IPSec phase-1 at the Spoke2 with ADVPN.
config vpn ipsec phase1-interface
edit "Spoke2-Phase1"
set ike-version 2
set proposal des-md5
set dhgrp 5
set authmethod psk
set psksecret test123
set nattraversal disable
set keylife 86400
set dpd on-idle
set dpd-retrycount 3
set dpd-retryinterval 20
!!!Internet facing interface
set interface "port8"!!!A spoke connects only to the Hub via IPSec.
!!!So, it will be static IPSec tunnel where it will connect to Hub's internet facing IP address.
set type static
set peertype any
set remote-gw 172.16.0.2
!!!As the spoke creates only P-to-P tunnel, no need to activate fortigate's new dial-up logic
set net-device enable
!!!Now we will activate ADVPN with one single command.
!!!As the Spoke2 expects topology information from Hub1, it will be always receiver in ADVPN
!!!With information from Hub1, Spoke2 can create point-to-point IPSec tunnel with other Spokes.
set auto-discovery-receiver enable
next
end
Next we will configure IPSec phase-2 at the Spoke2.
config vpn ipsec phase2-interface
edit "Spoke2-Phase2"
set encapsulation tunnel-mode
set proposal des-md5
set pfs disable
set keylife-type seconds
set keylifeseconds 43200
!!!Connecting phase-2 with phase-1
set phase1name "Spoke2-Phase1"
!!!The spoke will initiate the IPSec negotiation.
!!!The hub does not know the internet facing IP adress of the spokes beforehand.
!!!So, it is the responsibility of the spokes to initiate the IPSec negotiation.
!!!The hub acts as a responder in IPSec negotiation.
set auto-negotiate enable
next
end
As we will be running OSPF, we need to assign an IP adress to Spoke2's IPSec tunnel interface. We have reserved 192.168.10.0/24 network as our overlay network and Spoke2 will get an IP adress from this network.
edit "Spoke2-Phase1"
set vdom "root"
set ip 192.168.10.2 255.255.255.255
set allowaccess ping
set type tunnel
!!!Remote IP is set to Hub1's tunnel IP address
set remote-ip 192.168.10.1 255.255.255.0
set interface "port8"
next
end
Now we will configure the firewall policies. Our policy is very simple - we will allow everything from Spoke2's LAN to Hub1's LAN or other Spokes LAN and vice versa. For configuration flexibility we have assigned our interfaces to different zones which are pretty self-explanatory.
config system zone
edit "Lan-Zone"
set interface "port7"
next
edit "Vpn-Zone"
set interface "Spoke2-Phase1"
next
edit "Internet-Zone"
set interface "port8"
next
end
config firewall policy
edit 1
set srcintf "Lan-Zone"
set dstintf "Vpn-Zone"
set srcaddr "all"
set dstaddr "all"
set action accept
set schedule "always"
set service "ALL"
next
edit 2
set srcintf "Vpn-Zone"
set dstintf "Lan-Zone"
set srcaddr "all"
set dstaddr "all"
set action accept
set schedule "always"
set service "ALL"
next
end
Now we will configure OSPF routing as below for Spoke2. By default tunnel interface created by IPSec is of type point-to-point. So, for Spoke2 we don't need to specify explicitly that OSPF sees tunnel interface as point-to-point. But we will do that here for configuration clarity and consistency.
config router ospf
!!!Loopback address is router-id
set router-id 192.168.0.2
config area
edit 0.0.0.0
next
end
config ospf-interface
!!!Tunnel interface will have some OSPF settings applied to it
edit "Spoke2-Phase1"
set interface "Spoke2-Phase1"
!!!OSPF network type will be point-to-point
set network-type point-to-point
set dead-interval 40set hello-interval 10
set mtu-ignore enable
next
end
config network
edit 1
!!!Advertising the loopback network
set prefix 192.168.0.2 255.255.255.255
next
edit 2
!!!Advertising the LAN/inside network
set prefix 192.168.2.0 255.255.255.0
next
edit 3
!!!Advertising the IPSec VPN/Overlay network
set prefix 192.168.10.0 255.255.255.0
next
end
end
That's it. We are done with Spoke2's configuration. Spoke3 and Spoke4 are also configured similarly.
Verification
We will inspect what happens when PC-Spoke2 (192.168.2.101/24) tries to ping PC-Spoke3 (192.168.3.101/24).
Let's see Spoke2's routing table -
Spoke2 # get router info routing-table ospf | grep 192.168.3.0/24
O 192.168.3.0/24 [110/201] via 192.168.10.1, Spoke2-Phase1, 00:00:04
We can see that from Spoke2, 192.168.3.0/24 network is reachable via Hub1 (192.168.10.1).
Let's do a trace route from PC-Spoke2 (192.168.2.101/24).
PC-Spoke2> trace 192.168.3.101
trace to 192.168.3.101, 8 hops max (ICMP), press Ctrl+C to stop
1 192.168.2.1 0.407 ms 0.322 ms 0.415 ms --Spoke2
2 192.168.10.1 2.132 ms 2.223 ms 2.203 ms --Hub1
3 192.168.10.3 14.126 ms 6.504 ms 7.497 ms --Spoke3
4 192.168.3.101 16.480 ms 8.670 ms 7.434 ms
As expected the traffic goes through Hub1. In the background I was capturing what is happening in IKE negotiations with Spoke2 and Spoke3 from Hub1.
Hub1 # diagnose vpn ike log-filter mdst-addr4 172.16.0.6 172.16.0.10
Hub1 # diagnose debug application ike -1
Hub1 # diagnose debug enable
ike 0: shortcut Hub1-Phase1_0:172.16.0.6:0 to Hub1-Phase1_1:172.16.0.10:0 for 192.168.2.101->192.168.3.101
ike 0:Hub1-Phase1_0:29: sending NOTIFY msg
ike 0:Hub1-Phase1_0:7:29: send informational
ike 0:Hub1-Phase1_0:7: processing notify type SHORTCUT_QUERY
ike 0:Hub1-Phase1_0: recv shortcut-query 3222103791193211351 c2f61f5b72ffd527/0000000000000000 172.16.0.6 192.168.2.101->192.168.3.101 psk 64 ppk 0 ttl 32 nat 0 ver 2 mode 0
ike 0:Hub1-Phase1: iif 14 192.168.2.101->192.168.3.101 route lookup oif 14
ike 0:Hub1-Phase1_1: forward shortcut-query 3222103791193211351 c2f61f5b72ffd527/0000000000000000 172.16.0.6 192.168.2.101->192.168.3.101 psk 64 ppk 0 ttl 31 ver 2 mode 0
ike 0:Hub1-Phase1_1:8: processing notify type SHORTCUT_REPLY
ike 0:Hub1-Phase1_1: recv shortcut-reply 3222103791193211351 c2f61f5b72ffd527/10fdefdcfb408fa4 172.16.0.10 to 192.168.2.101 psk 64 ppk 0 ver 2 mode 0
ike 0:Hub1-Phase1_0: forward shortcut-reply 3222103791193211351 c2f61f5b72ffd527/10fdefdcfb408fa4 172.16.0.10 to 192.168.2.101 psk 64 ppk 0 ttl 31 ver 2 mode 0
We can see that Hub1 is informing Spoke2 (Hub1-Phase1_0) and Spoke3 (Hub1-Phase1_1) that they can communicate directly instead of going through Hub1 (shortcut messages).
Let's examine Spoke2's routing table again -
Spoke2 # get router info routing-table ospf | grep 192.168.3.0/24
O 192.168.3.0/24 [110/101] via 192.168.10.3, Spoke2-Phase1_0, 00:20:11
Now we can see that Spoke2 has established a direct IPSec tunnel (Spoke2-Phase1_0) with Spoke3. The next hop is now Spoke3 (192.168.10.3) instead of Hub1 (192.168.10.1).
Let's do another trace route from PC-Spoke2 -
PC-Spoke2> trace 192.168.3.101
trace to 192.168.3.101, 8 hops max (ICMP), press Ctrl+C to stop
1 192.168.2.1 0.488 ms 0.271 ms 0.205 ms --Spoke2
2 192.168.10.3 1.921 ms 1.877 ms 1.329 ms --Spoke3
3 192.168.3.101 2.606 ms 1.157 ms 1.616 ms
We can now see that with ADVPN, Spokes are now creating IPSec tunnels with each other and traffic between them are going through that tunnel. Spoke to Spoke communication are no longer going through the Hub. We are getting a fully-meshed network without much configuration effort which was our design goal.
Remarks
If someone was wondering how long those Spoke-to-Spoke tunnels are active. Currently FotiOS has no mechanism no tear up those tunnels on idle event. The only way those tunnels are flushed when the lifetime on IPSec phase-1 times out. For example, in our case the phase-1 timeout value in Spoke2 was set keylife 86400 or 1 day. After 1 day, Spoke2 flushes all the IPSec tunnels and only reestablishes the tunnel with Hub1. And again tunnel to Spoke3 is created dynamically by ADVPN when hosts behind Spoke2 is trying to communicate with hosts behind Spoke3.
We can also flush the tunnel created by ADVPN to Spoke3 by cli command below -
Spoke2 # diagnose vpn tunnel flush Spoke2-Phase1_0
Beautiful explanation !!!! Shariful Congrats.
ReplyDeleteI liked very much, you started with a Hub-and-Spoke topology config, and after that, you added few commands to implement ADVPN feature, you added too comments about certain commands line, and this is usefull to help to understand the config process, like i said beautifull explanation, so much better than Fortinet official documents that you cand find on the internet.
I already just did the config in the Fortigate units topology that i have, but with these information i can enhance the config.
If i have few doubts about the config or the behavior after config, can i do a few questions to you ???
Again, thanks so much beautifull post
Let me know if you have any questions.
DeleteHi Shariful,
ReplyDeleteThis is a great case study. Thanks
Further, is it possible to you to make case study on ADVPN topology over SD-Wan with BGP/OSPF.
Hi Shariful,
ReplyDeleteGreat blog, thanks.
Trying to setup an hub and spoke VPN solution where the hub is a fortigate and the spokes are Cisco router.
Have you come across that? Is it possible?
Thanks.
With Advpn it is not possible as far as I know. Creating these vpn tunnels between spokes are done with fortigate's proprietary implementation. They call it advpn. Like Cisco has similar proprietary implementation called dmvpn. You will find wrtings about dmvpn also in the blog. I hope someday there is a standard implementation apart from these proprietary implementation called advpn or dmvpn.
ReplyDeletethanks, make sense.
ReplyDelete