Cisco DMVPN Phase 3 and OSPF routing Configuration and Verification

In previous blog post, we have configured DMVPN phase 1 and eliminated any configuration change required at the Hub site when a new Spoke is added to the network. But there is a limitation. That is even though we have achieved configuration flexibility, our underlying topology is still hub-and-spoke. All spoke-to-spoke communication goes through the hub first.

With DMVPN phase 3, we can remove that limitation. We can achieve a fully meshed network by using phase 3 of DMVPN.

Before explaining how DMVPN phase 3 works, have a look at our physical and routing topology from phase 1 - 

DMVPN Physical Topology

DMVPN Routing Topology

For example, when Spoke2 tries to communicate with Spoke3 - as usual traffic goes to the Hub1. Hub1 knows the whole network topology. But with phase 3, when Hub1 sees that two spokes are trying to communicate with each other, it will intervene. The Hub1 will signal Spoke2 that there is a better path and that is - Spoke2 can communicate with Spoke3 directly. Hub1 passes all the required informations to create a shortcut tunnel between Spoke2 and Spoke3. So, with phase 3 we can convert a hub-and-spoke topology to fully-meshed topology with minimum configuration effort. After the shortcut tunnel creation the routing topology looks like below -

DMVPN Phase 3 routing with shortcuts

And how do we achieve this? To convert our hub-and-spoke topology to this new fully-meshed topology we just need add one single command in the Hub and the Spokes. Again I will show the full configuration commands from previous blog post and mark the changes that we have done to activate phase 3 of DMVPN.

The IP adressering is given in the table below -

Site	WAN	LAN	IPSec Tunnel	Loopback
Hub1	172.16.0.2/30	192.168.1.1/24	192.168.10.1/24	192.168.0.1/32
Spoke1	172.16.0.6/30	192.168.2.1/24	192.168.10.2/24	192.168.0.2/32
Spoke2	172.16.0.10/30	192.168.3.1/24	192.168.10.3/24	192.168.0.3/32
Spoke3	172.16.0.14/30	192.168.4.1/24	192.168.10.4/24	192.168.0.4/32

Router1 Configuration

Router1 has nothing to do with our DMVPN topology. It is just a router which takes the gateway responsibility and provides WAN connectivity between HUB and SPOKES.

interface GigabitEthernet0/0

 ip address 172.16.0.1 255.255.255.252
!
interface GigabitEthernet0/1
 ip address 172.16.0.5 255.255.255.252
!
interface GigabitEthernet0/2
 ip address 172.16.0.9 255.255.255.252
!
interface GigabitEthernet0/3
 ip address 172.16.0.13 255.255.255.252
!
!!!DHCP server for Spoke4 address assignment
ip dhcp pool pool-172.16.0.12/30
 network 172.16.0.12 255.255.255.252
 dns-server 172.16.0.13
 domain-name family.local
 default-router 172.16.0.13
 lease 0 8
!

Hub1 Configuration

Hub1 interface and IP configuration -

interface GigabitEthernet0/0
 description WAN

 ip address 172.16.0.2 255.255.255.252
!
interface GigabitEthernet0/3
 description LAN
 ip address 192.168.1.1 255.255.255.0
!
interface Loopback0
 ip address 192.168.0.1 255.255.255.255
!
ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/0 172.16.0.1
!

Hub1 GRE tunnel configuration -

interface Tunnel0
 description DMVPN-OVERLAY
 ip address 192.168.10.1 255.255.255.0
 !!!Manually setting mtu and tcp-mss to avoid fragmentation
 ip mtu 1400
 ip tcp adjust-mss 1360
 !!! Set a nhrp network id, which defines to what DMVPN network the tunnel interface is participating, locally significant only.
 ip nhrp network-id 1
 !!! Defines NHRP password, maximum 8 characters
 ip nhrp authentication NHRPKEY
 !!! Enables multicast support for routing protocols. Applicable to HUB routers only, applicable to   HUB only
 ip nhrp map multicast dynamic
 !!! As Hub tunnel is multipoint, we have only tunnel source, but no tunnel destination
 tunnel source GigabitEthernet0/0
 !!! Defines the tunnel as multipoint GRE (mGRE)
 tunnel mode gre multipoint
 !!! Tunnel key identifies the virtual tunnel interface if multiple tunnel interface uses the same source   interface. If configured, it must be same in all routers (HUB and SPOKE) in a DMVPN domain.
 tunnel key 1
 !!! New command in Hub which enables to send redirect messages to Spokes so that Spokes can communicate with each other directly
ip nhrp redirect
!

Hub1 OSPF routing configuration

router ospf 1
 passive-interface default
 no passive-interface Tunnel0
 network 192.168.0.1 0.0.0.0 area 0 --Loopback
 network 192.168.1.0 0.0.0.255 area 0 --LAN
 network 192.168.10.0 0.0.0.255 area 0 -- DMVPN Overlay
!
interface Loopback0
 ip ospf network point-to-point
!
interface Tunnel0
 !!! OSPF network type is p-to-mp (from HUB to each SPOKE)
 ip ospf network point-to-multipoint
 !!! Default OSPF p-to-mp timer is hello 30 sec, dead 120 sec. We are modifying that to 10/40     second, so that we do not need to change it in every Spoke.
 ip ospf hello-interval 10
 ip ospf dead-interval 40
 ip ospf mtu-ignore
!

Hub1 IPSec encryption configuration -

crypto ikev2 keyring Dmvpn-Keyring
 peer ANY
 address 0.0.0.0 0.0.0.0
 pre-shared-key test123
!
crypto ikev2 proposal Dmvpn-Ike-Proposal
 encryption aes-cbc-256
 integrity sha256
 group 15
!
crypto ikev2 policy Dmvpn-Ike-Policy
 proposal Dmvpn-Ike-Proposal
!
crypto ikev2 profile Dmvpn-Ike-Profile
 match identity remote address 0.0.0.0
 authentication remote pre-share
 authentication local pre-share
 keyring local Dmvpn-Keyring
 lifetime 28800
!
!
crypto ipsec transform-set Dmvpn-Ipsec-Tset esp-aes esp-sha256-hmac
 !!! IPSec in transport mode
 mode transport
!
crypto ipsec profile Dmvpn-Ipsec-Profile
 set transform-set Dmvpn-Ipsec-Tset
 set pfs group15
 set security-association lifetime seconds 3600
 set ikev2-profile Dmvpn-Ike-Profile
!
!
interface Tunnel0
 !!! Activating IPSec on the tunnel interface
 tunnel protection ipsec profile Dmvpn-Ipsec-Profile
!
Spoke2 Configuration

Spoke2 and Spoke3 has similar configuration; the difference is only in their interface IP addresses. We will show only the Spoke2 configuration.

Spoke2 interface and IP configuration -

interface Loopback0
 ip address 192.168.0.2 255.255.255.255
!
interface GigabitEthernet0/0
 description WAN
 ip address 172.16.0.6 255.255.255.252
!
interface GigabitEthernet0/3
 description LAN
 ip address 192.168.2.1 255.255.255.0
!
ip route 0.0.0.0 0.0.0.0 GigabitEthernet0/0 172.16.0.5
!

Spoke2 GRE tunnel configuration -

interface Tunnel0
 description DMVPN-Overlay
 ip address 192.168.10.2 255.255.255.0
 ip mtu 1400
 ip tcp adjust-mss 1360
 ip nhrp network-id 1
 ip nhrp authentication NHRPKEY
 !!! Defining NHRP server (hub's address information)
 ip nhrp nhs 192.168.10.1 nbma 172.16.0.2 multicast
 tunnel source GigabitEthernet0/0
 !!! As this is a GRE p-to-p tunnel we need a tunnel destination, which is Hub1's WAN IP address
 tunnel destination 172.16.0.2
 tunnel key 1
 !!! For phase 1, Spokes tunnel mode is p-to-p
 tunnel mode gre ip
 !!! Ties the tunnel line-protocol state to the reachability of the NHRP NHS (Hub), and if the NHS is unreachable, the tunnel line-protocol state changes to down.
if-state nhrp
!!! We will activate shortcuts in Spokes, so that Spokes can create shortcut tunnels between them
ip nhrp shortcut
!!! Creating shortcut means now we must run mGRE (tunnel to Hub and to Spokes)
tunnel mode gre multipoint
!

Spoke2 OSPF routing configuration -

router ospf 1
 passive-interface default
 no passive-interface Tunnel0
 network 192.168.0.2 0.0.0.0 area 0 --Loopback
 network 192.168.2.0 0.0.0.255 area 0 --LAN
 network 192.168.10.0 0.0.0.255 area 0 --DMVPN Overlay
!
interface Loopback0
 ip ospf network point-to-point
!
interface Tunnel0
 !!! Network type is p-to-p, as spokes forms neighbor-ship with Hub only
 ip ospf network point-to-point
 ip ospf mtu-ignore
 !!! For p-to-p default hello/dead value is 10/40 by default, here it is just shown for clarity of OSPF   configuration.
 ip ospf hello-interval 10
 ip ospf dead-interval 40
!

Spoke2 IPSec encryption configuration -

crypto ikev2 keyring Dmvpn-Keyring
 peer ANY
 address 0.0.0.0 0.0.0.0
 pre-shared-key test123
!
crypto ikev2 proposal Dmvpn-Ike-Proposal
 encryption aes-cbc-256
 integrity sha256
 group 15
!
crypto ikev2 policy Dmvpn-Ike-Policy
 proposal Dmvpn-Ike-Proposal
!
crypto ikev2 profile Dmvpn-Ike-Profile
 match identity remote address 0.0.0.0
 authentication remote pre-share
 authentication local pre-share
 keyring local Dmvpn-Keyring
 lifetime 28800
!
!
crypto ipsec transform-set Dmvpn-Ipsec-Tset esp-aes esp-sha256-hmac
 !!! IPSec in transport mode
 mode transport
!
crypto ipsec profile Dmvpn-Ipsec-Profile
 set transform-set Dmvpn-Ipsec-Tset
 set pfs group15
 set security-association lifetime seconds 3600
 set ikev2-profile Dmvpn-Ike-Profile
!
!
interface Tunnel0
 !!! Activating IPSec on the tunnel interface
 tunnel protection ipsec profile Dmvpn-Ipsec-Profile
!
Verification

Let's try to do a traceroute from Spoke3 LAN to Spoke2 LAN.

Spoke3#traceroute 192.168.2.1 source 192.168.3.1
Type escape sequence to abort.
Tracing the route to 192.168.2.1
VRF info: (vrf in name/id, vrf out name/id)
1 192.168.10.1 17 msec 4 msec 16 msec
2 192.168.10.2 26 msec * 10 msec

We can see that we can reach network 192.168.2.0/24 by going through Hub1 as expected. In the background, we have activated NHRP debugging in Hub1 as shown below and we can observe the NHRP redirect behavior.

Hub1#debug nhrp
*May 3 04:38:12.912: NHRP: nhrp_ifcache: Avl Root:10991CC8
*May 3 04:38:12.930: NHRP: nhrp_ifcache: Avl Root:10991CC8
*May 3 04:38:12.933: NHRP: nhrp_ifcache: Avl Root:10991CC8
*May 3 04:38:12.954: NHRP: Tunnels gave us remote_nbma: 172.16.0.10 for Redirect
*May 3 04:38:12.954: NHRP: Attempting to Redirect, remote_nbma:172.16.0.10, dst:192.168.2.1
*May 3 04:38:12.954: NHRP: inserting (172.16.0.10/192.168.2.1) in redirect table
*May 3 04:38:12.954: NHRP: Attempting to send packet through interface Tunnel0 via DEST dst 192.168.3.1
*May 3 04:38:12.954: NHRP: Send Traffic Indication via Tunnel0 vrf 0, packet size: 99
*May 3 04:38:12.954: src: 192.168.10.1, dst: 192.168.3.1
*May 3 04:38:12.954: NHRP: Encapsulation succeeded. Sending NHRP Control Packet NBMA Address: 172.16.0.10
*May 3 04:38:12.954: NHRP: 127 bytes out Tunnel0
*May 3 04:38:12.970: NHRP: Tunnels gave us remote_nbma: 172.16.0.6 for Redirect
*May 3 04:38:12.970: NHRP: Attempting to Redirect, remote_nbma:172.16.0.6, dst:192.168.3.1
*May 3 04:38:12.970: NHRP: inserting (172.16.0.6/192.168.3.1) in redirect table
*May 3 04:38:12.970: NHRP: Attempting to send packet through interface Tunnel0 via DEST dst 192.168.10.2
*May 3 04:38:12.970: NHRP: Send Traffic Indication via Tunnel0 vrf 0, packet size: 99

If we run the traceroute command again, this time we will see that our next-hop router is only Sopke2, which means traffic from Spoke3 is going to Spoke2 directly; bypassing Hub1. And we have achieve our design goal of creating a fully meshed network.

Spoke3#traceroute 192.168.2.1 source 192.168.3.1
Type escape sequence to abort.
Tracing the route to 192.168.2.1
VRF info: (vrf in name/id, vrf out name/id)
1 192.168.10.2 1 msec * 8 msec

Let's inspect what new information is available in the DMVPN, NHRP and routing table in Spoke3 which allows this spoke-to-spoke communication.

Spoke3#show dmvpn 

Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete

        N - NATed, L - Local, X - No Socket

        T1 - Route Installed, T2 - Nexthop-override

        C - CTS Capable

        # Ent --> Number of NHRP entries with same NBMA peer

        NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting

        UpDn Time --> Up or Down Time for a Tunnel

==========================================================================

Interface: Tunnel0, IPv4 NHRP Details 

Type:Spoke, NHRP Peers:2, 

 # Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb

 ----- --------------- --------------- ----- -------- -----

     2 172.16.0.6         192.168.10.2    UP 00:00:18   DT2

                                192.168.10.2    UP 00:00:18   DT1

     1 172.16.0.2         192.168.10.1    UP 02:46:42     S

In output above we can see that now we have a new dynamic mapping in the DMVPN database for Spoke2.

Spoke3#sh ip nhrp                               

192.168.2.0/24 via 192.168.10.2

   Tunnel0 created 00:02:36, expire 01:57:23

   Type: dynamic, Flags: router rib nho 

   NBMA address: 172.16.0.6 

192.168.10.1/32 via 192.168.10.1

   Tunnel0 created 02:49:54, never expire 

   Type: static, Flags: used 

   NBMA address: 172.16.0.2 

192.168.10.2/32 via 192.168.10.2

   Tunnel0 created 00:02:36, expire 01:57:23

   Type: dynamic, Flags: router used nhop rib 

   NBMA address: 172.16.0.6 

The same is also true for NHRP cache, now we have mapping for Spoke2 DMVPN overlay (tunnel IP) to underlay (WAN IP).

Spoke3#show ip route 

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP

       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 

       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

       E1 - OSPF external type 1, E2 - OSPF external type 2

       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2

       ia - IS-IS inter area, * - candidate default, U - per-user static route

       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP

       a - application route

       + - replicated route, % - next hop override

Gateway of last resort is 172.16.0.9 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 172.16.0.9, GigabitEthernet0/0

      172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks

C        172.16.0.8/30 is directly connected, GigabitEthernet0/0

L        172.16.0.10/32 is directly connected, GigabitEthernet0/0

      192.168.0.0/32 is subnetted, 4 subnets

O        192.168.0.1 [110/1001] via 192.168.10.1, 02:50:44, Tunnel0

O        192.168.0.2 [110/2001] via 192.168.10.1, 02:50:44, Tunnel0

C        192.168.0.3 is directly connected, Loopback0

O        192.168.0.4 [110/2001] via 192.168.10.1, 02:50:44, Tunnel0

O     192.168.1.0/24 [110/1001] via 192.168.10.1, 02:50:44, Tunnel0

O   % 192.168.2.0/24 [110/2001] via 192.168.10.1, 02:50:44, Tunnel0

      192.168.3.0/24 is variably subnetted, 2 subnets, 2 masks

C        192.168.3.0/24 is directly connected, GigabitEthernet0/3

L        192.168.3.1/32 is directly connected, GigabitEthernet0/3

O     192.168.4.0/24 [110/2001] via 192.168.10.1, 02:50:44, Tunnel0

      192.168.10.0/24 is variably subnetted, 4 subnets, 2 masks

C        192.168.10.0/24 is directly connected, Tunnel0

O        192.168.10.1/32 [110/1000] via 192.168.10.1, 02:50:44, Tunnel0

H        192.168.10.2/32 is directly connected, 00:04:35, Tunnel0

L        192.168.10.3/32 is directly connected, Tunnel0

Spoke3#show ip route next-hop-override 

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP

       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 

       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

       E1 - OSPF external type 1, E2 - OSPF external type 2

       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2

       ia - IS-IS inter area, * - candidate default, U - per-user static route

       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP

       a - application route

       + - replicated route, % - next hop override

Gateway of last resort is 172.16.0.9 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 172.16.0.9, GigabitEthernet0/0

      172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks

C        172.16.0.8/30 is directly connected, GigabitEthernet0/0

L        172.16.0.10/32 is directly connected, GigabitEthernet0/0

      192.168.0.0/32 is subnetted, 4 subnets

O        192.168.0.1 [110/1001] via 192.168.10.1, 02:50:48, Tunnel0

O        192.168.0.2 [110/2001] via 192.168.10.1, 02:50:48, Tunnel0

C        192.168.0.3 is directly connected, Loopback0

O        192.168.0.4 [110/2001] via 192.168.10.1, 02:50:48, Tunnel0

O     192.168.1.0/24 [110/1001] via 192.168.10.1, 02:50:48, Tunnel0

O   % 192.168.2.0/24 [110/2001] via 192.168.10.1, 02:50:48, Tunnel0

                     [NHO][110/1] via 192.168.10.2, 00:04:39, Tunnel0

      192.168.3.0/24 is variably subnetted, 2 subnets, 2 masks

C        192.168.3.0/24 is directly connected, GigabitEthernet0/3

L        192.168.3.1/32 is directly connected, GigabitEthernet0/3

O     192.168.4.0/24 [110/2001] via 192.168.10.1, 02:50:48, Tunnel0

      192.168.10.0/24 is variably subnetted, 4 subnets, 2 masks

C        192.168.10.0/24 is directly connected, Tunnel0

O        192.168.10.1/32 [110/1000] via 192.168.10.1, 02:50:48, Tunnel0

H        192.168.10.2/32 is directly connected, 00:04:39, Tunnel0

L        192.168.10.3/32 is directly connected, Tunnel0

Spoke3#show ip ospf neighbor 

Neighbor ID     Pri   State           Dead Time   Address         Interface

192.168.0.1       0   FULL/  -        00:00:31    192.168.10.1    Tunnel0

From the output of the routing table, we can see that for Spoke2 LAN, we have a route type of next-hop-override. This is the magic of DMVPN phase 3.

But be very clear, this new route is coming to routing table by NHRP redirect/shortcut, not by the routing protocol OSPF. If we look at the neighbor relationship for Spoke3, we can see that it still has only Hub1 as its OSPF neighbor. This addition of shortcut route in the routing table does not involve OSPF.

How long a shortcut tunnel is active

The shortcut tunnel created by NHRP redirect is by default valid for two hours. After two hours if there is no traffic that uses the shortcut, it will be torn down. We can tune this parameter as follows (only applicable in Spokes) - 

interface tunnel0
 ip nhrp holdtime ? --How long the shortcut tunnels are active
 <1-65535> Number of seconds
 ip nhrp registration timeout ?
 <1-65535> Number of seconds

Debugging commands

Some useful debugging commands are -

debug nhrp
debug nhrp packet
clear ip nhrp