VXLAN Journey - Part 14 - Multi-Site VXLAN - North-South Routing - VRFLite with Active/Passive Firewall - ECMP

Now we will provide our Multi-Site data center access to external networks. This is called north-south routing. We will use an active/passive firewall cluster - because it can act as both router and firewall for external traffic. When BGP neighbors are configured at the vrf-level - this is called VRFLite. From firewall perspective - all the paths towards the vxlan multi-site will be ecmp enabled. In vxlan side - each fabric will use two special switches (vPC enabled) - which are called - Border-Leafs.

Our topology looks like below - 

01 - Network Topology

For Spine, Leaf and Border-Gateway configurations have a look in the previous blogs. We will only show new configurations required of external-connectivity using Border-Leaf switches and active/passive firewall cluster. The topology shows only the devices participating for vrf-lite external routing.

We are connecting both active and passive firewall to both the child-sites (Dc-01 and Dc-02) because we want to enable ecmp using both fabrics. If Ns-Fw-01 is the active firewall logical topology looks like below - 

02 - Logical topology for Ns-Fw-01

If Ns-Fw-02 is the active firewall logical topology looks like below - 

02 - Logical topology for Ns-Fw-02

From firewall's perspective - if Ns-Fw-01 is active then traffic will pass through vpc-port-channel 5 or if Ns-Fw-02 is active then traffic will pass through vpc-port-channel 6. But whichever firewall is active - both of it's port-channels (one port-channel per fabric) will be used for traffic forwarding.

This type of connections though the active/passive firewall cluster always comes with some kind of tradeoffs - because firewall is a stateful network device and we are connecting it to both the fabrics. The limitations of our topology is that -
  • Firewall can never be gateway for a vlan/l2vni.
  • Firewall's routing ecmp is using two different fabrics. Because of that it will create sub-optimal and asymmetric routing. Sub-optimal routing is not a big problem because - all the links between firewalls and switches will be high-speed links. 
  • Asymmetric routing will be a big problem because firewall is a stateful device. I am using Fortinet's Fortigate firewalls - which has proprietary support for such kind of topology. It supports maintaining session table even if incoming/outgoing interfaces changes for an already established session.
  • To minimize the routing impact during firewall-failover we will be using BGP's graceful-restart capability. Do not forget the rule - graceful-restart and BFD does not work well together. In most cases we have to chose one above another considering the tradeoffs.
  • Also for minimum traffic disruption - enable session-pickup in firewall's high-availability configuration. 
Disclaimer: Our topology uses Fortigate firewalls which has proprietary support for ECMP when incoming/outgoing interfaces get changed for an established session. If you are implementing such topology in your environment - do not forget to consult with your firewall vendor first.

Let's start configuring our network.

External routing - VRFLite configuration

We are adding 2 new switches per fabric (vPC-pair). They need to be configured as route-reflector client in the Spine switches. We have covered in detail how to configure Spine switches in previous blogs. We will skip the spine configuration - but do not forget to configure the Spine switches. Also we will explain only the new configurations required for Border-Leaf's VRFLite implementation.

Dc-01-Border-Leaf-09 Configuration

hostname Dc-01-Border-Leaf-09

cfs eth distribute
nv overlay evpn
feature ospf
feature bgp
feature interface-vlan
feature vn-segment-vlan-based
feature lacp
feature vpc
feature nv overlay

fabric forwarding anycast-gateway-mac 2020.0000.00aa

vlan 100
  name Vlan_Vrf_Blue
  vn-segment 100000

vlan 3600
  name Vpc_Peer-Link_Svi_Vlan

route-map Rmap_Tag-12345 permit 10
  match tag 12345

vrf context Vrf_Blue
  vni 100000
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn

vrf context management

system nve infra-vlans 3600

vpc domain 1
  peer-switch
  role priority 10
  peer-keepalive destination 10.10.11.2 source 10.10.11.1
  delay restore 150
  peer-gateway
  layer3 peer-router
  auto-recovery reload-delay 360
  ip arp synchronize

interface Vlan100
  no shutdown
  mtu 9216
  vrf member Vrf_Blue
  no ip redirects
  ip forward
  no ipv6 redirects

interface Vlan3600
  description Svi_Vpc_Peer-Link
  no shutdown
  mtu 9216
  no ip redirects
  ip address 10.81.2.9/30
  no ipv6 redirects
  ip ospf network point-to-point
  ip router ospf Underlay area 0.0.0.0

interface port-channel1500
  description Vpc_Peer-Link_Po1500
  switchport mode trunk
  spanning-tree port type network
  vpc peer-link

interface nve1
  no shutdown
  description Vtep_Vxlan
  host-reachability protocol bgp
  advertise virtual-rmac
  source-interface loopback1
  member vni 100000 associate-vrf

interface Ethernet1/1
  description To_Dc-01-Spine-01_e1/7
  no switchport
  mtu 9216
  medium p2p
  no ip redirects
  ip unnumbered loopback0
  ip router ospf Underlay area 0.0.0.0
  no shutdown

interface Ethernet1/2
  description To_Dc-01-Spine-02_e1/7
  no switchport
  mtu 9216
  medium p2p
  no ip redirects
  ip unnumbered loopback0
  ip router ospf Underlay area 0.0.0.0
  no shutdown

interface Ethernet1/3
  description To_Dc-01-Border-Leaf-10_e1/3_Po1500
  switchport mode trunk
  channel-group 1500 mode active

interface Ethernet1/4
  description To_Dc-01-Border-Leaf-10_e1/4_Po1500
  switchport mode trunk
  channel-group 1500 mode active

interface mgmt0
  vrf member management
  ip address 10.10.11.1/24

interface loopback0
  description Underlay_Routing_Loopback
  ip address 10.81.0.9/32
  ip router ospf Underlay area 0.0.0.0

interface loopback1
  description Overlay_Vtep_Loopback
  ip address 10.81.1.9/32
  ip address 10.81.1.109/32 secondary
  ip router ospf Underlay area 0.0.0.0

router ospf Underlay
  router-id 10.81.0.9
  log-adjacency-changes detail

router bgp 4200000001
  router-id 10.81.0.9
  log-neighbor-changes
  address-family l2vpn evpn
    advertise-pip
  neighbor 10.81.0.1
    remote-as 4200000001
    description Dc-01-Spine-01_Evpn
    update-source loopback0
    address-family l2vpn evpn
      send-community
      send-community extended
  neighbor 10.81.0.2
    remote-as 4200000001
    description Dc-01-Spine-02_Evpn
    update-source loopback0
    address-family l2vpn evpn
      send-community
      send-community extended
  vrf Vrf_Blue
    address-family ipv4 unicast
      redistribute direct route-map Rmap_Tag-12345
      maximum-paths ibgp 2 

!!! Create the vpc port-channel 5 to Ns-Fw-01.
interface port-channel5
  description Port-Channel_Ns-Fw-01
  switchport mode trunk
  switchport trunk allowed vlan 3901
  spanning-tree port type edge trunk
  mtu 9216
  vpc 5

!!! Create the vpc port-channel 6 to Ns-Fw-02.
interface port-channel6
  description Port-Channel_Ns-Fw-02
  switchport mode trunk
  switchport trunk allowed vlan 3901
  spanning-tree port type edge trunk
  mtu 9216
  vpc 6

!!! Assign physical interface to vpc port-channel 5.
interface Ethernet1/5
  description To_Ns-Fw-01_port11_Po5
  switchport mode trunk
  switchport trunk allowed vlan 3901
  mtu 9216
  channel-group 5 mode active

!!! Assign physical interface to vpc port-channel 6.
interface Ethernet1/6
  description To_Ns-Fw-02_port11_Po6
  switchport mode trunk
  switchport trunk allowed vlan 3901
  mtu 9216
  channel-group 6 mode active

!!! Crete the vlan for vrf-lite configuration for Vrf_Blue.
vlan 3901
  name Vrf_Blue-Vrf_Lite-Vlan

!!! Crete the vlan-svi for vrf-lite configuration for Vrf_Blue.
interface Vlan3901
  description Svi-Vrf_Blue-Vrf_Lite-Vlan
  no shutdown
  mtu 9216
  vrf member Vrf_Blue
  no ip redirects
  ip address 10.81.3.17/29

!!! Vrf-lite ebgp routing for Vrf_Blue will be establish over loopback interface.
!!! Create a loopback interface for bgp in Vrf_Blue.
interface loopback199
  description Loopback-Vrf_Blue
  vrf member Vrf_Blue
  ip address 10.81.4.9/32

!!! Static routes to reach other bgp neighbor's loopback addresses.
!!! We have two neighbors - ibgp - Dc-01-Border-Leaf-10 and ebgp - Ns-Fw-Cluster.
vrf context Vrf_Blue
  !!! Static route for Dc-01-Border-Leaf-10's loopback.
  ip route 10.81.4.10/32 10.81.3.18
  !!! Static route for Ns-Fw-Cluster's loopback.
  ip route 10.85.4.1/32 10.81.3.19

ip prefix-list Pfx_All_Routes seq 10 permit 0.0.0.0/0 le 32 
ip prefix-list Pfx_Default_Route_Only seq 10 permit 0.0.0.0/0 
ip prefix-list Pfx_Host_Route_Only seq 10 permit 0.0.0.0/0 eq 32

!!! We will only accept a default-route from the Firewall.
route-map Rmap_Ns_Fw_Cluster_In permit 65534
  match ip address prefix-list Pfx_Default_Route_Only 
route-map Rmap_Ns_Fw_Cluster_In deny 65535

!!! We will only advertise subnet prefixes to the Firewall.
!!! We are dropping all the host-routes (/32).
route-map Rmap_Ns_Fw_Cluster_Out deny 65534
  match ip address prefix-list Pfx_Host_Route_Only 
route-map Rmap_Ns_Fw_Cluster_Out permit 65535

!!! Vrf-Lite configuration.
router bgp 4200000001
  vrf Vrf_Blue
    !!! Vrf's router id is loopback 199.
    router-id 10.81.4.9
    log-neighbor-changes

    !!! IBGP to Dc-01-Border-Leaf-10 because neighbor is vpc-peer.
    neighbor 10.81.4.10
      remote-as 4200000001
      description Dc-01-Border-Leaf-10
      update-source loopback199
      address-family ipv4 unicast
        next-hop-self
    
    !!! EBGP to Ns-Fw-Cluster.
    neighbor 10.85.4.1
      remote-as 65535
      description Ns-Fw-Cluster
      update-source loopback199
      !!! Enable ebgp-multihop.
      ebgp-multihop 5
      address-family ipv4 unicast
        !!! Accept only a default -route from the firewall.
        route-map Rmap_Ns_Fw_Cluster_In in
        !!! Advertise all routes except /32 host-routes.
        route-map Rmap_Ns_Fw_Cluster_Out out
        next-hop-self    

Dc-01-Border-Leaf-10 Configuration

hostname Dc-01-Border-Leaf-10

cfs eth distribute
nv overlay evpn
feature ospf
feature bgp
feature interface-vlan
feature vn-segment-vlan-based
feature lacp
feature vpc
feature nv overlay

fabric forwarding anycast-gateway-mac 2020.0000.00aa

vlan 100
  name Vlan_Vrf_Blue
  vn-segment 100000

vlan 3600
  name Vpc_Peer-Link_Svi_Vlan

vlan 3901
  name Vrf_Blue-Vrf_Lite-Vlan

ip prefix-list Pfx_Default_Route_Only seq 10 permit 0.0.0.0/0
ip prefix-list Pfx_Host_Route_Only seq 10 permit 0.0.0.0/0 eq 32

route-map Rmap_Ns_Fw_Cluster_In permit 65534
  match ip address prefix-list Pfx_Default_Route_Only
route-map Rmap_Ns_Fw_Cluster_In deny 65535

route-map Rmap_Ns_Fw_Cluster_Out deny 65534
  match ip address prefix-list Pfx_Host_Route_Only
route-map Rmap_Ns_Fw_Cluster_Out permit 65535

route-map Rmap_Tag-12345 permit 10
  match tag 12345

vrf context Vrf_Blue
  vni 100000
  ip route 10.81.4.9/32 10.81.3.17
  ip route 10.85.4.1/32 10.81.3.19
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn

vrf context management

system nve infra-vlans 3600

vpc domain 1
  peer-switch
  role priority 20
  peer-keepalive destination 10.10.11.1 source 10.10.11.2
  delay restore 150
  peer-gateway
  layer3 peer-router
  auto-recovery reload-delay 360
  ip arp synchronize

interface Vlan100
  no shutdown
  mtu 9216
  vrf member Vrf_Blue
  no ip redirects
  ip forward
  no ipv6 redirects

interface Vlan3600
  description Svi_Vpc_Peer-Link
  no shutdown
  mtu 9216
  no ip redirects
  ip address 10.81.2.10/30
  no ipv6 redirects
  ip ospf network point-to-point
  ip router ospf Underlay area 0.0.0.0

interface Vlan3901
  description Svi-Vrf_Blue-Vrf_Lite-Vlan
  no shutdown
  mtu 9216
  vrf member Vrf_Blue
  no ip redirects
  ip address 10.81.3.18/29
  no ipv6 redirects

interface port-channel5
  description Port-Channel_Ns-Fw-01
  switchport mode trunk
  switchport trunk allowed vlan 3901
  spanning-tree port type edge trunk
  mtu 9216
  vpc 5

interface port-channel6
  description Port-Channel_Ns-Fw-02
  switchport mode trunk
  switchport trunk allowed vlan 3901
  spanning-tree port type edge trunk
  mtu 9216
  vpc 6

interface port-channel1500
  description Vpc_Peer-Link_Po1500
  switchport mode trunk
  spanning-tree port type network
  vpc peer-link

interface nve1
  no shutdown
  description Vtep_Vxlan
  host-reachability protocol bgp
  advertise virtual-rmac
  source-interface loopback1
  member vni 100000 associate-vrf

interface Ethernet1/1
  description To_Dc-01-Spine-01_e1/8
  no switchport
  mtu 9216
  medium p2p
  no ip redirects
  ip unnumbered loopback0
  ip router ospf Underlay area 0.0.0.0
  no shutdown

interface Ethernet1/2
  description To_Dc-01-Spine-02_e1/8
  no switchport
  mtu 9216
  medium p2p
  no ip redirects
  ip unnumbered loopback0
  ip router ospf Underlay area 0.0.0.0
  no shutdown

interface Ethernet1/3
  description To_Dc-01-Border-Leaf-09_e1/3_Po1500
  switchport mode trunk
  channel-group 1500 mode active

interface Ethernet1/4
  description To_Dc-01-Border-Leaf-09_e1/4_Po1500
  switchport mode trunk
  channel-group 1500 mode active

interface Ethernet1/5
  description To_Ns-Fw-01_port12_Po5
  switchport mode trunk
  switchport trunk allowed vlan 3901
  mtu 9216
  channel-group 5 mode active

interface Ethernet1/6
  description To_Ns-Fw-02_port12_Po6
  switchport mode trunk
  switchport trunk allowed vlan 3901
  mtu 9216
  channel-group 6 mode active

interface mgmt0
  vrf member management
  ip address 10.10.11.2/24

interface loopback0
  description Underlay_Routing_Loopback
  ip address 10.81.0.10/32
  ip router ospf Underlay area 0.0.0.0

interface loopback1
  description Overlay_Vtep_Loopback
  ip address 10.81.1.10/32
  ip address 10.81.1.109/32 secondary
  ip router ospf Underlay area 0.0.0.0

interface loopback199
  description Loopback-Vrf_Blue
  vrf member Vrf_Blue
  ip address 10.81.4.10/32

router ospf Underlay
  router-id 10.81.0.10
  log-adjacency-changes detail

router bgp 4200000001
  router-id 10.81.0.10
  log-neighbor-changes
  address-family l2vpn evpn
    advertise-pip
  neighbor 10.81.0.1
    remote-as 4200000001
    description Dc-01-Spine-01_Evpn
    update-source loopback0
    address-family l2vpn evpn
      send-community
      send-community extended
  neighbor 10.81.0.2
    remote-as 4200000001
    description Dc-01-Spine-02_Evpn
    update-source loopback0
    address-family l2vpn evpn
      send-community
      send-community extended
  vrf Vrf_Blue
    router-id 10.81.4.10
    log-neighbor-changes
    address-family ipv4 unicast
      redistribute direct route-map Rmap_Tag-12345
      maximum-paths ibgp 2
    neighbor 10.81.4.9
      remote-as 4200000001
      description Dc-01-Border-Leaf-09
      update-source loopback199
      address-family ipv4 unicast
        next-hop-self
    neighbor 10.85.4.1
      remote-as 65535
      description Ns-Fw-Cluster
      update-source loopback199
      ebgp-multihop 5
      address-family ipv4 unicast
        route-map Rmap_Ns_Fw_Cluster_In in
        route-map Rmap_Ns_Fw_Cluster_Out out
        next-hop-self

Dc-02-Border-Leaf-09 Configuration

hostname Dc-02-Border-Leaf-09

cfs eth distribute
nv overlay evpn
feature ospf
feature bgp
feature interface-vlan
feature vn-segment-vlan-based
feature lacp
feature vpc
feature nv overlay

fabric forwarding anycast-gateway-mac 2020.0000.00aa

vlan 100
  name Vlan_Vrf_Blue
  vn-segment 100000
vlan 3600
  name Vpc_Peer-Link_Svi_Vlan
vlan 3902
  name Vrf_Blue-Vrf_Lite-Vlan

ip prefix-list Pfx_Default_Route_Only seq 10 permit 0.0.0.0/0 
ip prefix-list Pfx_Host_Route_Only seq 10 permit 0.0.0.0/0 eq 32 

route-map Rmap_Ns_Fw_Cluster_In permit 65534
  match ip address prefix-list Pfx_Default_Route_Only 
route-map Rmap_Ns_Fw_Cluster_In deny 65535

route-map Rmap_Ns_Fw_Cluster_Out deny 65534
  match ip address prefix-list Pfx_Host_Route_Only 
route-map Rmap_Ns_Fw_Cluster_Out permit 65535

route-map Rmap_Tag-12345 permit 10
  match tag 12345 

vrf context Vrf_Blue
  vni 100000
  ip route 10.82.4.10/32 10.82.3.18
  ip route 10.85.4.1/32 10.82.3.19
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn

vrf context management

system nve infra-vlans 3600

vpc domain 1
  peer-switch
  role priority 10
  peer-keepalive destination 10.10.11.2 source 10.10.11.1
  delay restore 150
  peer-gateway
  layer3 peer-router
  auto-recovery reload-delay 360
  ip arp synchronize

interface Vlan100
  no shutdown
  mtu 9216
  vrf member Vrf_Blue
  no ip redirects
  ip forward
  no ipv6 redirects

interface Vlan3600
  description Svi_Vpc_Peer-Link
  no shutdown
  mtu 9216
  no ip redirects
  ip address 10.82.2.9/30
  no ipv6 redirects
  ip ospf network point-to-point
  ip router ospf Underlay area 0.0.0.0

interface Vlan3902
  description Svi-Vrf_Blue-Vrf_Lite-Vlan
  no shutdown
  mtu 9216
  vrf member Vrf_Blue
  no ip redirects
  ip address 10.82.3.17/29
  no ipv6 redirects

interface port-channel5
  description Port-Channel_Ns-Fw-01
  switchport mode trunk
  switchport trunk allowed vlan 3902
  spanning-tree port type edge trunk
  mtu 9216
  vpc 5

interface port-channel6
  description Port-Channel_Ns-Fw-02
  switchport mode trunk
  switchport trunk allowed vlan 3902
  spanning-tree port type edge trunk
  mtu 9216
  vpc 6

interface port-channel1500
  description Vpc_Peer-Link_Po1500
  switchport mode trunk
  spanning-tree port type network
  vpc peer-link

interface nve1
  no shutdown
  description Vtep_Vxlan
  host-reachability protocol bgp
  advertise virtual-rmac
  source-interface loopback1
  member vni 100000 associate-vrf

interface Ethernet1/1
  description To_Dc-02-Spine-01_e1/7
  no switchport
  mtu 9216
  medium p2p
  no ip redirects
  ip unnumbered loopback0
  ip router ospf Underlay area 0.0.0.0
  no shutdown

interface Ethernet1/2
  description To_Dc-02-Spine-02_e1/7
  no switchport
  mtu 9216
  medium p2p
  no ip redirects
  ip unnumbered loopback0
  ip router ospf Underlay area 0.0.0.0
  no shutdown

interface Ethernet1/3
  description To_Dc-02-Border-Leaf-10_e1/3_Po1500
  switchport mode trunk
  channel-group 1500 mode active

interface Ethernet1/4
  description To_Dc-02-Border-Leaf-10_e1/4_Po1500
  switchport mode trunk
  channel-group 1500 mode active

interface Ethernet1/5
  description To_Ns-Fw-01_port13_Po5
  switchport mode trunk
  switchport trunk allowed vlan 3902
  mtu 9216
  channel-group 5 mode active

interface Ethernet1/6
  description To_Ns-Fw-02_port13_Po6
  switchport mode trunk
  switchport trunk allowed vlan 3902
  mtu 9216
  channel-group 6 mode active

interface mgmt0
  vrf member management
  ip address 10.10.11.1/24

interface loopback0
  description Underlay_Routing_Loopback
  ip address 10.82.0.9/32
  ip router ospf Underlay area 0.0.0.0

interface loopback1
  description Overlay_Vtep_Loopback
  ip address 10.82.1.9/32
  ip address 10.82.1.109/32 secondary
  ip router ospf Underlay area 0.0.0.0

interface loopback199
  description Loopback-Vrf_Blue
  vrf member Vrf_Blue
  ip address 10.82.4.9/32

router ospf Underlay
  router-id 10.82.0.9
  log-adjacency-changes detail

router bgp 4200000002
  router-id 10.82.0.9
  log-neighbor-changes
  address-family l2vpn evpn
    advertise-pip
  neighbor 10.82.0.1
    remote-as 4200000002
    description Dc-02-Spine-01_Evpn
    update-source loopback0
    address-family l2vpn evpn
      send-community
      send-community extended
  neighbor 10.82.0.2
    remote-as 4200000002
    description Dc-02-Spine-02_Evpn
    update-source loopback0
    address-family l2vpn evpn
      send-community
      send-community extended
  vrf Vrf_Blue
    router-id 10.82.4.9
    log-neighbor-changes
    address-family ipv4 unicast
      redistribute direct route-map Rmap_Tag-12345
      maximum-paths ibgp 2
    neighbor 10.82.4.10
      remote-as 4200000002
      description Dc-02-Border-Leaf-10
      update-source loopback199
      address-family ipv4 unicast
        next-hop-self
    neighbor 10.85.4.1
      remote-as 65535
      description Ns-Fw-Cluster
      update-source loopback199
      ebgp-multihop 5
      address-family ipv4 unicast
        route-map Rmap_Ns_Fw_Cluster_In in
        route-map Rmap_Ns_Fw_Cluster_Out out
        next-hop-self

Dc-02-Border-Leaf-10 Configuration

hostname Dc-02-Border-Leaf-10

cfs eth distribute
nv overlay evpn
feature ospf
feature bgp
feature interface-vlan
feature vn-segment-vlan-based
feature lacp
feature vpc
feature nv overlay

fabric forwarding anycast-gateway-mac 2020.0000.00aa

vlan 100
  name Vlan_Vrf_Blue
  vn-segment 100000
vlan 3600
  name Vpc_Peer-Link_Svi_Vlan
vlan 3902
  name Vrf_Blue-Vrf_Lite-Vlan

ip prefix-list Pfx_Default_Route_Only seq 10 permit 0.0.0.0/0 
ip prefix-list Pfx_Host_Route_Only seq 10 permit 0.0.0.0/0 eq 32 

route-map Rmap_Ns_Fw_Cluster_In permit 65534
  match ip address prefix-list Pfx_Default_Route_Only 
route-map Rmap_Ns_Fw_Cluster_In deny 65535

route-map Rmap_Ns_Fw_Cluster_Out deny 65534
  match ip address prefix-list Pfx_Host_Route_Only 
route-map Rmap_Ns_Fw_Cluster_Out permit 65535

route-map Rmap_Tag-12345 permit 10
  match tag 12345 

vrf context Vrf_Blue
  vni 100000
  ip route 10.82.4.9/32 10.82.3.17
  ip route 10.85.4.1/32 10.82.3.19
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn

vrf context management

system nve infra-vlans 3600

vpc domain 1
  peer-switch
  role priority 20
  peer-keepalive destination 10.10.11.1 source 10.10.11.2
  delay restore 150
  peer-gateway
  layer3 peer-router
  auto-recovery reload-delay 360
  ip arp synchronize

interface Vlan100
  no shutdown
  mtu 9216
  vrf member Vrf_Blue
  no ip redirects
  ip forward
  no ipv6 redirects

interface Vlan3600
  description Svi_Vpc_Peer-Link
  no shutdown
  mtu 9216
  no ip redirects
  ip address 10.82.2.10/30
  no ipv6 redirects
  ip ospf network point-to-point
  ip router ospf Underlay area 0.0.0.0

interface Vlan3902
  description Svi-Vrf_Blue-Vrf_Lite-Vlan
  no shutdown
  mtu 9216
  vrf member Vrf_Blue
  no ip redirects
  ip address 10.82.3.18/29
  no ipv6 redirects

interface port-channel5
  description Port-Channel_Ns-Fw-01
  switchport mode trunk
  switchport trunk allowed vlan 3902
  spanning-tree port type edge trunk
  mtu 9216
  vpc 5

interface port-channel6
  description Port-Channel_Ns-Fw-02
  switchport mode trunk
  switchport trunk allowed vlan 3902
  spanning-tree port type edge trunk
  mtu 9216
  vpc 6

interface port-channel1500
  description Vpc_Peer-Link_Po1500
  switchport mode trunk
  spanning-tree port type network
  vpc peer-link

interface nve1
  no shutdown
  description Vtep_Vxlan
  host-reachability protocol bgp
  advertise virtual-rmac
  source-interface loopback1
  member vni 100000 associate-vrf

interface Ethernet1/1
  description To_Dc-02-Spine-01_e1/8
  no switchport
  mtu 9216
  medium p2p
  no ip redirects
  ip unnumbered loopback0
  ip router ospf Underlay area 0.0.0.0
  no shutdown

interface Ethernet1/2
  description To_Dc-02-Spine-02_e1/8
  no switchport
  mtu 9216
  medium p2p
  no ip redirects
  ip unnumbered loopback0
  ip router ospf Underlay area 0.0.0.0
  no shutdown

interface Ethernet1/3
  description To_Dc-02-Border-Leaf-09_e1/3_Po1500
  switchport mode trunk
  channel-group 1500 mode active

interface Ethernet1/4
  description To_Dc-02-Border-Leaf-09_e1/4_Po1500
  switchport mode trunk
  channel-group 1500 mode active

interface Ethernet1/5
  description To_Ns-Fw-01_port14_Po5
  switchport mode trunk
  switchport trunk allowed vlan 3902
  mtu 9216
  channel-group 5 mode active

interface Ethernet1/6
  description To_Ns-Fw-02_port14_Po6
  switchport mode trunk
  switchport trunk allowed vlan 3902
  mtu 9216
  channel-group 6 mode active

interface mgmt0
  vrf member management
  ip address 10.10.11.2/24

interface loopback0
  description Underlay_Routing_Loopback
  ip address 10.82.0.10/32
  ip router ospf Underlay area 0.0.0.0

interface loopback1
  description Overlay_Vtep_Loopback
  ip address 10.82.1.10/32
  ip address 10.82.1.109/32 secondary
  ip router ospf Underlay area 0.0.0.0

interface loopback199
  description Loopback-Vrf_Blue
  vrf member Vrf_Blue
  ip address 10.82.4.10/32

router ospf Underlay
  router-id 10.82.0.10
  log-adjacency-changes detail

router bgp 4200000002
  router-id 10.82.0.10
  log-neighbor-changes
  address-family l2vpn evpn
    advertise-pip
  neighbor 10.82.0.1
    remote-as 4200000002
    description Dc-02-Spine-01_Evpn
    update-source loopback0
    address-family l2vpn evpn
      send-community
      send-community extended
  neighbor 10.82.0.2
    remote-as 4200000002
    description Dc-02-Spine-02_Evpn
    update-source loopback0
    address-family l2vpn evpn
      send-community
      send-community extended
  vrf Vrf_Blue
    router-id 10.82.4.10
    log-neighbor-changes
    address-family ipv4 unicast
      redistribute direct route-map Rmap_Tag-12345
      maximum-paths ibgp 2
    neighbor 10.82.4.9
      remote-as 4200000002
      description Dc-02-Border-Leaf-09
      update-source loopback199
      address-family ipv4 unicast
        next-hop-self
    neighbor 10.85.4.1
      remote-as 65535
      description Ns-Fw-Cluster
      update-source loopback199
      ebgp-multihop 5
      address-family ipv4 unicast
        route-map Rmap_Ns_Fw_Cluster_In in
        route-map Rmap_Ns_Fw_Cluster_Out out
        next-hop-self

Ns-Fw-Cluster Configuration

We are showing the firewall configuration as one device because it is an active/passive cluster. In the firewall we have enabled two vdoms - Vdom_root and Vdom_Blue. Vdom_root has connectivity to the internet. Vdom_Blue represent the Vrf_Blue in the fabric. There is vdom-link between Vdom_root and Vdom_Blue which allows the vdoms to communicate with each other.

config system interface
    !!! port2 is connected to an upstream router which has internet connectivity.
    edit "port2"
        set vdom "root"
        set ip 172.29.215.61 255.255.255.0
    next
    !!! Port 11 and 12 is connected with Dc-01 Border-Leaf switches.
    !!! These two ports are configured for port-channel.
    edit "port11"
        set vdom "root"
    next
    edit "port12"
        set vdom "root"
    next
    !!! Port 13 and 14 is connected with Dc-02 Border-Leaf switches.
    !!! These two ports are configured for port-channel.
    edit "port13"
        set vdom "root"
    next
    edit "port14"
        set vdom "root"
    next
    !!! Vdom-link interface in root vdom. 
    edit "vd-l-ro-blu0"
        set vdom "root"
        set ip 10.85.2.1 255.255.255.252
        set allowaccess ping
        set type vdom-link
    next
    !!! Vdom-link interface in Blue vdom. 
    edit "vd-l-ro-blu1"
        set vdom "Vdom_Blue"
        set ip 10.85.2.2 255.255.255.252
        set allowaccess ping
        set type vdom-link
    next
    !!! Dc-01 Border-Leaf port-channel.
    edit "Dc-01-Blf-Po1"
        set vdom "root"
        set type aggregate
        set member "port11" "port12"
        set mtu-override enable
        set mtu 9216
    next
    !!! Dc-02 Border-Leaf port-channel.
    edit "Dc-02-Blf-Po1"
        set vdom "root"
        set type aggregate
        set member "port13" "port14"
        set mtu-override enable
        set mtu 9216
    next
    !!! Vlan-Svi with Dc-01 Border-Leaf switches.
    edit "vlan-3901-dc01"
        set vdom "Vdom_Blue"
        set ip 10.81.3.19 255.255.255.248
        set allowaccess ping
        set interface "Dc-01-Blf-Po1"
        set vlanid 3901
    next
    !!! Vlan-Svi with Dc-02 Border-Leaf switches.
    edit "vlan-3902-dc02"
        set vdom "Vdom_Blue"
        set ip 10.82.3.19 255.255.255.248
        set allowaccess ping
        set interface "Dc-02-Blf-Po1"
        set vlanid 3902
    next
    !!! BGP routing loopback.
    edit "loop-ro-vd-blue"
        set vdom "Vdom_Blue"
        set ip 10.85.4.1 255.255.255.255
        set allowaccess ping
        set type loopback
        set snmp-index 28
    next
end

!!! Static route configuration in Vdom_root.
Ns-Fw-02 (root) # show router static 
config router static
    !!! Default static route.
    edit 1
        set gateway 172.29.215.1
        set device "port2"
    next
    !!! Static route to Vdom_Blue for 10.91.0.0/16.
    edit 2
        set dst 10.91.0.0 255.255.0.0
        set gateway 10.85.2.2
        set device "vd-l-ro-blu0"
    next
end

!!! Static route configuration in Vdom_Blue.
Ns-Fw-02 (Vdom_Blue) # show router static 
config router static
    !!! Default static route - which points to Vdom_root.
    edit 5
        set gateway 10.85.2.1
        set device "vd-l-ro-blu1"
    next
    !!! Static route route to Dc-01-Border-Leaf-09's loopback 199.
    edit 2
        set dst 10.81.4.9 255.255.255.255
        set gateway 10.81.3.17
        set device "vlan-3901-dc01"
    next
    !!! Static route route to Dc-01-Border-Leaf-10's loopback 199.
    edit 3
        set dst 10.81.4.10 255.255.255.255
        set gateway 10.81.3.18
        set device "vlan-3901-dc01"
    next
    !!! Static route route to Dc-02-Border-Leaf-09's loopback 199.
    edit 4
        set dst 10.82.4.9 255.255.255.255
        set gateway 10.82.3.17
        set device "vlan-3902-dc02"
    next
    !!! Static route route to Dc-02-Border-Leaf-10's loopback 199.
    edit 6
        set dst 10.82.4.10 255.255.255.255
        set gateway 10.82.3.18
        set device "vlan-3902-dc02"
    next
end

!!! Prefix-List for VRFLite routing. 
Ns-Fw-02 (Vdom_Blue) # show router prefix-list 
config router prefix-list
    !!! Match default-route.
    edit "Pfx_Default_Route_Only"
        config rule
            edit 1
                set prefix 0.0.0.0 0.0.0.0
                unset ge
                unset le
            next
        end
    next
    !!! Match host-route.
    edit "Pfx_Host_Route_Only"
        config rule
            edit 1
                set prefix 0.0.0.0 0.0.0.0
                set ge 32
                set le 32
            next
        end
    next
end

!!! Route-maps for VRFLite routing. 
Ns-Fw-02 (Vdom_Blue) # show router route-map 
config router route-map
    !!! Advertise only default route.
    edit "Rmap_Border_Leaf_Out"
        config rule
            edit 4294967294
                set match-ip-address "Pfx_Default_Route_Only"
            next
            edit 4294967295
                set action deny
            next
        end
    next
    !!! Accept all routes expect default and host routes.
    edit "Rmap_Border_Leaf_In"
        config rule
            edit 4294967293
                set action deny
                set match-ip-address "Pfx_Default_Route_Only"
            next
            edit 4294967294
                set action deny
                set match-ip-address "Pfx_Host_Route_Only"
            next
            edit 4294967295
            next
        end
    next
end

!!! BGP configuration in Blue vdom.
Ns-Fw-02 (Vdom_Blue) # show router bgp 
config router bgp
    set as 65535
    set router-id 10.85.4.1
    !!! Enable ebgp multipath for ecmp.
    set ebgp-multipath enable
    !!! Enable graceful-restart at global level.
    set graceful-restart enable
    config neighbor
        edit "10.81.4.9"
            !!! Enable BGP graceful-restart capability.
            set capability-graceful-restart enable
            !!! Enable ebgp-multihop.
            set ebgp-enforce-multihop enable
            set next-hop-self enable
            set description "Dc-01-Border-Leaf-09"
            set remote-as 4200000001
            !!! Accept all routes from neighbor expect default and host routes.
            set route-map-in "Rmap_Border_Leaf_In"
            !!! Advertise only the default-route to the neighbor.
            set route-map-out "Rmap_Border_Leaf_Out"
            set update-source "loop-ro-vd-blue"
        next
        edit "10.81.4.10"
            set capability-graceful-restart enable
            set ebgp-enforce-multihop enable
            set next-hop-self enable
            set description "Dc-01-Border-Leaf-10"
            set remote-as 4200000001
            set route-map-in "Rmap_Border_Leaf_In"
            set route-map-out "Rmap_Border_Leaf_Out"
            set update-source "loop-ro-vd-blue"
        next
        edit "10.82.4.9"
            set capability-graceful-restart enable
            set ebgp-enforce-multihop enable
            set next-hop-self enable
            set description "Dc-02-Border-Leaf-09"
            set remote-as 4200000002
            set route-map-in "Rmap_Border_Leaf_In"
            set route-map-out "Rmap_Border_Leaf_Out"
            set update-source "loop-ro-vd-blue"
        next
        edit "10.82.4.10"
            set capability-graceful-restart enable
            set ebgp-enforce-multihop enable
            set next-hop-self enable
            set description "Dc-01-Border-Leaf-10"
            set remote-as 4200000002
            set route-map-in "Rmap_Border_Leaf_In"
            set route-map-out "Rmap_Border_Leaf_Out"
            set update-source "loop-ro-vd-blue"
        next
    end
    !!! Enable static route redistribution into BGP so that we can announce the default-route to the neighbors.
    config redistribute "static"
        set status enable
    end
end

We also have required firewall rules in both vdoms - Vdom_root and Vdom_Blue. As this blog covers the routing part only, I am intentionally not showing the firewall rules. 

External routing - VRFLite Verification

Now our Border-Leaf switches should receive only a default-route from the Ns-Fw-Cluster and install it into the Vrf_Blue routing table. Also Border-Leaf switches should announce all the routes expect /32 host routes to the Ns-Fw-Cluster.

Lets start the control plane verification on the VXLAN side -  

!!! Default-route from the firewall is installed.
Dc-01-Border-Leaf-09# show ip route 0.0.0.0 vrf Vrf_Blue 
IP Route Table for VRF "Vrf_Blue"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

!!! Default-route from Ns-Fw-Cluster and next-hop is loopback.
0.0.0.0/0, ubest/mbest: 1/0
    *via 10.85.4.1, [20/0], 03:54:50, bgp-4200000001, external, tag 65535

!!! Verify the backup path over ibgp neighbor (the vPC member switch).
Dc-01-Border-Leaf-09# show ip bgp vrf Vrf_Blue 
BGP routing table information for VRF Vrf_Blue, address family IPv4 Unicast
BGP table version is 36, Local Router ID is 10.81.4.9
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2

   Network            Next Hop            Metric     LocPrf     Weight    Path
!!! Backup default-route from ibgp neighbor in the rib.
* i0.0.0.0/0          10.81.1.10                        100          0           65535 ?
!!! Primary default-route that is installed in the routing table.
*>e                     10.85.4.1                                         0           65535 ?


!!! Default-route in a site-local leaf switch.
Dc-01-Leaf-06# show ip route 0.0.0.0 vrf Vrf_Blue 
IP Route Table for VRF "Vrf_Blue"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

!!! ECMP routes to both site-local border-leaf switches.
0.0.0.0/0, ubest/mbest: 2/0
    !!! Route via Dc-01-Border-Leaf-09.
    *via 10.81.1.9%default, [200/0], 02:44:30, bgp-4200000001, internal, tag 65535, segid: 100000 tunnelid: 0xa510109 en
cap: VXLAN
    !!! Route via Dc-01-Border-Leaf-10.
    *via 10.81.1.10%default, [200/0], 02:44:30, bgp-4200000001, internal, tag 65535, segid: 100000 tunnelid: 0xa51010a e
ncap: VXLAN

!!! Default-route in a site-local border-gateway switch.
Dc-01-Border-Gateway-03# show ip route 0.0.0.0 vrf Vrf_Blue
IP Route Table for VRF "Vrf_Blue"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

!!! ECMP routes to both site-local border-leaf switches.
0.0.0.0/0, ubest/mbest: 2/0
    *via 10.81.1.9%default, [200/0], 02:45:07, bgp-4200000001, internal, tag 65535, segid: 100000 tunnelid: 0xa510109 en
cap: VXLAN
    *via 10.81.1.10%default, [200/0], 02:45:07, bgp-4200000001, internal, tag 65535, segid: 100000 tunnelid: 0xa51010a e

!!! Verify the backup path over multi-site DCI links.
Dc-01-Border-Gateway-03# show ip bgp vrf Vrf_Blue 
BGP routing table information for VRF Vrf_Blue, address family IPv4 Unicast
BGP table version is 17, Local Router ID is 10.81.0.3
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2

   Network            Next Hop            Metric     LocPrf     Weight    Path
!!! Primary default-route that is installed in the routing table.
*>i0.0.0.0/0        10.81.1.9                           100          0         65535 ?
!!! Primary default-route that is installed in the routing table.
*|i                      10.81.1.10                         100          0         65535 ?
!!! Backup default-route from multi-site ebgp neighbor over dci-links in the rib.
* e                     10.83.100.2          2000                     0          4200000002 65535 ?


Now we will do the verification on the firewall side - 

!!! Routing-table in the Ns-Fw-Cluster.
Ns-Fw-01 (Vdom_Blue) # get router info routing-table bgp 
Routing table for VRF=0
!!! 4-way ecmp to 10.91.101.0/0 because vlan 101/l2vni 100101 is stretched in both fabrics. 
B       10.91.101.0/24 
                       [20/0] via 10.81.4.9 (recursive via 10.81.3.17, vlan-3901-dc01), 00:00:12, [1/0]
                       [20/0] via 10.81.4.10 (recursive via 10.81.3.18, vlan-3901-dc01), 00:00:12, [1/0]
                       [20/0] via 10.82.4.9 (recursive via 10.82.3.17, vlan-3902-dc02), 00:00:12, [1/0]
                       [20/0] via 10.82.4.10 (recursive via 10.82.3.18, vlan-3902-dc02), 00:00:12, [1/0]
!!! 2-way ecmp to 10.91.102.0/0 because vlan 102/l2vni 100102 is only available in Dc-01.
B       10.91.102.0/24 
                       [20/0] via 10.81.4.9 (recursive via 10.81.3.17, vlan-3901-dc01), 00:00:12, [1/0]
                       [20/0] via 10.81.4.10 (recursive via 10.81.3.18, vlan-3901-dc01), 00:00:12, [1/0]
!!! 2-way ecmp to 10.91.103.0/0 because vlan 103/l2vni 100103 is only available in Dc-02.
B       10.91.103.0/24 
                       [20/0] via 10.82.4.9 (recursive via 10.82.3.17, vlan-3902-dc02), 00:00:12, [1/0]
                       [20/0] via 10.82.4.10 (recursive via 10.82.3.18, vlan-3902-dc02), 00:00:12, [1/0]

Now we should be able to ping something in the internet -

Vlan-101-Pc-01:~# ping -c 5 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=54 time=13.2 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=54 time=13.6 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=54 time=12.3 ms
64 bytes from 1.1.1.1: icmp_seq=4 ttl=54 time=14.2 ms
64 bytes from 1.1.1.1: icmp_seq=5 ttl=54 time=12.3 ms

--- 1.1.1.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms

That's all for today. There are several ways to provide north-south routing and traffic filtering in VXLAN multi-site design. I have covered one of those with design trade-offs - firewall cannot be gateway for vlan/l2vni and ecmp/asymmetric routing problem is solved by firewall vendor's proprietary solution.

Comments

Popular posts from this blog

Fortigate firewall AAA Configuration for management with TACACS+ protocol and Cisco ISE

802.1x wired authentication with Huawei VRP Switch - (Unified Mode)

High Availability (HA) Cluster Configuration With Fortigate Firewalls (Active-Passive)