VXLAN Journey - Part 14 - Multi-Site VXLAN - North-South Routing - VRFLite with Active/Passive Firewall - ECMP
Now we will provide our Multi-Site data center access to external networks. This is called north-south routing. We will use an active/passive firewall cluster - because it can act as both router and firewall for external traffic. When BGP neighbors are configured at the vrf-level - this is called VRFLite. From firewall perspective - all the paths towards the vxlan multi-site will be ecmp enabled. In vxlan side - each fabric will use two special switches (vPC enabled) - which are called - Border-Leafs.
Our topology looks like below -
For Spine, Leaf and Border-Gateway configurations have a look in the previous blogs. We will only show new configurations required of external-connectivity using Border-Leaf switches and active/passive firewall cluster. The topology shows only the devices participating for vrf-lite external routing.
We are connecting both active and passive firewall to both the child-sites (Dc-01 and Dc-02) because we want to enable ecmp using both fabrics. If Ns-Fw-01 is the active firewall logical topology looks like below -
If Ns-Fw-02 is the active firewall logical topology looks like below -
From firewall's perspective - if Ns-Fw-01 is active then traffic will pass through vpc-port-channel 5 or if Ns-Fw-02 is active then traffic will pass through vpc-port-channel 6. But whichever firewall is active - both of it's port-channels (one port-channel per fabric) will be used for traffic forwarding.
This type of connections though the active/passive firewall cluster always comes with some kind of tradeoffs - because firewall is a stateful network device and we are connecting it to both the fabrics. The limitations of our topology is that -
- Firewall can never be gateway for a vlan/l2vni.
- Firewall's routing ecmp is using two different fabrics. Because of that it will create sub-optimal and asymmetric routing. Sub-optimal routing is not a big problem because - all the links between firewalls and switches will be high-speed links.
- Asymmetric routing will be a big problem because firewall is a stateful device. I am using Fortinet's Fortigate firewalls - which has proprietary support for such kind of topology. It supports maintaining session table even if incoming/outgoing interfaces changes for an already established session.
- To minimize the routing impact during firewall-failover we will be using BGP's graceful-restart capability. Do not forget the rule - graceful-restart and BFD does not work well together. In most cases we have to chose one above another considering the tradeoffs.
- Also for minimum traffic disruption - enable session-pickup in firewall's high-availability configuration.
Disclaimer: Our topology uses Fortigate firewalls which has proprietary support for ECMP when incoming/outgoing interfaces get changed for an established session. If you are implementing such topology in your environment - do not forget to consult with your firewall vendor first.
Let's start configuring our network.
External routing - VRFLite configuration
We are adding 2 new switches per fabric (vPC-pair). They need to be configured as route-reflector client in the Spine switches. We have covered in detail how to configure Spine switches in previous blogs. We will skip the spine configuration - but do not forget to configure the Spine switches. Also we will explain only the new configurations required for Border-Leaf's VRFLite implementation.
Dc-01-Border-Leaf-09 Configuration
hostname Dc-01-Border-Leaf-09
cfs eth distribute
nv overlay evpn
feature ospf
feature bgp
feature interface-vlan
feature vn-segment-vlan-based
feature lacp
feature vpc
feature nv overlay
fabric forwarding anycast-gateway-mac 2020.0000.00aa
vlan 100
name Vlan_Vrf_Blue
vn-segment 100000
vlan 3600
name Vpc_Peer-Link_Svi_Vlan
route-map Rmap_Tag-12345 permit 10
match tag 12345
vrf context Vrf_Blue
vni 100000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context management
system nve infra-vlans 3600
vpc domain 1
peer-switch
role priority 10
peer-keepalive destination 10.10.11.2 source 10.10.11.1
delay restore 150
peer-gateway
layer3 peer-router
auto-recovery reload-delay 360
ip arp synchronize
interface Vlan100
no shutdown
mtu 9216
vrf member Vrf_Blue
no ip redirects
ip forward
no ipv6 redirects
interface Vlan3600
description Svi_Vpc_Peer-Link
no shutdown
mtu 9216
no ip redirects
ip address 10.81.2.9/30
no ipv6 redirects
ip ospf network point-to-point
ip router ospf Underlay area 0.0.0.0
interface port-channel1500
description Vpc_Peer-Link_Po1500
switchport mode trunk
spanning-tree port type network
vpc peer-link
interface nve1
no shutdown
description Vtep_Vxlan
host-reachability protocol bgp
advertise virtual-rmac
source-interface loopback1
member vni 100000 associate-vrf
interface Ethernet1/1
description To_Dc-01-Spine-01_e1/7
no switchport
mtu 9216
medium p2p
no ip redirects
ip unnumbered loopback0
ip router ospf Underlay area 0.0.0.0
no shutdown
interface Ethernet1/2
description To_Dc-01-Spine-02_e1/7
no switchport
mtu 9216
medium p2p
no ip redirects
ip unnumbered loopback0
ip router ospf Underlay area 0.0.0.0
no shutdown
interface Ethernet1/3
description To_Dc-01-Border-Leaf-10_e1/3_Po1500
switchport mode trunk
channel-group 1500 mode active
interface Ethernet1/4
description To_Dc-01-Border-Leaf-10_e1/4_Po1500
switchport mode trunk
channel-group 1500 mode active
interface mgmt0
vrf member management
ip address 10.10.11.1/24
interface loopback0
description Underlay_Routing_Loopback
ip address 10.81.0.9/32
ip router ospf Underlay area 0.0.0.0
interface loopback1
description Overlay_Vtep_Loopback
ip address 10.81.1.9/32
ip address 10.81.1.109/32 secondary
ip router ospf Underlay area 0.0.0.0
router ospf Underlay
router-id 10.81.0.9
log-adjacency-changes detail
router bgp 4200000001
router-id 10.81.0.9
log-neighbor-changes
address-family l2vpn evpn
advertise-pip
neighbor 10.81.0.1
remote-as 4200000001
description Dc-01-Spine-01_Evpn
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 10.81.0.2
remote-as 4200000001
description Dc-01-Spine-02_Evpn
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
vrf Vrf_Blue
address-family ipv4 unicast
redistribute direct route-map Rmap_Tag-12345
maximum-paths ibgp 2
!!! Create the vpc port-channel 5 to Ns-Fw-01.
interface port-channel5
description Port-Channel_Ns-Fw-01
switchport mode trunk
switchport trunk allowed vlan 3901
spanning-tree port type edge trunk
mtu 9216
vpc 5
!!! Create the vpc port-channel 6 to Ns-Fw-02.
interface port-channel6
description Port-Channel_Ns-Fw-02
switchport mode trunk
switchport trunk allowed vlan 3901
spanning-tree port type edge trunk
mtu 9216
vpc 6
!!! Assign physical interface to vpc port-channel 5.
interface Ethernet1/5
description To_Ns-Fw-01_port11_Po5
switchport mode trunk
switchport trunk allowed vlan 3901
mtu 9216
channel-group 5 mode active
!!! Assign physical interface to vpc port-channel 6.
interface Ethernet1/6
description To_Ns-Fw-02_port11_Po6
switchport mode trunk
switchport trunk allowed vlan 3901
mtu 9216
channel-group 6 mode active
!!! Crete the vlan for vrf-lite configuration for Vrf_Blue.
vlan 3901
name Vrf_Blue-Vrf_Lite-Vlan
!!! Crete the vlan-svi for vrf-lite configuration for Vrf_Blue.
interface Vlan3901
description Svi-Vrf_Blue-Vrf_Lite-Vlan
no shutdown
mtu 9216
vrf member Vrf_Blue
no ip redirects
ip address 10.81.3.17/29
!!! Vrf-lite ebgp routing for Vrf_Blue will be establish over loopback interface.
!!! Create a loopback interface for bgp in Vrf_Blue.
interface loopback199
description Loopback-Vrf_Blue
vrf member Vrf_Blue
ip address 10.81.4.9/32
!!! Static routes to reach other bgp neighbor's loopback addresses.
!!! We have two neighbors - ibgp - Dc-01-Border-Leaf-10 and ebgp - Ns-Fw-Cluster.
vrf context Vrf_Blue
!!! Static route for Dc-01-Border-Leaf-10's loopback.
ip route 10.81.4.10/32 10.81.3.18
!!! Static route for Ns-Fw-Cluster's loopback.
ip route 10.85.4.1/32 10.81.3.19
ip prefix-list Pfx_All_Routes seq 10 permit 0.0.0.0/0 le 32
ip prefix-list Pfx_Default_Route_Only seq 10 permit 0.0.0.0/0
ip prefix-list Pfx_Host_Route_Only seq 10 permit 0.0.0.0/0 eq 32
!!! We will only accept a default-route from the Firewall.
route-map Rmap_Ns_Fw_Cluster_In permit 65534
match ip address prefix-list Pfx_Default_Route_Only
route-map Rmap_Ns_Fw_Cluster_In deny 65535
!!! We will only advertise subnet prefixes to the Firewall.
!!! We are dropping all the host-routes (/32).
route-map Rmap_Ns_Fw_Cluster_Out deny 65534
match ip address prefix-list Pfx_Host_Route_Only
route-map Rmap_Ns_Fw_Cluster_Out permit 65535
!!! Vrf-Lite configuration.
router bgp 4200000001
vrf Vrf_Blue
!!! Vrf's router id is loopback 199.
router-id 10.81.4.9
log-neighbor-changes
!!! IBGP to Dc-01-Border-Leaf-10 because neighbor is vpc-peer.
neighbor 10.81.4.10
remote-as 4200000001
description Dc-01-Border-Leaf-10
update-source loopback199
address-family ipv4 unicast
next-hop-self
!!! EBGP to Ns-Fw-Cluster.
neighbor 10.85.4.1
remote-as 65535
description Ns-Fw-Cluster
update-source loopback199
!!! Enable ebgp-multihop.
ebgp-multihop 5
address-family ipv4 unicast
!!! Accept only a default -route from the firewall.
route-map Rmap_Ns_Fw_Cluster_In in
!!! Advertise all routes except /32 host-routes.
route-map Rmap_Ns_Fw_Cluster_Out out
next-hop-self
Dc-01-Border-Leaf-10 Configuration
hostname Dc-01-Border-Leaf-10
cfs eth distribute
nv overlay evpn
feature ospf
feature bgp
feature interface-vlan
feature vn-segment-vlan-based
feature lacp
feature vpc
feature nv overlay
fabric forwarding anycast-gateway-mac 2020.0000.00aa
vlan 100
name Vlan_Vrf_Blue
vn-segment 100000
vlan 3600
name Vpc_Peer-Link_Svi_Vlan
vlan 3901
name Vrf_Blue-Vrf_Lite-Vlan
ip prefix-list Pfx_Default_Route_Only seq 10 permit 0.0.0.0/0
ip prefix-list Pfx_Host_Route_Only seq 10 permit 0.0.0.0/0 eq 32
route-map Rmap_Ns_Fw_Cluster_In permit 65534
match ip address prefix-list Pfx_Default_Route_Only
route-map Rmap_Ns_Fw_Cluster_In deny 65535
route-map Rmap_Ns_Fw_Cluster_Out deny 65534
match ip address prefix-list Pfx_Host_Route_Only
route-map Rmap_Ns_Fw_Cluster_Out permit 65535
route-map Rmap_Tag-12345 permit 10
match tag 12345
vrf context Vrf_Blue
vni 100000
ip route 10.81.4.9/32 10.81.3.17
ip route 10.85.4.1/32 10.81.3.19
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context management
system nve infra-vlans 3600
vpc domain 1
peer-switch
role priority 20
peer-keepalive destination 10.10.11.1 source 10.10.11.2
delay restore 150
peer-gateway
layer3 peer-router
auto-recovery reload-delay 360
ip arp synchronize
interface Vlan100
no shutdown
mtu 9216
vrf member Vrf_Blue
no ip redirects
ip forward
no ipv6 redirects
interface Vlan3600
description Svi_Vpc_Peer-Link
no shutdown
mtu 9216
no ip redirects
ip address 10.81.2.10/30
no ipv6 redirects
ip ospf network point-to-point
ip router ospf Underlay area 0.0.0.0
interface Vlan3901
description Svi-Vrf_Blue-Vrf_Lite-Vlan
no shutdown
mtu 9216
vrf member Vrf_Blue
no ip redirects
ip address 10.81.3.18/29
no ipv6 redirects
interface port-channel5
description Port-Channel_Ns-Fw-01
switchport mode trunk
switchport trunk allowed vlan 3901
spanning-tree port type edge trunk
mtu 9216
vpc 5
interface port-channel6
description Port-Channel_Ns-Fw-02
switchport mode trunk
switchport trunk allowed vlan 3901
spanning-tree port type edge trunk
mtu 9216
vpc 6
interface port-channel1500
description Vpc_Peer-Link_Po1500
switchport mode trunk
spanning-tree port type network
vpc peer-link
interface nve1
no shutdown
description Vtep_Vxlan
host-reachability protocol bgp
advertise virtual-rmac
source-interface loopback1
member vni 100000 associate-vrf
interface Ethernet1/1
description To_Dc-01-Spine-01_e1/8
no switchport
mtu 9216
medium p2p
no ip redirects
ip unnumbered loopback0
ip router ospf Underlay area 0.0.0.0
no shutdown
interface Ethernet1/2
description To_Dc-01-Spine-02_e1/8
no switchport
mtu 9216
medium p2p
no ip redirects
ip unnumbered loopback0
ip router ospf Underlay area 0.0.0.0
no shutdown
interface Ethernet1/3
description To_Dc-01-Border-Leaf-09_e1/3_Po1500
switchport mode trunk
channel-group 1500 mode active
interface Ethernet1/4
description To_Dc-01-Border-Leaf-09_e1/4_Po1500
switchport mode trunk
channel-group 1500 mode active
interface Ethernet1/5
description To_Ns-Fw-01_port12_Po5
switchport mode trunk
switchport trunk allowed vlan 3901
mtu 9216
channel-group 5 mode active
interface Ethernet1/6
description To_Ns-Fw-02_port12_Po6
switchport mode trunk
switchport trunk allowed vlan 3901
mtu 9216
channel-group 6 mode active
interface mgmt0
vrf member management
ip address 10.10.11.2/24
interface loopback0
description Underlay_Routing_Loopback
ip address 10.81.0.10/32
ip router ospf Underlay area 0.0.0.0
interface loopback1
description Overlay_Vtep_Loopback
ip address 10.81.1.10/32
ip address 10.81.1.109/32 secondary
ip router ospf Underlay area 0.0.0.0
interface loopback199
description Loopback-Vrf_Blue
vrf member Vrf_Blue
ip address 10.81.4.10/32
router ospf Underlay
router-id 10.81.0.10
log-adjacency-changes detail
router bgp 4200000001
router-id 10.81.0.10
log-neighbor-changes
address-family l2vpn evpn
advertise-pip
neighbor 10.81.0.1
remote-as 4200000001
description Dc-01-Spine-01_Evpn
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 10.81.0.2
remote-as 4200000001
description Dc-01-Spine-02_Evpn
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
vrf Vrf_Blue
router-id 10.81.4.10
log-neighbor-changes
address-family ipv4 unicast
redistribute direct route-map Rmap_Tag-12345
maximum-paths ibgp 2
neighbor 10.81.4.9
remote-as 4200000001
description Dc-01-Border-Leaf-09
update-source loopback199
address-family ipv4 unicast
next-hop-self
neighbor 10.85.4.1
remote-as 65535
description Ns-Fw-Cluster
update-source loopback199
ebgp-multihop 5
address-family ipv4 unicast
route-map Rmap_Ns_Fw_Cluster_In in
route-map Rmap_Ns_Fw_Cluster_Out out
next-hop-self
Dc-02-Border-Leaf-09 Configuration
hostname Dc-02-Border-Leaf-09
cfs eth distribute
nv overlay evpn
feature ospf
feature bgp
feature interface-vlan
feature vn-segment-vlan-based
feature lacp
feature vpc
feature nv overlay
fabric forwarding anycast-gateway-mac 2020.0000.00aa
vlan 100
name Vlan_Vrf_Blue
vn-segment 100000
vlan 3600
name Vpc_Peer-Link_Svi_Vlan
vlan 3902
name Vrf_Blue-Vrf_Lite-Vlan
ip prefix-list Pfx_Default_Route_Only seq 10 permit 0.0.0.0/0
ip prefix-list Pfx_Host_Route_Only seq 10 permit 0.0.0.0/0 eq 32
route-map Rmap_Ns_Fw_Cluster_In permit 65534
match ip address prefix-list Pfx_Default_Route_Only
route-map Rmap_Ns_Fw_Cluster_In deny 65535
route-map Rmap_Ns_Fw_Cluster_Out deny 65534
match ip address prefix-list Pfx_Host_Route_Only
route-map Rmap_Ns_Fw_Cluster_Out permit 65535
route-map Rmap_Tag-12345 permit 10
match tag 12345
vrf context Vrf_Blue
vni 100000
ip route 10.82.4.10/32 10.82.3.18
ip route 10.85.4.1/32 10.82.3.19
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context management
system nve infra-vlans 3600
vpc domain 1
peer-switch
role priority 10
peer-keepalive destination 10.10.11.2 source 10.10.11.1
delay restore 150
peer-gateway
layer3 peer-router
auto-recovery reload-delay 360
ip arp synchronize
interface Vlan100
no shutdown
mtu 9216
vrf member Vrf_Blue
no ip redirects
ip forward
no ipv6 redirects
interface Vlan3600
description Svi_Vpc_Peer-Link
no shutdown
mtu 9216
no ip redirects
ip address 10.82.2.9/30
no ipv6 redirects
ip ospf network point-to-point
ip router ospf Underlay area 0.0.0.0
interface Vlan3902
description Svi-Vrf_Blue-Vrf_Lite-Vlan
no shutdown
mtu 9216
vrf member Vrf_Blue
no ip redirects
ip address 10.82.3.17/29
no ipv6 redirects
interface port-channel5
description Port-Channel_Ns-Fw-01
switchport mode trunk
switchport trunk allowed vlan 3902
spanning-tree port type edge trunk
mtu 9216
vpc 5
interface port-channel6
description Port-Channel_Ns-Fw-02
switchport mode trunk
switchport trunk allowed vlan 3902
spanning-tree port type edge trunk
mtu 9216
vpc 6
interface port-channel1500
description Vpc_Peer-Link_Po1500
switchport mode trunk
spanning-tree port type network
vpc peer-link
interface nve1
no shutdown
description Vtep_Vxlan
host-reachability protocol bgp
advertise virtual-rmac
source-interface loopback1
member vni 100000 associate-vrf
interface Ethernet1/1
description To_Dc-02-Spine-01_e1/7
no switchport
mtu 9216
medium p2p
no ip redirects
ip unnumbered loopback0
ip router ospf Underlay area 0.0.0.0
no shutdown
interface Ethernet1/2
description To_Dc-02-Spine-02_e1/7
no switchport
mtu 9216
medium p2p
no ip redirects
ip unnumbered loopback0
ip router ospf Underlay area 0.0.0.0
no shutdown
interface Ethernet1/3
description To_Dc-02-Border-Leaf-10_e1/3_Po1500
switchport mode trunk
channel-group 1500 mode active
interface Ethernet1/4
description To_Dc-02-Border-Leaf-10_e1/4_Po1500
switchport mode trunk
channel-group 1500 mode active
interface Ethernet1/5
description To_Ns-Fw-01_port13_Po5
switchport mode trunk
switchport trunk allowed vlan 3902
mtu 9216
channel-group 5 mode active
interface Ethernet1/6
description To_Ns-Fw-02_port13_Po6
switchport mode trunk
switchport trunk allowed vlan 3902
mtu 9216
channel-group 6 mode active
interface mgmt0
vrf member management
ip address 10.10.11.1/24
interface loopback0
description Underlay_Routing_Loopback
ip address 10.82.0.9/32
ip router ospf Underlay area 0.0.0.0
interface loopback1
description Overlay_Vtep_Loopback
ip address 10.82.1.9/32
ip address 10.82.1.109/32 secondary
ip router ospf Underlay area 0.0.0.0
interface loopback199
description Loopback-Vrf_Blue
vrf member Vrf_Blue
ip address 10.82.4.9/32
router ospf Underlay
router-id 10.82.0.9
log-adjacency-changes detail
router bgp 4200000002
router-id 10.82.0.9
log-neighbor-changes
address-family l2vpn evpn
advertise-pip
neighbor 10.82.0.1
remote-as 4200000002
description Dc-02-Spine-01_Evpn
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 10.82.0.2
remote-as 4200000002
description Dc-02-Spine-02_Evpn
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
vrf Vrf_Blue
router-id 10.82.4.9
log-neighbor-changes
address-family ipv4 unicast
redistribute direct route-map Rmap_Tag-12345
maximum-paths ibgp 2
neighbor 10.82.4.10
remote-as 4200000002
description Dc-02-Border-Leaf-10
update-source loopback199
address-family ipv4 unicast
next-hop-self
neighbor 10.85.4.1
remote-as 65535
description Ns-Fw-Cluster
update-source loopback199
ebgp-multihop 5
address-family ipv4 unicast
route-map Rmap_Ns_Fw_Cluster_In in
route-map Rmap_Ns_Fw_Cluster_Out out
next-hop-self
Dc-02-Border-Leaf-10 Configuration
hostname Dc-02-Border-Leaf-10
cfs eth distribute
nv overlay evpn
feature ospf
feature bgp
feature interface-vlan
feature vn-segment-vlan-based
feature lacp
feature vpc
feature nv overlay
fabric forwarding anycast-gateway-mac 2020.0000.00aa
vlan 100
name Vlan_Vrf_Blue
vn-segment 100000
vlan 3600
name Vpc_Peer-Link_Svi_Vlan
vlan 3902
name Vrf_Blue-Vrf_Lite-Vlan
ip prefix-list Pfx_Default_Route_Only seq 10 permit 0.0.0.0/0
ip prefix-list Pfx_Host_Route_Only seq 10 permit 0.0.0.0/0 eq 32
route-map Rmap_Ns_Fw_Cluster_In permit 65534
match ip address prefix-list Pfx_Default_Route_Only
route-map Rmap_Ns_Fw_Cluster_In deny 65535
route-map Rmap_Ns_Fw_Cluster_Out deny 65534
match ip address prefix-list Pfx_Host_Route_Only
route-map Rmap_Ns_Fw_Cluster_Out permit 65535
route-map Rmap_Tag-12345 permit 10
match tag 12345
vrf context Vrf_Blue
vni 100000
ip route 10.82.4.9/32 10.82.3.17
ip route 10.85.4.1/32 10.82.3.19
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context management
system nve infra-vlans 3600
vpc domain 1
peer-switch
role priority 20
peer-keepalive destination 10.10.11.1 source 10.10.11.2
delay restore 150
peer-gateway
layer3 peer-router
auto-recovery reload-delay 360
ip arp synchronize
interface Vlan100
no shutdown
mtu 9216
vrf member Vrf_Blue
no ip redirects
ip forward
no ipv6 redirects
interface Vlan3600
description Svi_Vpc_Peer-Link
no shutdown
mtu 9216
no ip redirects
ip address 10.82.2.10/30
no ipv6 redirects
ip ospf network point-to-point
ip router ospf Underlay area 0.0.0.0
interface Vlan3902
description Svi-Vrf_Blue-Vrf_Lite-Vlan
no shutdown
mtu 9216
vrf member Vrf_Blue
no ip redirects
ip address 10.82.3.18/29
no ipv6 redirects
interface port-channel5
description Port-Channel_Ns-Fw-01
switchport mode trunk
switchport trunk allowed vlan 3902
spanning-tree port type edge trunk
mtu 9216
vpc 5
interface port-channel6
description Port-Channel_Ns-Fw-02
switchport mode trunk
switchport trunk allowed vlan 3902
spanning-tree port type edge trunk
mtu 9216
vpc 6
interface port-channel1500
description Vpc_Peer-Link_Po1500
switchport mode trunk
spanning-tree port type network
vpc peer-link
interface nve1
no shutdown
description Vtep_Vxlan
host-reachability protocol bgp
advertise virtual-rmac
source-interface loopback1
member vni 100000 associate-vrf
interface Ethernet1/1
description To_Dc-02-Spine-01_e1/8
no switchport
mtu 9216
medium p2p
no ip redirects
ip unnumbered loopback0
ip router ospf Underlay area 0.0.0.0
no shutdown
interface Ethernet1/2
description To_Dc-02-Spine-02_e1/8
no switchport
mtu 9216
medium p2p
no ip redirects
ip unnumbered loopback0
ip router ospf Underlay area 0.0.0.0
no shutdown
interface Ethernet1/3
description To_Dc-02-Border-Leaf-09_e1/3_Po1500
switchport mode trunk
channel-group 1500 mode active
interface Ethernet1/4
description To_Dc-02-Border-Leaf-09_e1/4_Po1500
switchport mode trunk
channel-group 1500 mode active
interface Ethernet1/5
description To_Ns-Fw-01_port14_Po5
switchport mode trunk
switchport trunk allowed vlan 3902
mtu 9216
channel-group 5 mode active
interface Ethernet1/6
description To_Ns-Fw-02_port14_Po6
switchport mode trunk
switchport trunk allowed vlan 3902
mtu 9216
channel-group 6 mode active
interface mgmt0
vrf member management
ip address 10.10.11.2/24
interface loopback0
description Underlay_Routing_Loopback
ip address 10.82.0.10/32
ip router ospf Underlay area 0.0.0.0
interface loopback1
description Overlay_Vtep_Loopback
ip address 10.82.1.10/32
ip address 10.82.1.109/32 secondary
ip router ospf Underlay area 0.0.0.0
interface loopback199
description Loopback-Vrf_Blue
vrf member Vrf_Blue
ip address 10.82.4.10/32
router ospf Underlay
router-id 10.82.0.10
log-adjacency-changes detail
router bgp 4200000002
router-id 10.82.0.10
log-neighbor-changes
address-family l2vpn evpn
advertise-pip
neighbor 10.82.0.1
remote-as 4200000002
description Dc-02-Spine-01_Evpn
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 10.82.0.2
remote-as 4200000002
description Dc-02-Spine-02_Evpn
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
vrf Vrf_Blue
router-id 10.82.4.10
log-neighbor-changes
address-family ipv4 unicast
redistribute direct route-map Rmap_Tag-12345
maximum-paths ibgp 2
neighbor 10.82.4.9
remote-as 4200000002
description Dc-02-Border-Leaf-09
update-source loopback199
address-family ipv4 unicast
next-hop-self
neighbor 10.85.4.1
remote-as 65535
description Ns-Fw-Cluster
update-source loopback199
ebgp-multihop 5
address-family ipv4 unicast
route-map Rmap_Ns_Fw_Cluster_In in
route-map Rmap_Ns_Fw_Cluster_Out out
next-hop-self
Ns-Fw-Cluster Configuration
We are showing the firewall configuration as one device because it is an active/passive cluster. In the firewall we have enabled two vdoms - Vdom_root and Vdom_Blue. Vdom_root has connectivity to the internet. Vdom_Blue represent the Vrf_Blue in the fabric. There is vdom-link between Vdom_root and Vdom_Blue which allows the vdoms to communicate with each other.
config system interface
!!! port2 is connected to an upstream router which has internet connectivity.
edit "port2"
set vdom "root"
set ip 172.29.215.61 255.255.255.0
next
!!! Port 11 and 12 is connected with Dc-01 Border-Leaf switches.
!!! These two ports are configured for port-channel.
edit "port11"
set vdom "root"
next
edit "port12"
set vdom "root"
next
!!! Port 13 and 14 is connected with Dc-02 Border-Leaf switches.
!!! These two ports are configured for port-channel.
edit "port13"
set vdom "root"
next
edit "port14"
set vdom "root"
next
!!! Vdom-link interface in root vdom.
edit "vd-l-ro-blu0"
set vdom "root"
set ip 10.85.2.1 255.255.255.252
set allowaccess ping
set type vdom-link
next
!!! Vdom-link interface in Blue vdom.
edit "vd-l-ro-blu1"
set vdom "Vdom_Blue"
set ip 10.85.2.2 255.255.255.252
set allowaccess ping
set type vdom-link
next
!!! Dc-01 Border-Leaf port-channel.
edit "Dc-01-Blf-Po1"
set vdom "root"
set type aggregate
set member "port11" "port12"
set mtu-override enable
set mtu 9216
next
!!! Dc-02 Border-Leaf port-channel.
edit "Dc-02-Blf-Po1"
set vdom "root"
set type aggregate
set member "port13" "port14"
set mtu-override enable
set mtu 9216
next
!!! Vlan-Svi with Dc-01 Border-Leaf switches.
edit "vlan-3901-dc01"
set vdom "Vdom_Blue"
set ip 10.81.3.19 255.255.255.248
set allowaccess ping
set interface "Dc-01-Blf-Po1"
set vlanid 3901
next
!!! Vlan-Svi with Dc-02 Border-Leaf switches.
edit "vlan-3902-dc02"
set vdom "Vdom_Blue"
set ip 10.82.3.19 255.255.255.248
set allowaccess ping
set interface "Dc-02-Blf-Po1"
set vlanid 3902
next
!!! BGP routing loopback.
edit "loop-ro-vd-blue"
set vdom "Vdom_Blue"
set ip 10.85.4.1 255.255.255.255
set allowaccess ping
set type loopback
set snmp-index 28
next
end
!!! Static route configuration in Vdom_root.
Ns-Fw-02 (root) # show router static
config router static
!!! Default static route.
edit 1
set gateway 172.29.215.1
set device "port2"
next
!!! Static route to Vdom_Blue for 10.91.0.0/16.
edit 2
set dst 10.91.0.0 255.255.0.0
set gateway 10.85.2.2
set device "vd-l-ro-blu0"
next
end
!!! Static route configuration in Vdom_Blue.
Ns-Fw-02 (Vdom_Blue) # show router static
config router static
!!! Default static route - which points to Vdom_root.
edit 5
set gateway 10.85.2.1
set device "vd-l-ro-blu1"
next
!!! Static route route to Dc-01-Border-Leaf-09's loopback 199.
edit 2
set dst 10.81.4.9 255.255.255.255
set gateway 10.81.3.17
set device "vlan-3901-dc01"
next
!!! Static route route to Dc-01-Border-Leaf-10's loopback 199.
edit 3
set dst 10.81.4.10 255.255.255.255
set gateway 10.81.3.18
set device "vlan-3901-dc01"
next
!!! Static route route to Dc-02-Border-Leaf-09's loopback 199.
edit 4
set dst 10.82.4.9 255.255.255.255
set gateway 10.82.3.17
set device "vlan-3902-dc02"
next
!!! Static route route to Dc-02-Border-Leaf-10's loopback 199.
edit 6
set dst 10.82.4.10 255.255.255.255
set gateway 10.82.3.18
set device "vlan-3902-dc02"
next
end
!!! Prefix-List for VRFLite routing.
Ns-Fw-02 (Vdom_Blue) # show router prefix-list
config router prefix-list
!!! Match default-route.
edit "Pfx_Default_Route_Only"
config rule
edit 1
set prefix 0.0.0.0 0.0.0.0
unset ge
unset le
next
end
next
!!! Match host-route.
edit "Pfx_Host_Route_Only"
config rule
edit 1
set prefix 0.0.0.0 0.0.0.0
set ge 32
set le 32
next
end
next
end
!!! Route-maps for VRFLite routing.
Ns-Fw-02 (Vdom_Blue) # show router route-map
config router route-map
!!! Advertise only default route.
edit "Rmap_Border_Leaf_Out"
config rule
edit 4294967294
set match-ip-address "Pfx_Default_Route_Only"
next
edit 4294967295
set action deny
next
end
next
!!! Accept all routes expect default and host routes.
edit "Rmap_Border_Leaf_In"
config rule
edit 4294967293
set action deny
set match-ip-address "Pfx_Default_Route_Only"
next
edit 4294967294
set action deny
set match-ip-address "Pfx_Host_Route_Only"
next
edit 4294967295
next
end
next
end
!!! BGP configuration in Blue vdom.
Ns-Fw-02 (Vdom_Blue) # show router bgp
config router bgp
set as 65535
set router-id 10.85.4.1
!!! Enable ebgp multipath for ecmp.
set ebgp-multipath enable
!!! Enable graceful-restart at global level.
set graceful-restart enable
config neighbor
edit "10.81.4.9"
!!! Enable BGP graceful-restart capability.
set capability-graceful-restart enable
!!! Enable ebgp-multihop.
set ebgp-enforce-multihop enable
set next-hop-self enable
set description "Dc-01-Border-Leaf-09"
set remote-as 4200000001
!!! Accept all routes from neighbor expect default and host routes.
set route-map-in "Rmap_Border_Leaf_In"
!!! Advertise only the default-route to the neighbor.
set route-map-out "Rmap_Border_Leaf_Out"
set update-source "loop-ro-vd-blue"
next
edit "10.81.4.10"
set capability-graceful-restart enable
set ebgp-enforce-multihop enable
set next-hop-self enable
set description "Dc-01-Border-Leaf-10"
set remote-as 4200000001
set route-map-in "Rmap_Border_Leaf_In"
set route-map-out "Rmap_Border_Leaf_Out"
set update-source "loop-ro-vd-blue"
next
edit "10.82.4.9"
set capability-graceful-restart enable
set ebgp-enforce-multihop enable
set next-hop-self enable
set description "Dc-02-Border-Leaf-09"
set remote-as 4200000002
set route-map-in "Rmap_Border_Leaf_In"
set route-map-out "Rmap_Border_Leaf_Out"
set update-source "loop-ro-vd-blue"
next
edit "10.82.4.10"
set capability-graceful-restart enable
set ebgp-enforce-multihop enable
set next-hop-self enable
set description "Dc-01-Border-Leaf-10"
set remote-as 4200000002
set route-map-in "Rmap_Border_Leaf_In"
set route-map-out "Rmap_Border_Leaf_Out"
set update-source "loop-ro-vd-blue"
next
end
!!! Enable static route redistribution into BGP so that we can announce the default-route to the neighbors.
config redistribute "static"
set status enable
end
end
We also have required firewall rules in both vdoms - Vdom_root and Vdom_Blue. As this blog covers the routing part only, I am intentionally not showing the firewall rules.
External routing - VRFLite Verification
Now our Border-Leaf switches should receive only a default-route from the Ns-Fw-Cluster and install it into the Vrf_Blue routing table. Also Border-Leaf switches should announce all the routes expect /32 host routes to the Ns-Fw-Cluster.
Lets start the control plane verification on the VXLAN side -
!!! Default-route from the firewall is installed.
Dc-01-Border-Leaf-09# show ip route 0.0.0.0 vrf Vrf_Blue
IP Route Table for VRF "Vrf_Blue"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
!!! Default-route from Ns-Fw-Cluster and next-hop is loopback.
0.0.0.0/0, ubest/mbest: 1/0
*via 10.85.4.1, [20/0], 03:54:50, bgp-4200000001, external, tag 65535
!!! Verify the backup path over ibgp neighbor (the vPC member switch).
Dc-01-Border-Leaf-09# show ip bgp vrf Vrf_Blue
BGP routing table information for VRF Vrf_Blue, address family IPv4 Unicast
BGP table version is 36, Local Router ID is 10.81.4.9
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2
Network Next Hop Metric LocPrf Weight Path
!!! Backup default-route from ibgp neighbor in the rib.
* i0.0.0.0/0 10.81.1.10 100 0 65535 ?
!!! Primary default-route that is installed in the routing table.
*>e 10.85.4.1 0 65535 ?
!!! Default-route in a site-local leaf switch.
Dc-01-Leaf-06# show ip route 0.0.0.0 vrf Vrf_Blue
IP Route Table for VRF "Vrf_Blue"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
!!! ECMP routes to both site-local border-leaf switches.
0.0.0.0/0, ubest/mbest: 2/0
!!! Route via Dc-01-Border-Leaf-09.
*via 10.81.1.9%default, [200/0], 02:44:30, bgp-4200000001, internal, tag 65535, segid: 100000 tunnelid: 0xa510109 en
cap: VXLAN
!!! Route via Dc-01-Border-Leaf-10.
*via 10.81.1.10%default, [200/0], 02:44:30, bgp-4200000001, internal, tag 65535, segid: 100000 tunnelid: 0xa51010a e
ncap: VXLAN
!!! Default-route in a site-local border-gateway switch.
Dc-01-Border-Gateway-03# show ip route 0.0.0.0 vrf Vrf_Blue
IP Route Table for VRF "Vrf_Blue"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
!!! ECMP routes to both site-local border-leaf switches.
0.0.0.0/0, ubest/mbest: 2/0
*via 10.81.1.9%default, [200/0], 02:45:07, bgp-4200000001, internal, tag 65535, segid: 100000 tunnelid: 0xa510109 en
cap: VXLAN
*via 10.81.1.10%default, [200/0], 02:45:07, bgp-4200000001, internal, tag 65535, segid: 100000 tunnelid: 0xa51010a e
!!! Verify the backup path over multi-site DCI links.
Dc-01-Border-Gateway-03# show ip bgp vrf Vrf_Blue
BGP routing table information for VRF Vrf_Blue, address family IPv4 Unicast
BGP table version is 17, Local Router ID is 10.81.0.3
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2
Network Next Hop Metric LocPrf Weight Path
!!! Primary default-route that is installed in the routing table.
*>i0.0.0.0/0 10.81.1.9 100 0 65535 ?
!!! Primary default-route that is installed in the routing table.
*|i 10.81.1.10 100 0 65535 ?
!!! Backup default-route from multi-site ebgp neighbor over dci-links in the rib.
* e 10.83.100.2 2000 0 4200000002 65535 ?
Now we will do the verification on the firewall side -
!!! Routing-table in the Ns-Fw-Cluster.
Ns-Fw-01 (Vdom_Blue) # get router info routing-table bgp
Routing table for VRF=0
!!! 4-way ecmp to 10.91.101.0/0 because vlan 101/l2vni 100101 is stretched in both fabrics.
B 10.91.101.0/24
[20/0] via 10.81.4.9 (recursive via 10.81.3.17, vlan-3901-dc01), 00:00:12, [1/0]
[20/0] via 10.81.4.10 (recursive via 10.81.3.18, vlan-3901-dc01), 00:00:12, [1/0]
[20/0] via 10.82.4.9 (recursive via 10.82.3.17, vlan-3902-dc02), 00:00:12, [1/0]
[20/0] via 10.82.4.10 (recursive via 10.82.3.18, vlan-3902-dc02), 00:00:12, [1/0]
!!! 2-way ecmp to 10.91.102.0/0 because vlan 102/l2vni 100102 is only available in Dc-01.
B 10.91.102.0/24
[20/0] via 10.81.4.9 (recursive via 10.81.3.17, vlan-3901-dc01), 00:00:12, [1/0]
[20/0] via 10.81.4.10 (recursive via 10.81.3.18, vlan-3901-dc01), 00:00:12, [1/0]
!!! 2-way ecmp to 10.91.103.0/0 because vlan 103/l2vni 100103 is only available in Dc-02.
B 10.91.103.0/24
[20/0] via 10.82.4.9 (recursive via 10.82.3.17, vlan-3902-dc02), 00:00:12, [1/0]
[20/0] via 10.82.4.10 (recursive via 10.82.3.18, vlan-3902-dc02), 00:00:12, [1/0]
Now we should be able to ping something in the internet -
Vlan-101-Pc-01:~# ping -c 5 1.1.1.1
PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.
64 bytes from 1.1.1.1: icmp_seq=1 ttl=54 time=13.2 ms
64 bytes from 1.1.1.1: icmp_seq=2 ttl=54 time=13.6 ms
64 bytes from 1.1.1.1: icmp_seq=3 ttl=54 time=12.3 ms
64 bytes from 1.1.1.1: icmp_seq=4 ttl=54 time=14.2 ms
64 bytes from 1.1.1.1: icmp_seq=5 ttl=54 time=12.3 ms
--- 1.1.1.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
That's all for today. There are several ways to provide north-south routing and traffic filtering in VXLAN multi-site design. I have covered one of those with design trade-offs - firewall cannot be gateway for vlan/l2vni and ecmp/asymmetric routing problem is solved by firewall vendor's proprietary solution.



Comments
Post a Comment