VXLAN Journey - Part 10 - Multi-Site VXLAN Anycast Border Gateway Ip-Vrf (L3VNI) extension - advertise-pip
In part - 09 we have prepared our anycast border-gateways for underlay and overlay bgp configuration. But we have not done any l3vni/l2vni extension. In this blog we will configure ip-vrf (l3vni) extension in our border-gateways. Along the way we will troubleshoot something called rouge-vtep which is Cisco's proprietary always-on native data plane security. And solve that rouge-vtep problem with another proprietary solution called advertise-pip.
What is multi-site? When we connect two different dc-fabrics (different BGP ASN) together with eBGP l2vpn-evpn - we call it vxlan multi-site extension.
What is l3vni extension? If vlan - A, under vrf - X, from Site - 1 can communicate over the border-gateways with vlan - B, under vrf - X from Site - 2; we call it l3vni/ip-vrf extension over multisite. The common thing in the extension is that communicating vlans belong to the same vrf (l3vni).
Our topology looks like below -
01 - Network Topology
For Spine, Leaf and Border-Gateway configuration have a look in the previous blogs. We will only show new configurations required of l3vni extension over two different data-center fabrics. Also we are only showing the relevant Leaf switches which has an end-host relevant for our extension. Other Leaf switches are still there; they are not just relevant for this blog.
Dc-01 L3VNI Extension Configuration
Dc-01-Border-Gateway-03 Configuration
!!! Define the vrf.
vrf context Vrf_Blue
description Vrf_Blue
vni 100000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
!!! Create the vlan required of vrf/l3vni.
vlan 100
name Vrf_Blue_Vlan
vn-segment 100000
!!! Create the vlan svi for l3vni/vrf.
!!! This is the only vlan svi type allowed in a anycast border-gateway switch.
!!! Never create any other type of vlan svi in anycast border-gateway switch.
interface Vlan100
description Svi_Vrf_Blue
no shutdown
mtu 9216
vrf member Vrf_Blue
no ip redirects
ip forward
!!! Associate the l3vni with nve interface for vxlan forwarding.
interface nve1
member vni 100000 associate-vrf
!!! Enable bgp route-leaking between global (l2vpn-evpn) and vrf routing (ipv4 unicast) tables.
router bgp 4200000001
vrf Vrf_Blue
address-family ipv4 unicast
maximum-paths 2
maximum-paths ibgp 2
Dc-01-Border-Gateway-04 Configuration
vrf context Vrf_Blue
description Vrf_Blue
vni 100000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vlan 100
name Vrf_Blue_Vlan
vn-segment 100000
interface Vlan100
description Svi_Vrf_Blue
no shutdown
mtu 9216
vrf member Vrf_Blue
no ip redirects
ip forward
interface nve1
member vni 100000 associate-vrf
router bgp 4200000001
vrf Vrf_Blue
address-family ipv4 unicast
maximum-paths 2
maximum-paths ibgp 2
Dc-02 L3VNI Extension Configuration
Dc-02-Border-Gateway-03 Configuration
vrf context Vrf_Blue
description Vrf_Blue
vni 100000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vlan 100
name Vrf_Blue_Vlan
vn-segment 100000
interface Vlan100
description Svi_Vrf_Blue
no shutdown
mtu 9216
vrf member Vrf_Blue
no ip redirects
ip forward
interface nve1
member vni 100000 associate-vrf
router bgp 4200000001
vrf Vrf_Blue
address-family ipv4 unicast
maximum-paths 2
maximum-paths ibgp 2
Dc-02-Border-Gateway-04 Configuration
vrf context Vrf_Blue
description Vrf_Blue
vni 100000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vlan 100
name Vrf_Blue_Vlan
vn-segment 100000
interface Vlan100
description Svi_Vrf_Blue
no shutdown
mtu 9216
vrf member Vrf_Blue
no ip redirects
ip forward
interface nve1
member vni 100000 associate-vrf
router bgp 4200000001
vrf Vrf_Blue
address-family ipv4 unicast
maximum-paths 2
maximum-paths ibgp 2
VRF/L3VNI extension control-plane verification
We will verify the routes that is required for communication between Vlan-102-Pc-02 (10.91.102.12) and Vlan-103-Pc-01 (10.91.103.11).
Vlan-102-Pc-02 (10.91.102.12) is located at Dc-01. Let's have look what the Dc-01 border-gateways are doing with the routes -
!!! Type 2 (mac+ip) route for Vlan-102-Pc-02 (10.91.102.12).
Dc-01-Border-Gateway-04# show bgp l2vpn evpn 10.91.102.12
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 10.81.0.6:32869
BGP routing table entry for [2]:[0]:[0]:[48]:[5000.4d00.7e00]:[32]:[10.91.102.12]/272, version 1170
Paths: (2 available, best #2)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
!!! type2 (mac+ip) reflected route received from Dc-01-Spine-02.
Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
AS-Path: NONE, path sourced internal to AS
10.81.1.6 (metric 81) from 10.81.0.2 (10.81.0.2)
Origin IGP, MED not set, localpref 100, weight 0
Received label 100102 100000
Extcommunity: RT:23456:100000 RT:23456:100102 ENCAP:8 Router MAC:5000.2900.1b08
Originator: 10.81.0.6 Cluster list: 10.81.0.2
Advertised path-id 1
!!! type2 (mac+ip) reflected route received from Dc-01-Spine-01.
Path type: internal, path is valid, is best path, no labeled nexthop
Imported to 2 destination(s)
Imported paths list: Vrf_Blue L3-100000
AS-Path: NONE, path sourced internal to AS
10.81.1.6 (metric 81) from 10.81.0.1 (10.81.0.1)
Origin IGP, MED not set, localpref 100, weight 0
Received label 100102 100000
Extcommunity: RT:23456:100000 RT:23456:100102 ENCAP:8 Router MAC:5000.2900.1b08
Originator: 10.81.0.6 Cluster list: 10.81.0.1
Path-id 1 not advertised to any peer
!!! type2 (mac+ip) route installed in local rib.
Route Distinguisher: 10.81.0.4:6 (L3VNI 100000)
BGP routing table entry for [2]:[0]:[0]:[48]:[5000.4d00.7e00]:[32]:[10.91.102.12]/272, version 1178
Paths: (1 available, best #1)
Flags: (0x000202) (high32 0x000400) on xmit-list, is not in l2rib/evpn, is not in HW
Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop
Imported from 10.81.0.6:32869:[2]:[0]:[0]:[48]:[5000.4d00.7e00]:[32]:[10.91.102.12]/272
AS-Path: NONE, path sourced internal to AS
10.81.1.6 (metric 81) from 10.81.0.1 (10.81.0.1)
Origin IGP, MED not set, localpref 100, weight 0
Received label 100102 100000
Extcommunity: RT:23456:100000 RT:23456:100102 ENCAP:8 Router MAC:5000.2900.1b08
Originator: 10.81.0.6 Cluster list: 10.81.0.1
!!! type2 (mac+ip) is not announced to Dc-02 border-gateways because we are doing l3vni extension only.
Path-id 1 (dual) not advertised to any peer
From above output important thing is that even though we have received a /32 host route for Vlan-102-Pc-02 (10.91.102.12), it is suppressed by the Dc-01 border-gateways and it is not announced towards Dc-02 border-gateways. This is expected as we are doing l3vni extension only.
!!! type-5 (prefix-route) for vlan 102 - 10.91.102.0/24.
Dc-01-Border-Gateway-04# show bgp l2vpn evpn 10.91.102.0
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 10.81.0.6:4
BGP routing table entry for [5]:[0]:[0]:[24]:[10.91.102.0]/224, version 1171
Paths: (2 available, best #2)
Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
!!! type5 (prefix-route) reflected route received from Dc-01-Spine-02 originated by Dc-01-Leaf-06.
Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
Gateway IP: 0.0.0.0
AS-Path: NONE, path sourced internal to AS
10.81.1.6 (metric 81) from 10.81.0.2 (10.81.0.2)
Origin incomplete, MED 0, localpref 100, weight 0
Received label 100000
Extcommunity: RT:23456:100000 ENCAP:8 Router MAC:5000.2900.1b08
Originator: 10.81.0.6 Cluster list: 10.81.0.2
Advertised path-id 1
!!! type5 (prefix-route) reflected route received from Dc-01-Spine-01 originated by Dc-01-Leaf-06.
Path type: internal, path is valid, is best path, no labeled nexthop
Imported to 1 destination(s)
Imported paths list: Vrf_Blue
Gateway IP: 0.0.0.0
AS-Path: NONE, path sourced internal to AS
10.81.1.6 (metric 81) from 10.81.0.1 (10.81.0.1)
Origin incomplete, MED 0, localpref 100, weight 0
Received label 100000
Extcommunity: RT:23456:100000 ENCAP:8 Router MAC:5000.2900.1b08
Originator: 10.81.0.6 Cluster list: 10.81.0.1
Path-id 1 not advertised to any peer
Route Distinguisher: 10.81.0.7:4
BGP routing table entry for [5]:[0]:[0]:[24]:[10.91.102.0]/224, version 1168
Paths: (2 available, best #2)
Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
!!! type5 (prefix-route) reflected route received from Dc-01-Spine-02 originated by Dc-01-Leaf-07.
Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
Gateway IP: 0.0.0.0
AS-Path: NONE, path sourced internal to AS
10.81.1.7 (metric 81) from 10.81.0.2 (10.81.0.2)
Origin incomplete, MED 0, localpref 100, weight 0
Received label 100000
Extcommunity: RT:23456:100000 ENCAP:8 Router MAC:5000.2700.1b08
Originator: 10.81.0.7 Cluster list: 10.81.0.2
Advertised path-id 1
!!! type5 (prefix-route) reflected route received from Dc-01-Spine-01 originated by Dc-01-Leaf-07.
Path type: internal, path is valid, is best path, no labeled nexthop
Imported to 1 destination(s)
Imported paths list: Vrf_Blue
Gateway IP: 0.0.0.0
AS-Path: NONE, path sourced internal to AS
10.81.1.7 (metric 81) from 10.81.0.1 (10.81.0.1)
Origin incomplete, MED 0, localpref 100, weight 0
Received label 100000
Extcommunity: RT:23456:100000 ENCAP:8 Router MAC:5000.2700.1b08
Originator: 10.81.0.7 Cluster list: 10.81.0.1
Path-id 1 not advertised to any peer
Route Distinguisher: 10.81.0.8:4
BGP routing table entry for [5]:[0]:[0]:[24]:[10.91.102.0]/224, version 1173
Paths: (2 available, best #2)
Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
!!! type5 (prefix-route) reflected route received from Dc-01-Spine-02 originated by Dc-01-Leaf-08.
Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
Gateway IP: 0.0.0.0
AS-Path: NONE, path sourced internal to AS
10.81.1.8 (metric 81) from 10.81.0.2 (10.81.0.2)
Origin incomplete, MED 0, localpref 100, weight 0
Received label 100000
Extcommunity: RT:23456:100000 ENCAP:8 Router MAC:5000.2800.1b08
Originator: 10.81.0.8 Cluster list: 10.81.0.2
Advertised path-id 1
!!! type5 (prefix-route) reflected route received from Dc-01-Spine-01 originated by Dc-01-Leaf-08.
Path type: internal, path is valid, is best path, no labeled nexthop
Imported to 1 destination(s)
Imported paths list: Vrf_Blue
Gateway IP: 0.0.0.0
AS-Path: NONE, path sourced internal to AS
10.81.1.8 (metric 81) from 10.81.0.1 (10.81.0.1)
Origin incomplete, MED 0, localpref 100, weight 0
Received label 100000
Extcommunity: RT:23456:100000 ENCAP:8 Router MAC:5000.2800.1b08
Originator: 10.81.0.8 Cluster list: 10.81.0.1
Path-id 1 not advertised to any peer
Route Distinguisher: 10.81.0.4:6 (L3VNI 100000)
BGP routing table entry for [5]:[0]:[0]:[24]:[10.91.102.0]/224, version 1182
Paths: (1 available, best #1)
Flags: (0x000002) (high32 0x000400) on xmit-list, is not in l2rib/evpn
Advertised path-id 1
!!! type5 (prefix-route) route installed in local rib.
Path type: local, path is valid, is best path, reoriginated, no labeled nexthop
Gateway IP: 0.0.0.0
AS-Path: NONE, path locally originated
10.81.1.4 (metric 0) from 0.0.0.0 (10.81.0.4)
Origin incomplete, MED 0, localpref 100, weight 0
Received label 100000
Extcommunity: RT:23456:100000 ENCAP:8 Site-ID:0:4200000001
Router MAC:5000.2600.1b08
Originator: 10.81.0.6 Cluster list: 10.81.0.1
!!! type5 (prefix-route) is announced to Dc-02 border-gateways because we are doing l3vni extension only.
Path-id 1 (dual) advertised to peers:
10.82.0.3 10.82.0.4
As expected the type-5 prefix-route is announced by the Dc-01 border-gateways towards Dc-02 border-gateways.
Now we will look at the 10.91.102.0/24 route in Dc-02 border-gateways.
Dc-02-Border-Gateway-04# show bgp l2vpn evpn 10.91.102.0
BGP routing table information for VRF default, address family L2VPN EVPN
!!! Dc-01 leaf routes are suppressed.
!!! Dc-01 border-gateways are re-originating the route with their own identity.
!!! RD is now the manually configured dual-rd in Dc-01 border-gateways.
Route Distinguisher: 59905:100000
BGP routing table entry for [5]:[0]:[0]:[24]:[10.91.102.0]/224, version 1060
Paths: (2 available, best #2)
Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
!!! type-5 route received from Dc-01-Border-Gateway-04.
Path type: external, path is valid, not best reason: Router Id, no labeled nexthop
Gateway IP: 0.0.0.0
AS-Path: 4200000001 , path sourced external to AS
!!! Next-hop is changed to shared vtep vip address - loopback 100.
10.83.100.1 (metric 0) from 10.81.0.4 (10.81.0.4)
Origin incomplete, MED 1, localpref 100, weight 0
Received label 100000
!!! Site-ID for Dc-01 is carried in a bgp extended community.
Extcommunity: RT:23456:100000 ENCAP:8 Site-ID:0:4200000001
Router MAC:0200.0a53.6401
Advertised path-id 1
!!! type-5 route received from Dc-01-Border-Gateway-03.
Path type: external, path is valid, is best path, no labeled nexthop
Imported to 1 destination(s)
Imported paths list: Vrf_Blue
Gateway IP: 0.0.0.0
AS-Path: 4200000001 , path sourced external to AS
10.83.100.1 (metric 0) from 10.81.0.3 (10.81.0.3)
Origin incomplete, MED 1, localpref 100, weight 0
Received label 100000
Extcommunity: RT:23456:100000 ENCAP:8 Site-ID:0:4200000001
Router MAC:0200.0a53.6401
Path-id 1 not advertised to any peer
!!! type-5 route installed in the local rib.
Route Distinguisher: 10.82.0.4:6 (L3VNI 100000)
BGP routing table entry for [5]:[0]:[0]:[24]:[10.91.102.0]/224, version 1064
Paths: (1 available, best #1)
Flags: (0x000002) (high32 0x000400) on xmit-list, is not in l2rib/evpn
Advertised path-id 1
Path type: local, path is valid, is best path, reoriginated, no labeled nexthop
Gateway IP: 0.0.0.0
AS-Path: 4200000001 , path sourced external to AS
10.82.1.4 (metric 0) from 0.0.0.0 (10.82.0.4)
Origin incomplete, MED 1, localpref 100, weight 0
Received label 100000
Extcommunity: RT:23456:100000 ENCAP:8 Site-ID:0:4200000001
Router MAC:5000.2e00.1b08
Path-id 1 (dual) advertised to peers:
!!! type-5 routes are announced to site-local spines for reflection.
10.82.0.1 10.82.0.2
!!! Route installed in the vrf's routing table because of l3vni extension.
Dc-02-Border-Gateway-04# show ip route 10.91.102.0 vrf Vrf_Blue
IP Route Table for VRF "Vrf_Blue"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
10.91.102.0/24, ubest/mbest: 1/0
*via 10.83.100.1%default, [20/1], 02:22:03, bgp-4200000002, external, tag 4200000001, segid: 100000 tunnelid: 0xa536
401 encap: VXLAN
From above we can see that Dc-01 border-gateways are re-originating the type-5 routes using their own identity - RD will be the dual-rd configured in border-gateways and next-hop will be the shared vip address configured in loopback 100.
Let's see how the Dc-02 border-gateways announced the same route towards site-local vteps (through spines to the leafs).
!!! Dc-02 border-gateways will change the RD to their shared dual-rd value and next-hop to their shared vip (loopback 100) and announce the route to site-local vteps.
Dc-02-Leaf-07# show bgp l2vpn evpn 10.91.102.0
BGP routing table information for VRF default, address family L2VPN EVPN
!!! RD is chnaged.
Route Distinguisher: 59906:100000
BGP routing table entry for [5]:[0]:[0]:[24]:[10.91.102.0]/224, version 440
Paths: (2 available, best #1)
Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop
Imported to 2 destination(s)
Imported paths list: Vrf_Blue L3-100000
Gateway IP: 0.0.0.0
AS-Path: 4200000001 , path sourced external to AS
!!! Next-hop is chnaged.
10.83.100.2 (metric 81) from 10.82.0.1 (10.82.0.1)
Origin incomplete, MED 1, localpref 100, weight 0
Received label 100000
Extcommunity: RT:23456:100000 ENCAP:8 Site-ID:0:4200000001
Router MAC:0200.0a53.6402
Originator: 10.82.0.3 Cluster list: 10.82.0.1
Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
Gateway IP: 0.0.0.0
AS-Path: 4200000001 , path sourced external to AS
10.83.100.2 (metric 81) from 10.82.0.2 (10.82.0.2)
Origin incomplete, MED 1, localpref 100, weight 0
Received label 100000
Extcommunity: RT:23456:100000 ENCAP:8 Site-ID:0:4200000001
Router MAC:0200.0a53.6402
Originator: 10.82.0.3 Cluster list: 10.82.0.2
Path-id 1 not advertised to any peer
Route Distinguisher: 10.82.0.7:4 (L3VNI 100000)
BGP routing table entry for [5]:[0]:[0]:[24]:[10.91.102.0]/224, version 442
Paths: (1 available, best #1)
Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop
Imported from 59906:100000:[5]:[0]:[0]:[24]:[10.91.102.0]/224
Gateway IP: 0.0.0.0
AS-Path: 4200000001 , path sourced external to AS
10.83.100.2 (metric 81) from 10.82.0.1 (10.82.0.1)
Origin incomplete, MED 1, localpref 100, weight 0
Received label 100000
Extcommunity: RT:23456:100000 ENCAP:8 Site-ID:0:4200000001
Router MAC:0200.0a53.6402
Originator: 10.82.0.3 Cluster list: 10.82.0.1
Path-id 1 not advertised to any peer
!!! Route is installed in the vrf's routing table.
Dc-02-Leaf-07# show ip route 10.91.102.0 vrf Vrf_Blue
IP Route Table for VRF "Vrf_Blue"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
10.91.102.0/24, ubest/mbest: 1/0
*via 10.83.100.2%default, [200/1], 02:38:56, bgp-4200000002, internal, tag 4200000001, segid: 100000 tunnelid: 0xa53
6402 encap: VXLAN
We can see from above; the Dc-02 border-gateway re-originates the route towards site-local vteps - it changes the RD with shared dual-rd and next-hop to shared vip address (loopback 100).
VRF/L3VNI extension data-plane verification and troubleshooting
We have verified in control-plane all the necessary routes are there and propagated properly toward intra and inter site.
Now we will ping from Vlan-102-Pc-02 (10.91.102.12) to Vlan-103-Pc-01 (10.91.103.11).
Vlan-102-Pc-02:~# ping -c 5 10.91.103.11
PING 10.91.103.11 (10.91.103.11) 56(84) bytes of data.
^C
--- 10.91.103.11 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4120ms
Our ping in unsuccessful - bad news !!! What just happened !!!
We have verified the control-plane which is correct. Then there must be something wrong in the data-plane. Let's find the data-plane problem.
Source of the ping is in Dc-01 and destination is in Dc-02. We will do a packet capture on Dc-02-Border-Gateway-04 to verify if the ping is received on the destination Dc-02 fabric.
02 - VXLAN packet capture in Dc-02-Border-Gateway-04
Above capture shows us anycast border-gateways use source address as pip (Dc-01-Border-Gateway-03 - loopback 1) and destination address as Dc-02 border-gateways shared vip (loopback 100) in vxlan outer-header. Cisco has implemented a proprietary vxlan data-plane protection called rouge-vtep (which is on by default and no way to disable it). Any vxlan packet's source IP address, must appear as nve-peer in the data-plane; otherwise the received packet will be silently dropped by the switch because of rouge-vtep. This is actually what happened to our ping packet; Dc-02 border-gateways are silently dropping the packets because Dc-01 border-gateways - loopback1 are not in their data-plane's nve peer list.
Dc-02-Border-Gateway-04# show nve peers control-plane-vni vni 100000
Peer VNI Learn-Source Learn-Source-Mask Gateway-MAC Peer-type Egress-VNI SW-BD IRB State
Underlay-VRF TE-Flags Group-policy
-------------------- -------- ------------ ----------------- -------------- ---------- ---------- ----- ------ -------
-------------------- ------------ ------------ ------------
!!! Site-internal leafs are our peer with type FAB.
10.82.1.6 100000 BGP BGP_RNH 5000.3300.1b08 FAB 100000 100 SYM peer-vn
i-add-complete 1 0 0
10.82.1.7 100000 BGP BGP_RNH 5000.3100.1b08 FAB 100000 100 SYM peer-vn
i-add-complete 1 0 0
!!! Site-external border-gateways are our peer with type DCI.
!!! Only peering to vip address of site-external border-gateways.
!!! No peering to pip address of site-external border-gateways; hence packets are dropped.
10.83.100.1 100000 BGP BGP_RNH 0200.0a53.6401 DCI 100000 100 SYM peer-vn
i-add-complete 1 0 0
How these peers are programmed in the data-plane? It is programmed by the control-plane's bgp routing updates. As we are doing l3vni extension, we have received type-5 routes from Dc-01 border-gateways where next-hop is their vip address. That's why in the data-plane we have vip as nve peer. We need to send some additional bgp l2vpn-evpn updates from both Dc-01 and Dc-02 border-gateways where the next-hop is respective bgws loopback 1, then they will make pip as their nve peers in the data plane also. There are several ways to that - we will use a Cisco proprietary configuration to solve their proprietary rouge-vtep problem.
The solution to the rouge-vtep problem is to configure in all the border-gateways an additional command called "advertise-pip". Do not confuse it with advertise-pip and advertise virtual-rmac - we use in vPC leaf switches. Commands are the same; but they does completely different things according to switch role - vPC Leaf and Anycast Border-Gateway.
Let's configure the command in all the bgws -
Dc-01-Border-Gateway-03 & Dc-01-Border-Gateway-04 configuration
router bgp 4200000001
address-family l2vpn evpn
!!! One command only. advertise virtual-rmac is not valid/required here.
advertise-pip
Dc-02-Border-Gateway-03 & Dc-02-Border-Gateway-04 configuration
router bgp 4200000002
address-family l2vpn evpn
!!! One command only. advertise virtual-rmac is not valid/required here.
advertise-pip
What actually advertise-pip does here? Every border-gateway create a type-2 (mac) route for their local rmac and announces that route to each other with l3vni and next-hop as loopback 1.
!!! Dc-01-Border-Gateway-04's local rmac.
Dc-01-Border-Gateway-04# show nve interface
Interface: nve1, State: Up, encapsulation: VXLAN
VPC Capability: VPC-VIP-Only [not-notified]
Local Router MAC: 5000.2600.1b08
Host Learning Mode: Control-Plane
Source-Interface: loopback1 (primary: 10.81.1.4, secondary: 0.0.0.0)
!!! Dc-01-Border-Gateway-04 local rmac as type-2 (mac) announced to Dc-02-Border-Gateway-04.
Dc-02-Border-Gateway-04# show bgp l2vpn evpn 5000.2600.1b08
BGP routing table information for VRF default, address family L2VPN EVPN
!!! This route do not use dual-rd.
Route Distinguisher: 10.81.0.4:6
BGP routing table entry for [2]:[0]:[0]:[48]:[5000.2600.1b08]:[0]:[0.0.0.0]/216, version 91
Paths: (1 available, best #1)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
Advertised path-id 1
Path type: external, path is valid, is best path, no labeled nexthop
Imported to 1 destination(s)
Imported paths list: L3-100000
AS-Path: 4200000001 , path sourced external to AS
!!! Next-hop is loopback 1 (pip) of Dc-01-Border-Gateway-04.
10.81.1.4 (metric 0) from 10.81.0.4 (10.81.0.4)
Origin IGP, MED not set, localpref 100, weight 0
!!! l3vni 100000 is Vrf_Blue.
Received label 100000
Extcommunity: RT:23456:100000 ENCAP:8 Site-ID:0:4200000001
Router MAC:5000.2600.1b08
Path-id 1 not advertised to any peer
Route Distinguisher: 10.82.0.4:5 (L3VNI 100000)
BGP routing table entry for [2]:[0]:[0]:[48]:[5000.2600.1b08]:[0]:[0.0.0.0]/216, version 92
Paths: (1 available, best #1)
Flags: (0x000202) (high32 0x000400) on xmit-list, is not in l2rib/evpn, is not in HW
Advertised path-id 1
Path type: external, path is valid, is best path, no labeled nexthop
Imported from 10.81.0.4:6:[2]:[0]:[0]:[48]:[5000.2600.1b08]:[0]:[0.0.0.0]/216
AS-Path: 4200000001 , path sourced external to AS
10.81.1.4 (metric 0) from 10.81.0.4 (10.81.0.4)
Origin IGP, MED not set, localpref 100, weight 0
Received label 100000
Extcommunity: RT:23456:100000 ENCAP:8 Site-ID:0:4200000001
Router MAC:5000.2600.1b08
Path-id 1 (dual) not advertised to any peer
Above is actually an interesting way to send a bogus-route to the control-plane bgp and force the data-plane to make pip address of border-gateways as nve-peers. Why bogus-route - because in ip-vrf (l3vni) we always deal with ip address; not mac address.
Dc-02-Border-Gateway-04# show nve peers control-plane-vni vni 100000
Peer VNI Learn-Source Learn-Source-Mask Gateway-MAC Peer-type Egress-VNI SW-BD IRB State
Underlay-VRF TE-Flags Group-policy
-------------------- -------- ------------ ----------------- -------------- ---------- ---------- ----- ------ -------
-------------------- ------------ ------------ ------------
!!! Nve peer - Dc-01-Border-Gateway-03 pip.
10.81.1.3 100000 BGP BGP_RNH 5000.2500.1b08 DCI 100000 100 SYM peer-vn
i-add-complete 1 0 0
!!! Nve peer - Dc-01-Border-Gateway-04 pip.
10.81.1.4 100000 BGP BGP_RNH 5000.2600.1b08 DCI 100000 100 SYM peer-vn
i-add-complete 1 0 0
!!! Nve peer - Dc-02-Border-Gateway-03 pip.
10.82.1.3 100000 BGP BGP_RNH 5000.2d00.1b08 FAB 100000 100 SYM peer-vn
i-add-complete 1 0 0
10.82.1.6 100000 BGP BGP_RNH 5000.3300.1b08 FAB 100000 100 SYM peer-vn
i-add-complete 1 0 0
10.82.1.7 100000 BGP BGP_RNH 5000.3100.1b08 FAB 100000 100 SYM peer-vn
i-add-complete 1 0 0
!!! Nve peer - Dc-01 Border-Gateways vip.
10.83.100.1 100000 BGP BGP_RNH 0200.0a53.6401 DCI 100000 100 SYM peer-vn
i-add-complete 1 0 0
Now our ping is working as expected as there is no rouge-vtep problem anymore.
Vlan-102-Pc-02:~# ping -c 5 10.91.103.11
PING 10.91.103.11 (10.91.103.11) 56(84) bytes of data.
64 bytes from 10.91.103.11: icmp_seq=1 ttl=60 time=13.1 ms
64 bytes from 10.91.103.11: icmp_seq=2 ttl=60 time=13.2 ms
64 bytes from 10.91.103.11: icmp_seq=3 ttl=60 time=11.9 ms
64 bytes from 10.91.103.11: icmp_seq=4 ttl=60 time=12.2 ms
64 bytes from 10.91.103.11: icmp_seq=5 ttl=60 time=19.0 ms
--- 10.91.103.11 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4007ms
Routed Packet Walkthrough For Multi-Site VXLAN
Let's do a multi-site routed packet walkthrough between source vtep - Dc-01-Leaf-06 (Vlan-102-Pc-02 - 10.91.102.12) and destination vtep Dc-02-Leaf-07 (Vlan-103-Pc-01 - 10.91.102.12).
03 - Dc-01-Leaf-06 to Dc-01-Border-Gateways VIP - Echo Request
Echo-request from Dc-01-Leaf-06 will use outer src ip - it's loopback 1, dst ip - Dc-01 Border-Gateways vip - loopback 100. It will use l3vni - 100000 as this is a routed-packet. And both inner mac address will be changed - src mac will be Dc-01-Leaf-06's local RMAC (5000.2900.1b08) and dst mac will be virtual-rmac derived from Dc-01's border-gateways vip loopback100 address (0200.0a53.6401). This packet will use will use underlay's ospf ecmp to reach one of the Dc-01's border-gateways.
04 - Dc-01-Border-Gateway-04 to Dc-02-Border-Gateways VIP - Echo Request
The echo request is received by Dc-01-Border-Gateway-04 because of ecmp. It will forward the packet - outer src ip will be Dc-01-Border-Gateway-04's PIP and dst ip will be Dc-02 Border-Gateway's VIP. It will use l3vni and both inner mac addresses will be replaced by rmac/virtual-rmac. Important to remember here is that anycast border-gateways always use their pip - loopback 1 for sourcing a vxlan packet.
05 - Dc-02-Border-Gateway-04 to Dc-02-Leaf-07 - Echo Request
The echo request is received by Dc-02-Border-Gateway-04 because of ecmp. It will forward the packet - outer src ip will be Dc-01-Border-Gateway-04's PIP and dst ip will be Dc-02-Leaf-07 PIP. It will use l3vni and both inner mac addresses will be replaced by rmac. Important to remember here is that anycast border-gateways always use their pip - loopback 1 for sourcing a vxlan packet.
Now we have configured and verified ip-vrf (l3vni) extension between our separate data-center fabrics (Dc-01 and Dc-02). Stay tuned for part - 11, where we will configure mac-vrf (l2vni) extension between our two dc-fabrics.
Comments
Post a Comment