VXLAN Journey - Part 07 - Single Site VXLAN With Spine And Leaf Switches (Dc-01)
In the previous blogs, we have covered most of the theories/technical details about how VXLAN works using Cisco Nexus Leaf switches. But we had a scalability problem because of ibgp's full mesh requirements. Now we will solve that problem by introducing Spine switches (route-reflectors) in our topology. Let's start building our single site vxlan data center using Spine and Leaf switches.
Our topology looks like below for Dc-01 -
 |
| 01 - Network Topology |
We have 2 spine switches as ibgp route-reflectors for l2vpn-evpn address family and 3 leaf switches (2 of them are a vpc pair). Our addressing look like below -
BGP ASN - 4200000001
Loopback 0 - 10.81.0.X/32 where X is the switch number.
Loopback 1 - 10.81.1.X/32 where X is the switch number.
Loopback 1 secondary - 10.81.1.XY/32 where X,Y is the number of vpc switches.
L3VNI - Vrf_Blue (vlan 100 with l3vni 100000).
L2VNI - vlan 101 with l2vni 100001, 10.91.101.0/24 in Vrf_Blue.
L2VNI - vlan 102 with l2vni 100002, 10.91.102.0/24 in Vrf_Blue.
Also we will make a design decision for how BUM traffic will be forwarded in our fabric. We will not configure any multicast protocols (PIM) in Spine or Leaf switches. The leaf switches will use route-type 3 (imet) for vxlan encapsulated BUM traffic. Personally I do not support using multicast protocol - PIM; for BUM traffic forwarding and introduce another layer of complexity in the fabric. As BGP's ingress replication (HER) already supports that - we will be using that.
Spine Switch Configuration
Dc-01-Spine-01 Configuration
hostname Dc-01-Spine-01
!!! No multicast routing configuration will be done in Spine switches for BUM traffic forwarding.
!!! We will use HER in leaf switches for BUM traffic forwarding.
!!! Enable ospf, bgp for unicast and l2vpn-evpn.
nv overlay evpn
feature ospf
feature bgp
!!! Physical interface towards the leaf switches.
interface Ethernet1/4
description To_Dc-01-Leaf-06_e1/1
no switchport
mtu 9216
medium p2p
no ip redirects
ip unnumbered loopback0
ip router ospf Underlay area 0.0.0.0
no shutdown
interface Ethernet1/5
description To_Dc-01-Leaf-07_e1/1
no switchport
mtu 9216
medium p2p
no ip redirects
ip unnumbered loopback0
ip router ospf Underlay area 0.0.0.0
no shutdown
interface Ethernet1/6
description To_Dc-01-Leaf-08_e1/1
no switchport
mtu 9216
medium p2p
no ip redirects
ip unnumbered loopback0
ip router ospf Underlay area 0.0.0.0
no shutdown
!!! Underlay loopback 0 interface.
interface loopback0
description Underlay_Routing_Loopback
ip address 10.81.0.1/32
ip router ospf Underlay area 0.0.0.0
!!! Enable ospf routing.
router ospf Underlay
router-id 10.81.0.1
log-adjacency-changes detail
router bgp 4200000001
router-id 10.81.0.1
log-neighbor-changes
neighbor 10.81.0.6
remote-as 4200000001
description Dc-01-Leaf-06
update-source loopback0
!!! Spine will be route-reflectors for l2vpn-evpn address family only.
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
neighbor 10.81.0.7
remote-as 4200000001
description Dc-01-Leaf-07
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
neighbor 10.81.0.8
remote-as 4200000001
description Dc-01-Leaf-08
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
Dc-01-Spine-02 Configuration
hostname Dc-01-Spine-02
nv overlay evpn
feature ospf
feature bgp
interface Ethernet1/4
description To_Dc-01-Leaf-06_e1/2
no switchport
mtu 9216
medium p2p
no ip redirects
ip unnumbered loopback0
ip router ospf Underlay area 0.0.0.0
no shutdown
interface Ethernet1/5
description To_Dc-01-Leaf-07_e1/2
no switchport
mtu 9216
medium p2p
no ip redirects
ip unnumbered loopback0
ip router ospf Underlay area 0.0.0.0
no shutdown
interface Ethernet1/6
description To_Dc-01-Leaf-08_e1/2
no switchport
mtu 9216
medium p2p
no ip redirects
ip unnumbered loopback0
ip router ospf Underlay area 0.0.0.0
no shutdown
interface loopback0
description Underlay_Routing_Loopback
ip address 10.81.0.2/32
ip router ospf Underlay area 0.0.0.0
router ospf Underlay
router-id 10.81.0.2
log-adjacency-changes detail
router bgp 4200000001
router-id 10.81.0.2
log-neighbor-changes
neighbor 10.81.0.6
remote-as 4200000001
description Dc-01-Leaf-06
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
neighbor 10.81.0.7
remote-as 4200000001
description Dc-01-Leaf-07
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
neighbor 10.81.0.8
remote-as 4200000001
description Dc-01-Leaf-08
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
Leaf Switch Configuration
Dc-01-Leaf-06 Configuration
hostname Dc-01-Leaf-06
!!! Enable required features for Vxlan - l2vpn-evpn.
nv overlay evpn
feature ospf
feature bgp
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
!!! Anycast Gw Mac.
fabric forwarding anycast-gateway-mac 2020.0000.00aa
!!! Vrf_Blue Vlan.
vlan 100
name Vlan_Vrf_Blue
vn-segment 100000
!!! vlan 102.
vlan 102
vn-segment 100102
route-map Rmap_Tag-12345 permit 10
match tag 12345
!!! Vrf definition with autogenerated RD and RT.
vrf context Vrf_Blue
vni 100000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
!!! Vrf_Blue SVI.
interface Vlan100
no shutdown
mtu 9216
vrf member Vrf_Blue
no ip redirects
ip forward
!!! Vlan 102 svi which is a member of Vrf_Blue.
interface Vlan102
no shutdown
mtu 9216
vrf member Vrf_Blue
no ip redirects
ip address 10.91.102.1/24 tag 12345
fabric forwarding mode anycast-gateway
!!! Vtep/Nve interface.
interface nve1
no shutdown
description Vtep_Vxlan
host-reachability protocol bgp
source-interface loopback1
member vni 100000 associate-vrf
!!! BUM traffic for l2vni will use ingress replication/HER.
!!! Multicast (PIM) is not used for BUM forwarding.
member vni 100102
ingress-replication protocol bgp
interface Ethernet1/1
description To_Dc-01-Spine-01_e1/4
no switchport
mtu 9216
medium p2p
no ip redirects
ip unnumbered loopback0
ip router ospf Underlay area 0.0.0.0
no shutdown
interface Ethernet1/2
description To_Dc-01-Spine-02_e1/4
no switchport
mtu 9216
medium p2p
no ip redirects
ip unnumbered loopback0
ip router ospf Underlay area 0.0.0.0
no shutdown
!!! Connection to Vlan-102-Pc-02.
interface Ethernet1/15
description To_Vlan-102-Pc-02_eth0
switchport access vlan 102
spanning-tree port type edge
mtu 9216
interface loopback0
description Underlay_Routing_Loopback
ip address 10.81.0.6/32
ip router ospf Underlay area 0.0.0.0
interface loopback1
description Overlay_Vtep_Loopback
ip address 10.81.1.6/32
ip router ospf Underlay area 0.0.0.0
router ospf Underlay
router-id 10.81.0.6
log-adjacency-changes detail
router bgp 4200000001
router-id 10.81.0.6
log-neighbor-changes
!!! BGP neighborship only to the Spines in l2vpn-evpn address family.
neighbor 10.81.0.1
remote-as 4200000001
description Dc-01-Spine-01_Evpn
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 10.81.0.2
remote-as 4200000001
description Dc-01-Spine-02_Evpn
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
vrf Vrf_Blue
address-family ipv4 unicast
!!! Redistribute Vrf connected routes using tag 12345.
redistribute direct route-map Rmap_Tag-12345
!!! We will install 2 paths from ibgp for ecmp.
maximum-paths ibgp 2
evpn
!!! l2vni rd and rt is auto generated.
vni 100102 l2
rd auto
route-target import auto
route-target export auto
Dc-01-Leaf-07 Configuration (vpc pair)
hostname Dc-01-Leaf-07
!!! Enable required features for Vxlan - l2vpn-evpn and vpc.
cfs eth distribute
nv overlay evpn
feature ospf
feature bgp
feature interface-vlan
feature vn-segment-vlan-based
feature lacp
feature vpc
feature nv overlay
!!! Anycast Gw Mac.
fabric forwarding anycast-gateway-mac 2020.0000.00aa
!!! Vrf_Blue Vlan.
vlan 100
name Vlan_Vrf_Blue
vn-segment 100000
!!! vlan 102.
vlan 101
vn-segment 100101
!!! vlan 102.
vlan 102
vn-segment 100102
!!! vlan 3600 for carrying vxlan encapsulated traffic over vPC-Peerlink.
vlan 3600
name Vpc_Peer-Link_Svi_Vlan
route-map Rmap_Tag-12345 permit 10
match tag 12345
!!! Vrf_Blue SVI.
vrf context Vrf_Blue
vni 100000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context management
!!! vlan 3600 is defined as infra-vlan, only infra vlan svi can forward vxlan encapsulated traffic.
system nve infra-vlans 3600
!!! vpc configuration.
vpc domain 1
peer-switch
role priority 10
!!! vpc-keepalive is run over mgmt0 interface in management vrf.
peer-keepalive destination 10.10.11.2 source 10.10.11.1
delay restore 150
peer-gateway
layer3 peer-router
auto-recovery reload-delay 360
ip arp synchronize
!!! Vrf_Blue SVI.
interface Vlan100
no shutdown
mtu 9216
vrf member Vrf_Blue
no ip redirects
ip forward
!!! Vlan 101 svi which is a member of Vrf_Blue.
interface Vlan101
no shutdown
mtu 9216
vrf member Vrf_Blue
no ip redirects
ip address 10.91.101.1/24 tag 12345
fabric forwarding mode anycast-gateway
!!! Vlan 102 svi which is a member of Vrf_Blue.
interface Vlan102
no shutdown
mtu 9216
vrf member Vrf_Blue
no ip redirects
ip address 10.91.102.1/24 tag 12345
fabric forwarding mode anycast-gateway
!!! Vlan 3600 svi to forward vxlan encapsulated traffic over vpc-peerlink interface.
interface Vlan3600
description Svi_Vpc_Peer-Link
no shutdown
mtu 9216
no ip redirects
ip address 10.81.2.5/30
ip ospf network point-to-point
ip router ospf Underlay area 0.0.0.0
!!! vpc port-channel interface for Vlan-102-Pc-01.
interface port-channel14
description Port-Channel_Vlan-102-Pc-01
switchport access vlan 102
spanning-tree port type edge
mtu 9216
vpc 14
!!! vpc port-channel interface for Vlan-101-Pc-01.
interface port-channel15
description Port-Channel_Vlan-101-Pc-01
switchport access vlan 101
spanning-tree port type edge
mtu 9216
vpc 15
!!! vpc-peerlink port-channel.
interface port-channel1500
description Vpc_Peer-Link_Po1500
switchport mode trunk
spanning-tree port type network
vpc peer-link
!!! Vtep/Nve interface.
interface nve1
no shutdown
description Vtep_Vxlan
host-reachability protocol bgp
!!! advertising virtual-rmac for vpc switch.
advertise virtual-rmac
source-interface loopback1
member vni 100000 associate-vrf
!!! BUM traffic for l2vni will use ingress replication/HER.
!!! Multicast (PIM) is not used for BUM forwarding.
member vni 100101
ingress-replication protocol bgp
member vni 100102
ingress-replication protocol bgp
interface Ethernet1/1
description To_Dc-01-Spine-01_e1/5
no switchport
mtu 9216
medium p2p
no ip redirects
ip unnumbered loopback0
ip router ospf Underlay area 0.0.0.0
no shutdown
interface Ethernet1/2
description To_Dc-01-Spine-02_e1/5
no switchport
mtu 9216
medium p2p
no ip redirects
ip unnumbered loopback0
ip router ospf Underlay area 0.0.0.0
no shutdown
!!! Physical interface which is member of vpc-peerlink port-channel.
interface Ethernet1/3
description To_Dc-01-Leaf-08_e1/3_Po1500
switchport mode trunk
channel-group 1500 mode active
!!! Physical interface which is member of vpc-peerlink port-channel.
interface Ethernet1/4
description To_Dc-01-Leaf-08_e1/4_Po1500
switchport mode trunk
channel-group 1500 mode active
!!! Physical interface which is member of port-channel for Vlan-102-Pc-01.
interface Ethernet1/14
description To_Vlan-102-Pc-01_eth0
switchport access vlan 102
mtu 9216
channel-group 14 mode active
!!! Physical interface which is member of port-channel for Vlan-101-Pc-01.
interface Ethernet1/15
description To_Vlan-101-Pc-01_eth0
switchport access vlan 101
mtu 9216
channel-group 15 mode active
!!! vpc-keepalive runs over mgmt0 in management vrf.
interface mgmt0
vrf member management
ip address 10.10.11.1/24
interface loopback0
description Underlay_Routing_Loopback
ip address 10.81.0.7/32
ip router ospf Underlay area 0.0.0.0
interface loopback1
description Overlay_Vtep_Loopback
ip address 10.81.1.7/32
ip address 10.81.1.78/32 secondary
ip router ospf Underlay area 0.0.0.0
router ospf Underlay
router-id 10.81.0.7
log-adjacency-changes detail
router bgp 4200000001
router-id 10.81.0.7
log-neighbor-changes
address-family l2vpn evpn
!!! Advertising pip for vpc switch.
advertise-pip
!!! BGP neighborship only to the Spines in l2vpn-evpn address family.
neighbor 10.81.0.1
remote-as 4200000001
description Dc-01-Spine-01_Evpn
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 10.81.0.2
remote-as 4200000001
description Dc-01-Spine-02_Evpn
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
vrf Vrf_Blue
address-family ipv4 unicast
!!! Redistribute Vrf connected routes using tag 12345.
redistribute direct route-map Rmap_Tag-12345
!!! We will install 2 paths from ibgp for ecmp.
maximum-paths ibgp 2
evpn
!!! l2vni rd and rt is auto generated.
vni 100101 l2
rd auto
route-target import auto
route-target export auto
vni 100102 l2
rd auto
route-target import auto
route-target export auto
Dc-01-Leaf-08 Configuration (vpc pair)
hostname Dc-01-Leaf-08
cfs eth distribute
nv overlay evpn
feature ospf
feature bgp
feature interface-vlan
feature vn-segment-vlan-based
feature lacp
feature vpc
feature nv overlay
fabric forwarding anycast-gateway-mac 2020.0000.00aa
vlan 100
name Vlan_Vrf_Blue
vn-segment 100000
vlan 101
vn-segment 100101
vlan 102
vn-segment 100102
vlan 3600
name Vpc_Peer-Link_Svi_Vlan
route-map Rmap_Tag-12345 permit 10
match tag 12345
vrf context Vrf_Blue
vni 100000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context management
system nve infra-vlans 3600
vpc domain 1
peer-switch
role priority 20
peer-keepalive destination 10.10.11.1 source 10.10.11.2
delay restore 150
peer-gateway
layer3 peer-router
auto-recovery reload-delay 360
ip arp synchronize
interface Vlan100
no shutdown
mtu 9216
vrf member Vrf_Blue
no ip redirects
ip forward
interface Vlan101
no shutdown
mtu 9216
vrf member Vrf_Blue
no ip redirects
ip address 10.91.101.1/24 tag 12345
fabric forwarding mode anycast-gateway
interface Vlan102
no shutdown
mtu 9216
vrf member Vrf_Blue
no ip redirects
ip address 10.91.102.1/24 tag 12345
fabric forwarding mode anycast-gateway
interface Vlan3600
description Svi_Vpc_Peer-Link
no shutdown
mtu 9216
no ip redirects
ip address 10.81.2.6/30
ip ospf network point-to-point
ip router ospf Underlay area 0.0.0.0
interface port-channel14
description Port-Channel_Vlan-102-Pc-01
switchport access vlan 102
spanning-tree port type edge
mtu 9216
vpc 14
interface port-channel15
description Port-Channel_Vlan-101-Pc-01
switchport access vlan 101
spanning-tree port type edge
mtu 9216
vpc 15
interface port-channel1500
description Vpc_Peer-Link_Po1500
switchport mode trunk
spanning-tree port type network
vpc peer-link
interface nve1
no shutdown
description Vtep_Vxlan
host-reachability protocol bgp
advertise virtual-rmac
source-interface loopback1
member vni 100000 associate-vrf
member vni 100101
ingress-replication protocol bgp
member vni 100102
ingress-replication protocol bgp
interface Ethernet1/1
description To_Dc-01-Spine-01_e1/6
no switchport
mtu 9216
medium p2p
no ip redirects
ip unnumbered loopback0
ip router ospf Underlay area 0.0.0.0
no shutdown
interface Ethernet1/2
description To_Dc-01-Spine-02_e1/6
no switchport
mtu 9216
medium p2p
no ip redirects
ip unnumbered loopback0
ip router ospf Underlay area 0.0.0.0
no shutdown
interface Ethernet1/3
description To_Dc-01-Leaf-07_e1/3_Po1500
switchport mode trunk
channel-group 1500 mode active
interface Ethernet1/4
description To_Dc-01-Leaf-07_e1/4_Po1500
switchport mode trunk
channel-group 1500 mode active
interface Ethernet1/14
description To_Vlan-102-Pc-01_eth1
switchport access vlan 102
mtu 9216
channel-group 14 mode active
interface Ethernet1/15
description To_Vlan-101-Pc-01_eth1
switchport access vlan 101
mtu 9216
channel-group 15 mode active
interface mgmt0
vrf member management
ip address 10.10.11.2/24
interface loopback0
description Underlay_Routing_Loopback
ip address 10.81.0.8/32
ip router ospf Underlay area 0.0.0.0
interface loopback1
description Overlay_Vtep_Loopback
ip address 10.81.1.8/32
ip address 10.81.1.78/32 secondary
ip router ospf Underlay area 0.0.0.0
router ospf Underlay
router-id 10.81.0.8
log-adjacency-changes detail
router bgp 4200000001
router-id 10.81.0.8
log-neighbor-changes
address-family l2vpn evpn
advertise-pip
neighbor 10.81.0.1
remote-as 4200000001
description Dc-01-Spine-01_Evpn
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 10.81.0.2
remote-as 4200000001
description Dc-01-Spine-02_Evpn
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
vrf Vrf_Blue
address-family ipv4 unicast
redistribute direct route-map Rmap_Tag-12345
maximum-paths ibgp 2
evpn
vni 100101 l2
rd auto
route-target import auto
route-target export auto
vni 100102 l2
rd auto
route-target import auto
route-target export auto
Verification
Now for our single site Dc-01 - the VXLAN configuration is complete with Spine and Leaf switches. We will verify the configuration.
!!! Dc-01-Spine-01 l2vpn-evpn bgp neighborship.
Dc-01-Spine-01# show bgp l2vpn evpn summary
BGP summary information for VRF default, address family L2VPN EVPN
BGP router identifier 10.81.0.1, local AS number 4200000001
BGP table version is 482, L2VPN EVPN config peers 6, capable peers 3
20 network entries and 20 paths using 7280 bytes of memory
BGP attribute entries [12/4416], BGP AS path entries [0/0]
BGP community entries [0/0], BGP clusterlist entries [0/0]
!!! Neighborship is established with all the Leaf switches.
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.81.0.6 4 4200000001 1405 1407 482 0 0 17:06:20 4
10.81.0.7 4 4200000001 1426 1388 482 0 0 17:09:50 8
10.81.0.8 4 4200000001 1430 1389 482 0 0 17:10:18 8
Neighbor T AS Type-1 Type-2 Type-3 Type-4 Type-5
10.81.0.6 I 4200000001 0 2 1 0 1
10.81.0.7 I 4200000001 0 4 2 0 2
10.81.0.8 I 4200000001 0 4 2 0 2
!!! vlan 102 - 10.91.102.0/24 type-5 evpn route verification from Spine switch.
Dc-01-Spine-01# show bgp l2vpn evpn 10.91.102.0
BGP routing table information for VRF default, address family L2VPN EVPN
!!! type-5 route announced by Dc-01-Leaf-06.
Route Distinguisher: 10.81.0.6:4
BGP routing table entry for [5]:[0]:[0]:[24]:[10.91.102.0]/224, version 284
Paths: (1 available, best #1)
Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop
Gateway IP: 0.0.0.0
AS-Path: NONE, path sourced internal to AS
10.81.1.6 (metric 41) from 10.81.0.6 (10.81.0.6)
Origin incomplete, MED 0, localpref 100, weight 0
Received label 100000
Extcommunity: RT:23456:100000 ENCAP:8 Router MAC:5000.2900.1b08
Path-id 1 advertised to peers:
10.81.0.7 10.81.0.8
!!! type-5 route announced by Dc-01-Leaf-07.
Route Distinguisher: 10.81.0.7:4
BGP routing table entry for [5]:[0]:[0]:[24]:[10.91.102.0]/224, version 268
Paths: (1 available, best #1)
Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop
Gateway IP: 0.0.0.0
AS-Path: NONE, path sourced internal to AS
10.81.1.7 (metric 41) from 10.81.0.7 (10.81.0.7)
Origin incomplete, MED 0, localpref 100, weight 0
Received label 100000
Extcommunity: RT:23456:100000 ENCAP:8 Router MAC:5000.2700.1b08
Path-id 1 advertised to peers:
10.81.0.6 10.81.0.8
!!! type-5 route announced by Dc-01-Leaf-08.
Route Distinguisher: 10.81.0.8:4
BGP routing table entry for [5]:[0]:[0]:[24]:[10.91.102.0]/224, version 296
Paths: (1 available, best #1)
Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop
Gateway IP: 0.0.0.0
AS-Path: NONE, path sourced internal to AS
10.81.1.8 (metric 41) from 10.81.0.8 (10.81.0.8)
Origin incomplete, MED 0, localpref 100, weight 0
Received label 100000
Extcommunity: RT:23456:100000 ENCAP:8 Router MAC:5000.2800.1b08
Path-id 1 advertised to peers:
10.81.0.6 10.81.0.7
That's it from Spine switches. As they are just route reflectors; there is not much to verify.
!!! Dc-01-Leaf-06 has l2vpn-evpn neighborship to Spine switches only.
Dc-01-Leaf-06# show bgp l2vpn evpn summary
BGP summary information for VRF default, address family L2VPN EVPN
BGP router identifier 10.81.0.6, local AS number 4200000001
BGP table version is 296, L2VPN EVPN config peers 2, capable peers 2
23 network entries and 42 paths using 8892 bytes of memory
BGP attribute entries [36/13248], BGP AS path entries [0/0]
BGP community entries [0/0], BGP clusterlist entries [4/16]
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.81.0.1 4 4200000001 1191 1072 296 0 0 17:20:56 12
10.81.0.2 4 4200000001 1192 1072 296 0 0 17:20:55 12
Neighbor T AS Type-1 Type-2 Type-3 Type-4 Type-5
10.81.0.1 I 4200000001 0 6 2 0 4
10.81.0.2 I 4200000001 0 6 2 0 4
Dc-01-Leaf-06# show bgp l2vpn evpn 10.91.101.0
BGP routing table information for VRF default, address family L2VPN EVPN
!!! type-5 route originated by Dc-01-Leaf-07.
Route Distinguisher: 10.81.0.7:4
BGP routing table entry for [5]:[0]:[0]:[24]:[10.91.101.0]/224, version 42
Paths: (2 available, best #2)
Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
Gateway IP: 0.0.0.0
AS-Path: NONE, path sourced internal to AS
!!! Reflected route from Dc-01-Spine-02.
10.81.1.7 (metric 81) from 10.81.0.2 (10.81.0.2)
Origin incomplete, MED 0, localpref 100, weight 0
Received label 100000
Extcommunity: RT:23456:100000 ENCAP:8 Router MAC:5000.2700.1b08
Originator: 10.81.0.7 Cluster list: 10.81.0.2
Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop
Imported to 2 destination(s)
Imported paths list: Vrf_Blue L3-100000
Gateway IP: 0.0.0.0
AS-Path: NONE, path sourced internal to AS
!!! Reflected route from Dc-01-Spine-01.
10.81.1.7 (metric 81) from 10.81.0.1 (10.81.0.1)
Origin incomplete, MED 0, localpref 100, weight 0
Received label 100000
Extcommunity: RT:23456:100000 ENCAP:8 Router MAC:5000.2700.1b08
Originator: 10.81.0.7 Cluster list: 10.81.0.1
Path-id 1 not advertised to any peer
!!! type-5 route originated by Dc-01-Leaf-08
Route Distinguisher: 10.81.0.8:4
BGP routing table entry for [5]:[0]:[0]:[24]:[10.91.101.0]/224, version 56
Paths: (2 available, best #1)
Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop
Imported to 2 destination(s)
Imported paths list: Vrf_Blue L3-100000
Gateway IP: 0.0.0.0
AS-Path: NONE, path sourced internal to AS
!!! Reflected route from Dc-01-Spine-01.
10.81.1.8 (metric 81) from 10.81.0.1 (10.81.0.1)
Origin incomplete, MED 0, localpref 100, weight 0
Received label 100000
Extcommunity: RT:23456:100000 ENCAP:8 Router MAC:5000.2800.1b08
Originator: 10.81.0.8 Cluster list: 10.81.0.1
Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
Gateway IP: 0.0.0.0
AS-Path: NONE, path sourced internal to AS
!!! Reflected route from Dc-01-Spine-02.
10.81.1.8 (metric 81) from 10.81.0.2 (10.81.0.2)
Origin incomplete, MED 0, localpref 100, weight 0
Received label 100000
Extcommunity: RT:23456:100000 ENCAP:8 Router MAC:5000.2800.1b08
Originator: 10.81.0.8 Cluster list: 10.81.0.2
Path-id 1 not advertised to any peer
!!! type-5 route installed in local routing table.
Route Distinguisher: 10.81.0.6:4 (L3VNI 100000)
BGP routing table entry for [5]:[0]:[0]:[24]:[10.91.101.0]/224, version 58
Paths: (2 available, best #1)
Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
!!! Two routes for 10.91.101.0/24 installed because of ECMP.
Advertised path-id 1
!!! One route from Dc-01-Leaf-07.
Path type: internal, path is valid, is best path, no labeled nexthop
Imported from 10.81.0.7:4:[5]:[0]:[0]:[24]:[10.91.101.0]/224
Gateway IP: 0.0.0.0
AS-Path: NONE, path sourced internal to AS
10.81.1.7 (metric 81) from 10.81.0.1 (10.81.0.1)
Origin incomplete, MED 0, localpref 100, weight 0
Received label 100000
Extcommunity: RT:23456:100000 ENCAP:8 Router MAC:5000.2700.1b08
Originator: 10.81.0.7 Cluster list: 10.81.0.1
!!! One route from Dc-01-Leaf-08.
Path type: internal, path is valid, not best reason: Router Id, no labeled nexthop
Imported from 10.81.0.8:4:[5]:[0]:[0]:[24]:[10.91.101.0]/224
Gateway IP: 0.0.0.0
AS-Path: NONE, path sourced internal to AS
10.81.1.8 (metric 81) from 10.81.0.1 (10.81.0.1)
Origin incomplete, MED 0, localpref 100, weight 0
Received label 100000
Extcommunity: RT:23456:100000 ENCAP:8 Router MAC:5000.2800.1b08
Originator: 10.81.0.8 Cluster list: 10.81.0.1
Path-id 1 not advertised to any peer
With vxlan routes are learned by l2vpn-evpn address family in global bgp routing table, then the same route is leaked into the vrf's ipv4 unicast routing table.
Dc-01-Leaf-06# show ip bgp vrf Vrf_Blue
BGP routing table information for VRF Vrf_Blue, address family IPv4 Unicast
BGP table version is 17, Local Router ID is 10.91.102.1
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2
Network Next Hop Metric LocPrf Weight Path
*>i10.91.101.0/24 10.81.1.7 0 100 0 ?
*|i 10.81.1.8 0 100 0 ?
*>i10.91.101.11/32 10.81.1.78 100 0 i
* i 10.81.1.78 100 0 i
* i10.91.102.0/24 10.81.1.7 0 100 0 ?
* i 10.81.1.8 0 100 0 ?
*>r 0.0.0.0 0 100 32768 ?
*>i10.91.102.11/32 10.81.1.78 100 0 i
* i 10.81.1.78 100 0 i
Then from vrf's bgp rib, it is imported into the vrf's unicast routing table.
Dc-01-Leaf-06# show ip route 10.91.101.0 vrf Vrf_Blue
IP Route Table for VRF "Vrf_Blue"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
10.91.101.0/24, ubest/mbest: 2/0
!!! Route via Dc-01-Leaf-07.
*via 10.81.1.7%default, [200/0], 18:06:48, bgp-4200000001, internal, tag 4200000001, segid: 100000 tunnelid: 0xa5101
07 encap: VXLAN
!!! Route via Dc-01-Leaf-08.
*via 10.81.1.8%default, [200/0], 18:04:39, bgp-4200000001, internal, tag 4200000001, segid: 100000 tunnelid: 0xa5101
08 encap: VXLAN
!!! l3vni rd and rt auto generation.
!!! Because of using 32-bit ASN (4200000001), in RT ASN in compensated with 23456.
Dc-01-Leaf-06# show bgp evi 100000
-----------------------------------------------
L3VNI ID : 100000 (L3-100000)
RD : 10.81.0.6:4
Prefixes (local/total) : 1/5
Created : Oct 25 21:57:20.636802
Last Oper Up/Down : Oct 25 21:57:31.449193 / Oct 25 21:57:31.449004
Enabled : Yes
Associated IP-VRF : Vrf_Blue
Address-family IPv4 Unicast
Active Export RT list :
23456:100000
Active Import RT list :
23456:100000
Active EVPN Export RT list :
23456:100000
Active EVPN Import RT list :
23456:100000
Active MVPN Export RT list :
23456:100000
Active MVPN Import RT list :
23456:100000
!!! l2vni rd and rt auto generation.
!!! Because of using 32-bit ASN (4200000001), in RT ASN in compensated with 23456.
Dc-01-Leaf-06# show bgp evi 100102
-----------------------------------------------
L2VNI ID : 100102 (L2-100102)
RD : 10.81.0.6:32869
Prefixes (local/total) : 3/6
Created : Oct 25 21:57:24.441246
Last Oper Up/Down : Oct 25 21:57:31.449347 / Oct 25 21:57:31.448900
Enabled : Yes
Associated IP-VRF : Vrf_Blue
Active Export RT list :
23456:100102
Active Import RT list :
23456:100102 EVI-RT EC:23456:100102
Now we will verify the vxlan data-plane in the leaf switch.
!!! For l3vni we will have 3 nve-peers - Dc-01-Leaf-07 Pip, Dc-01-Leaf-08 Pip and Vip of the vpc switches (Dc-01-Leaf-07 and Dc-01-Leaf-08).
Dc-01-Leaf-06# show nve peers control-plane-vni vni 100000
Peer VNI Learn-Source Learn-Source-Mask Gateway-MAC Peer-type Egress-VNI SW-BD IRB State
Underlay-VRF TE-Flags Group-policy
-------------------- -------- ------------ ----------------- -------------- ---------- ---------- ----- ------ -------
-------------------- ------------ ------------ ------------
!!! PIP Dc-01-Leaf-07.
10.81.1.7 100000 BGP BGP_RNH 5000.2700.1b08 FAB 100000 100 SYM peer-vn
i-add-complete 1 0 0
!!! PIP Dc-01-Leaf-08.
10.81.1.8 100000 BGP BGP_RNH 5000.2800.1b08 FAB 100000 100 SYM peer-vn
i-add-complete 1 0 0
!!! Vip of Dc-01-Leaf-07 and Dc-01-Leaf-08.
10.81.1.78 100000 BGP BGP_RNH 0200.0a51.014e FAB 100000 100 SYM peer-vn
i-add-complete 1 0 0
!!! For l2vni we will have 1 nve-peer - Vip of the vpc switches (Dc-01-Leaf-07 and Dc-01-Leaf-08).
Dc-01-Leaf-06# show nve peers control-plane-vni vni 100102
Peer VNI Learn-Source Learn-Source-Mask Gateway-MAC Peer-type Egress-VNI SW-BD IRB State
Underlay-VRF TE-Flags Group-policy
-------------------- -------- ------------ ----------------- -------------- ---------- ---------- ----- ------ -------
-------------------- ------------ ------------ ------------
!!! Vip of Dc-01-Leaf-07 and Dc-01-Leaf-08.
10.81.1.78 100102 BGP BGP_RNH,BGP_IMET 0000.0000.0000 FAB 100102 102 SYM peer-vn
i-add-complete 1 0 0
Packet Walkthrough
Bridged Packet
Now we will ping from Vlan-102-Pc-02 (10.91.102.12 - 50:00:4d:00:7e:00) to Vlan-102-Pc-01 (10.91.102.11 - 50:f8:e8:00:38:00). This is bridged traffic within vlan 102.
Vlan-102-Pc-02:~# ping -c 5 10.91.102.11
PING 10.91.102.11 (10.91.102.11) 56(84) bytes of data.
64 bytes from 10.91.102.11: icmp_seq=1 ttl=64 time=5.91 ms
64 bytes from 10.91.102.11: icmp_seq=2 ttl=64 time=4.53 ms
64 bytes from 10.91.102.11: icmp_seq=3 ttl=64 time=4.68 ms
64 bytes from 10.91.102.11: icmp_seq=4 ttl=64 time=4.34 ms
64 bytes from 10.91.102.11: icmp_seq=5 ttl=64 time=4.28 ms
--- 10.91.102.11 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss
Dc-01-Leaf-06 routing table for Vlan-102-Pc-01 (10.91.102.11) -
Dc-01-Leaf-06# show ip route 10.91.102.11 vrf Vrf_Blue
IP Route Table for VRF "Vrf_Blue"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
10.91.102.11/32, ubest/mbest: 1/0
!!! Next-hop is the Vip of vpc leaf switches.
*via 10.81.1.78%default, [200/0], 18:15:58, bgp-4200000001, internal, tag 4200000001, segid: 100000 tunnelid: 0xa510
14e encap: VXLAN
We have captured echo-request from Dc-01-Leaf-06 which is provided below. Important thing for echo-request is that - for vxlan bridged traffic; outer vxlan source ip will be loopback1 of Dc-01-Leaf-06 and destination ip will be the shared secondary ip address of loopback1 configured in vpc leaf switches (vip of vpc-switch). The packet's VNID will be l2vni - 100102. Also inner mac addresses will not be changed as this is bridged traffic only.
 |
| 02 - ICMP Echo-Request (Bridged) |
We have captured echo-reply in Dc-01-Leaf-06 which is provided below. Important thing for echo-reply is that - for vxlan bridged traffic; outer vxlan source ip will be the shared secondary ip address of loopback1 configured in vpc leaf switches (vip of vpc-switch) and destination ip will be the loopback1 of Dc-01-Leaf-06 . The packet's VNID will be l2vni - 100102. Also inner mac addresses will not be changed as this is bridged traffic only.
 |
| 03 - ICMP Echo-Reply (Bridged) |
Routed Packet
Now we will ping from Vlan-102-Pc-02 (10.91.102.12 - 50:00:4d:00:7e:00) to Vlan-101-Pc-01 (10.91.101.11 - 50:da:22:00:39:00). This is routed traffic between vlan 102 and vlan 101.
Vlan-102-Pc-02:~# ping -c 5 10.91.101.11
PING 10.91.101.11 (10.91.101.11) 56(84) bytes of data.
64 bytes from 10.91.101.11: icmp_seq=1 ttl=62 time=6.18 ms
64 bytes from 10.91.101.11: icmp_seq=2 ttl=62 time=9.77 ms
64 bytes from 10.91.101.11: icmp_seq=3 ttl=62 time=6.05 ms
64 bytes from 10.91.101.11: icmp_seq=4 ttl=62 time=5.35 ms
64 bytes from 10.91.101.11: icmp_seq=5 ttl=62 time=6.00 ms
--- 10.91.101.11 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss
Dc-01-Leaf-06 routing table for Vlan-101-Pc-01 (10.91.101.11) -
!!! vlan 101 is not stretched to Dc-01-Leaf-06.
!!! But it's host route (10.91.101.11/32) will be imported into the vrf's routing table because of type-2 (mac+ip) route.
Dc-01-Leaf-06# show ip route 10.91.101.11 vrf Vrf_Blue
IP Route Table for VRF "Vrf_Blue"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
10.91.101.11/32, ubest/mbest: 1/0
*via 10.81.1.78%default, [200/0], 03:49:22, bgp-4200000001, internal, tag 4200000001, segid: 100000 tunnelid: 0xa510
14e encap: VXLAN
We have captured echo-request in Dc-01-Leaf-06 which is provided below. Important thing for echo-request is that - for vxlan routed traffic; outer vxlan source ip will be loopback1 of Dc-01-Leaf-06 and destination ip will be the shared secondary ip address of loopback1 configured in vpc leaf switches (vip of vpc-switch). The packet's VNID will be l3vni - 100000. Also both inner mac address will be changed with rmac - source will be the rmac for Dc-01-Leaf-06 and destination will be the virtual rmac of the vpc leaf switch.
 |
| 04 - ICMP Echo-Request (Routed) Dc-01-Leaf-06# show nve interface nve 1 Interface: nve1, State: Up, encapsulation: VXLAN VPC Capability: VPC-VIP-Only [not-notified] !!! Inner source mac of the icmp packet. Local Router MAC: 5000.2900.1b08 Host Learning Mode: Control-Plane Source-Interface: loopback1 (primary: 10.81.1.6, secondary: 0.0.0.0) Dc-01-Leaf-06# show nve peers peer-ip 10.81.1.78 Interface Peer-IP State LearnType Uptime Router-Mac --------- -------------------------------------- ----- --------- -------- ----------------- !!! Outer destination mac of the icmp packet. nve1 10.81.1.78 Up CP 22:51:17 0200.0a51.014e We have captured echo-reply in Dc-01-Leaf-06 which is provided below. Important thing for echo-reply is that - for vxlan routed traffic; outer vxlan source ip will be the shared secondary ip address of loopback1 configured in vpc leaf switches (vip of vpc-switch) and destination ip will be the loopback1 of Dc-01-Leaf-06 . The packet's VNID will be l3vni - 100000. Also both inner mac address will be changed with rmac - source will be the virtual rmac of the vpc leaf switch and destination will be the rmac for Dc-01-Leaf-06.  05 - ICMP Echo-Reply (Routed) Most important lesson when vpc leaf switches are involved in vxlan packet forwarding - they use their vip - secondary ip address configured on the loopback1 interface. Bridged vxlan (l2vni) encapsulated packets will not change the inner src/dst mac addresses. But routed vxlan (l3vni) encapsulated packets will change the inner src/dst mac addresses to router macs. |
Comments
Post a Comment