VXLAN Journey - Part 08 - Single Site VXLAN With Spine And Leaf Switches (Dc-02)
In this blog we will build another data center using vxlan which we will call Dc-02. This data center will use settings/configuration exactly same as our previous blog - part -07. We need another Dc, because soon we will integrate/connect both data-centers using vxlan's multisite connectivity for seamless l3vni/l2vni handoff.
Our topology looks like below for Dc-02 -
01 - Network Topology
We have 2 spine switches as ibgp route-reflectors for l2vpn-evpn address family and 2 leaf switches. Bum traffic will use BGP's ingress replication (HER). Our addressing look like below -
BGP ASN - 4200000002
Loopback 0 - 10.82.0.X/32 where X is the switch number.
Loopback 1 - 10.82.1.X/32 where X is the switch number.
L3VNI - Vrf_Blue (vlan 100 with l3vni 100000).
L2VNI - vlan 101 with l2vni 100001, 10.91.101.0/24 in Vrf_Blue.
L2VNI - vlan 103 with l2vni 100003, 10.91.103.0/24 in Vrf_Blue.
Spine Switch Configuration
Dc-02-Spine-01 Configuration
hostname Dc-02-Spine-01
nv overlay evpn
feature ospf
feature bgp
interface Ethernet1/4
description To_Dc-02-Leaf-06_e1/1
no switchport
mtu 9216
medium p2p
no ip redirects
ip unnumbered loopback0
ip router ospf Underlay area 0.0.0.0
no shutdown
interface Ethernet1/5
description To_Dc-02-Leaf-07_e1/1
no switchport
mtu 9216
medium p2p
no ip redirects
ip unnumbered loopback0
ip router ospf Underlay area 0.0.0.0
no shutdown
interface loopback0
description Underlay_Routing_Loopback
ip address 10.82.0.1/32
ip router ospf Underlay area 0.0.0.0
router ospf Underlay
router-id 10.82.0.1
log-adjacency-changes detail
router bgp 4200000002
router-id 10.82.0.1
log-neighbor-changes
neighbor 10.82.0.6
remote-as 4200000002
description Dc-02-Leaf-06
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
neighbor 10.82.0.7
remote-as 4200000002
description Dc-02-Leaf-07
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
hostname Dc-02-Spine-02
nv overlay evpn
feature ospf
feature bgp
interface Ethernet1/4
description To_Dc-02-Leaf-06_e1/2
no switchport
mtu 9216
medium p2p
no ip redirects
ip unnumbered loopback0
ip router ospf Underlay area 0.0.0.0
no shutdown
interface Ethernet1/5
description To_Dc-02-Leaf-07_e1/2
no switchport
mtu 9216
medium p2p
no ip redirects
ip unnumbered loopback0
ip router ospf Underlay area 0.0.0.0
no shutdown
interface loopback0
description Underlay_Routing_Loopback
ip address 10.82.0.2/32
ip router ospf Underlay area 0.0.0.0
router ospf Underlay
router-id 10.82.0.2
log-adjacency-changes detail
router bgp 4200000002
router-id 10.82.0.2
log-neighbor-changes
neighbor 10.82.0.6
remote-as 4200000002
description Dc-02-Leaf-06
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
neighbor 10.82.0.7
remote-as 4200000002
description Dc-02-Leaf-07
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
route-reflector-client
Leaf Switch Configuration
Dc-02-Leaf-06 Configuration
hostname Dc-02-Leaf-06
nv overlay evpn
feature ospf
feature bgp
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
fabric forwarding anycast-gateway-mac 2020.0000.00aa
vlan 100
name Vlan_Vrf_Blue
vn-segment 100000
vlan 101
vn-segment 100101
route-map Rmap_Tag-12345 permit 10
match tag 12345
vrf context Vrf_Blue
vni 100000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
interface Vlan100
no shutdown
mtu 9216
vrf member Vrf_Blue
no ip redirects
ip forward
interface Vlan101
no shutdown
mtu 9216
vrf member Vrf_Blue
no ip redirects
ip address 10.91.101.1/24 tag 12345
fabric forwarding mode anycast-gateway
interface nve1
no shutdown
description Vtep_Vxlan
host-reachability protocol bgp
source-interface loopback1
member vni 100000 associate-vrf
member vni 100101
ingress-replication protocol bgp
interface Ethernet1/1
description To_Dc-02-Spine-01_e1/4
no switchport
mtu 9216
medium p2p
no ip redirects
ip unnumbered loopback0
ip router ospf Underlay area 0.0.0.0
no shutdown
interface Ethernet1/2
description To_Dc-02-Spine-02_e1/4
no switchport
mtu 9216
medium p2p
no ip redirects
ip unnumbered loopback0
ip router ospf Underlay area 0.0.0.0
no shutdown
interface Ethernet1/15
description To_Vlan-101-Pc-02_eth0
switchport access vlan 101
spanning-tree port type edge
mtu 9216
interface loopback0
description Underlay_Routing_Loopback
ip address 10.82.0.6/32
ip router ospf Underlay area 0.0.0.0
interface loopback1
description Overlay_Vtep_Loopback
ip address 10.82.1.6/32
ip router ospf Underlay area 0.0.0.0
router ospf Underlay
router-id 10.82.0.6
log-adjacency-changes detail
router bgp 4200000002
router-id 10.82.0.6
log-neighbor-changes
neighbor 10.82.0.1
remote-as 4200000002
description Dc-02-Spine-01_Evpn
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 10.82.0.2
remote-as 4200000002
description Dc-02-Spine-02_Evpn
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
vrf Vrf_Blue
address-family ipv4 unicast
redistribute direct route-map Rmap_Tag-12345
maximum-paths ibgp 2
evpn
vni 100101 l2
rd auto
route-target import auto
route-target export auto
Dc-02-Leaf-07 Configuration
hostname Dc-02-Leaf-07
nv overlay evpn
feature ospf
feature bgp
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay
fabric forwarding anycast-gateway-mac 2020.0000.00aa
vlan 100
name Vlan_Vrf_Blue
vn-segment 100000
vlan 103
vn-segment 100103
route-map Rmap_Tag-12345 permit 10
match tag 12345
vrf context Vrf_Blue
vni 100000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
interface Vlan100
no shutdown
mtu 9216
vrf member Vrf_Blue
no ip redirects
ip forward
no ipv6 redirects
interface Vlan103
no shutdown
mtu 9216
vrf member Vrf_Blue
no ip redirects
ip address 10.91.103.1/24 tag 12345
no ipv6 redirects
fabric forwarding mode anycast-gateway
interface nve1
no shutdown
description Vtep_Vxlan
host-reachability protocol bgp
source-interface loopback1
member vni 100000 associate-vrf
member vni 100103
ingress-replication protocol bgp
interface Ethernet1/1
description To_Dc-02-Spine-01_e1/5
no switchport
mtu 9216
medium p2p
no ip redirects
ip unnumbered loopback0
ip router ospf Underlay area 0.0.0.0
no shutdown
interface Ethernet1/2
description To_Dc-02-Spine-02_e1/5
no switchport
mtu 9216
medium p2p
no ip redirects
ip unnumbered loopback0
ip router ospf Underlay area 0.0.0.0
no shutdown
interface Ethernet1/14
description To_Vlan-103-Pc-01_eth0
switchport access vlan 103
spanning-tree port type edge
mtu 9216
interface loopback0
description Underlay_Routing_Loopback
ip address 10.82.0.7/32
ip router ospf Underlay area 0.0.0.0
interface loopback1
description Overlay_Vtep_Loopback
ip address 10.82.1.7/32
ip router ospf Underlay area 0.0.0.0
router ospf Underlay
router-id 10.82.0.7
log-adjacency-changes detail
router bgp 4200000002
router-id 10.82.0.7
log-neighbor-changes
neighbor 10.82.0.1
remote-as 4200000002
description Dc-02-Spine-01_Evpn
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
neighbor 10.82.0.2
remote-as 4200000002
description Dc-02-Spine-02_Evpn
update-source loopback0
address-family l2vpn evpn
send-community
send-community extended
vrf Vrf_Blue
address-family ipv4 unicast
redistribute direct route-map Rmap_Tag-12345
maximum-paths ibgp 2
evpn
vni 100103 l2
rd auto
route-target import auto
route-target export auto
Verification
Our other single site Dc-02 - the VXLAN configuration is complete with Spine and Leaf switches. We will verify the configuration.
!!! Dc-02-Spine-02 l2vpn-evpn bgp neighborship.
Dc-02-Spine-02# show bgp l2vpn evpn summary
BGP summary information for VRF default, address family L2VPN EVPN
BGP router identifier 10.82.0.2, local AS number 4200000002
BGP table version is 119, L2VPN EVPN config peers 5, capable peers 2
8 network entries and 8 paths using 2912 bytes of memory
BGP attribute entries [8/2944], BGP AS path entries [0/0]
BGP community entries [0/0], BGP clusterlist entries [0/0]
!!! Neighborship is established with all the Leaf switches.
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.82.0.6 4 4200000002 1431 1400 119 0 0 22:49:17 4
10.82.0.7 4 4200000002 1433 1397 119 0 0 22:49:45 4
Neighbor T AS Type-1 Type-2 Type-3 Type-4 Type-5
10.82.0.6 I 4200000002 0 2 1 0 1
10.82.0.7 I 4200000002 0 2 1 0 1
!!! vlan 103 - 10.91.103.0/24 type-5 evpn route verification from Spine switch.
Dc-02-Spine-02# show bgp l2vpn evpn 10.91.103.0
BGP routing table information for VRF default, address family L2VPN EVPN
!!! Type-5 route announced by Dc-02-Leaf-07.
Route Distinguisher: 10.82.0.7:4
BGP routing table entry for [5]:[0]:[0]:[24]:[10.91.103.0]/224, version 5
Paths: (1 available, best #1)
Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop
Gateway IP: 0.0.0.0
AS-Path: NONE, path sourced internal to AS
10.82.1.7 (metric 41) from 10.82.0.7 (10.82.0.7)
Origin incomplete, MED 0, localpref 100, weight 0
Received label 100000
Extcommunity: RT:23456:100000 ENCAP:8 Router MAC:5000.3100.1b08
!!! Type-5 route is advertised to Dc-02-Leaf-06.
Path-id 1 advertised to peers:
10.82.0.6
As Spine switches are just route-reflectors there is not much to verify in those switches.
!!! Dc-02-Leaf-07 has l2vpn-evpn neighborship to Spine switches only.
Dc-02-Leaf-07# show bgp l2vpn evpn summary
BGP summary information for VRF default, address family L2VPN EVPN
BGP router identifier 10.82.0.7, local AS number 4200000002
BGP table version is 87, L2VPN EVPN config peers 2, capable peers 2
8 network entries and 10 paths using 2912 bytes of memory
BGP attribute entries [10/3680], BGP AS path entries [0/0]
BGP community entries [0/0], BGP clusterlist entries [2/8]
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.82.0.1 4 4200000002 1440 1412 87 0 0 23:00:30 2
10.82.0.2 4 4200000002 1440 1410 87 0 0 23:00:32 2
Neighbor T AS Type-1 Type-2 Type-3 Type-4 Type-5
10.82.0.1 I 4200000002 0 1 0 0 1
10.82.0.2 I 4200000002 0 1 0 0 1
Dc-02-Leaf-07# show bgp l2vpn evpn 10.91.101.0
BGP routing table information for VRF default, address family L2VPN EVPN
!!! type-5 route originated by Dc-02-Leaf-06.
Route Distinguisher: 10.82.0.6:4
BGP routing table entry for [5]:[0]:[0]:[24]:[10.91.101.0]/224, version 10
Paths: (2 available, best #2)
Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
Path type: internal, path is valid, not best reason: Neighbor Address, no labeled nexthop
Gateway IP: 0.0.0.0
!!! Reflected route from Dc-02-Spine-02.
AS-Path: NONE, path sourced internal to AS
10.82.1.6 (metric 81) from 10.82.0.2 (10.82.0.2)
Origin incomplete, MED 0, localpref 100, weight 0
Received label 100000
Extcommunity: RT:23456:100000 ENCAP:8 Router MAC:5000.3300.1b08
Originator: 10.82.0.6 Cluster list: 10.82.0.2
Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop
Imported to 2 destination(s)
Imported paths list: Vrf_Blue L3-100000
!!! Reflected route from Dc-02-Spine-01.
Gateway IP: 0.0.0.0
AS-Path: NONE, path sourced internal to AS
10.82.1.6 (metric 81) from 10.82.0.1 (10.82.0.1)
Origin incomplete, MED 0, localpref 100, weight 0
Received label 100000
Extcommunity: RT:23456:100000 ENCAP:8 Router MAC:5000.3300.1b08
Originator: 10.82.0.6 Cluster list: 10.82.0.1
Path-id 1 not advertised to any peer
!!! type-5 route installed in local routing table.
Route Distinguisher: 10.82.0.7:4 (L3VNI 100000)
BGP routing table entry for [5]:[0]:[0]:[24]:[10.91.101.0]/224, version 77
Paths: (1 available, best #1)
Flags: (0x000002) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW
Advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop
Imported from 10.82.0.6:4:[5]:[0]:[0]:[24]:[10.91.101.0]/224
Gateway IP: 0.0.0.0
AS-Path: NONE, path sourced internal to AS
10.82.1.6 (metric 81) from 10.82.0.1 (10.82.0.1)
Origin incomplete, MED 0, localpref 100, weight 0
Received label 100000
Extcommunity: RT:23456:100000 ENCAP:8 Router MAC:5000.3300.1b08
Originator: 10.82.0.6 Cluster list: 10.82.0.1
Path-id 1 not advertised to any peer
Now the above type-5 will be leaked into the vrf's ipv4 unicast routing table from the global/default l2vpn-evpn routing table.
Dc-02-Leaf-07# show ip bgp 10.91.101.0 vrf Vrf_Blue
BGP routing table information for VRF Vrf_Blue, address family IPv4 Unicast
BGP routing table entry for 10.91.101.0/24, version 16
Paths: (1 available, best #1)
Flags: (0x808001a) (high32 00000000) on xmit-list, is in urib, is best urib route, is in HW
vpn: version 11, (0x00000000100002) on xmit-list
Multipath: iBGP
Advertised path-id 1, VPN AF advertised path-id 1
Path type: internal, path is valid, is best path, no labeled nexthop, in rib
Imported from 10.82.0.6:4:[5]:[0]:[0]:[24]:[10.91.101.0]/224
AS-Path: NONE, path sourced internal to AS
10.82.1.6 (metric 81) from 10.82.0.1 (10.82.0.1)
Origin incomplete, MED 0, localpref 100, weight 0
Received label 100000
Extcommunity: RT:23456:100000 ENCAP:8 Router MAC:5000.3300.1b08
Originator: 10.82.0.6 Cluster list: 10.82.0.1
VRF advertise information:
Path-id 1 not advertised to any peer
VPN AF advertise information:
Path-id 1 not advertised to any peer
Dc-02-Leaf-07# show ip route 10.91.101.0 vrf Vrf_Blue
IP Route Table for VRF "Vrf_Blue"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
10.91.101.0/24, ubest/mbest: 1/0
*via 10.82.1.6%default, [200/0], 22:50:26, bgp-4200000002, internal, tag 4200000002, segid: 100000 tunnelid: 0xa520106 encap: VXLAN
Dc-02-Leaf-07# show nve vni
Codes: CP - Control Plane DP - Data Plane
UC - Unconfigured SA - Suppress ARP
S-ND - Suppress ND
SU - Suppress Unknown Unicast
Xconn - Crossconnect
SAS-VLAN - vni associated to service vlan
MS-IR - Multisite Ingress Replication
HYB - Hybrid IRB mode
Interface VNI Multicast-group State Mode Type [BD/VRF] Flags
--------- -------- ----------------- ----- ---- ------------------ -----
nve1 100000 n/a Up CP L3 [Vrf_Blue]
nve1 100103 UnicastBGP Up CP L2 [103]
Dc-02-Leaf-07# show bgp evi 100000
-----------------------------------------------
L3VNI ID : 100000 (L3-100000)
RD : 10.82.0.7:4
Prefixes (local/total) : 1/3
Created : Oct 27 19:48:37.033243
Last Oper Up/Down : Oct 28 18:30:13.345476 / Oct 28 18:30:13.344745
Enabled : Yes
Associated IP-VRF : Vrf_Blue
Address-family IPv4 Unicast
Active Export RT list :
23456:100000
Active Import RT list :
23456:100000
Active EVPN Export RT list :
23456:100000
Active EVPN Import RT list :
23456:100000
Active MVPN Export RT list :
23456:100000
Active MVPN Import RT list :
23456:100000
Dc-02-Leaf-07# show bgp evi 100103
-----------------------------------------------
L2VNI ID : 100103 (L2-100103)
RD : 10.82.0.7:32870
Prefixes (local/total) : 3/3
Created : Oct 27 19:48:40.721686
Last Oper Up/Down : Oct 28 18:30:13.345982 / Oct 28 18:30:13.344526
Enabled : Yes
Associated IP-VRF : Vrf_Blue
Active Export RT list :
23456:100103
Active Import RT list :
23456:100103 EVI-RT EC:23456:100103
!!! Nve peer for l3vni 100000.
Dc-02-Leaf-07# show nve peers control-plane-vni vni 100000
Peer VNI Learn-Source Learn-Source-Mask Gateway-MAC Peer-type Egress-VNI SW-BD IRB State
Underlay-VRF TE-Flags Group-policy
-------------------- -------- ------------ ----------------- -------------- ---------- ---------- ----- ------ -------
-------------------- ------------ ------------ ------------
!!! Dc-02-Leaf-06 is nve peer for l3vni.
10.82.1.6 100000 BGP BGP_RNH 5000.3300.1b08 FAB 100000 100 SYM peer-vn
i-add-complete 1 0 0
!!! No nve peer for l2vni 100103 because it is not stretched to other leaf switch (Dc-02-Leaf-06).
Dc-02-Leaf-07# show nve peers control-plane-vni vni 100103
Packet Walkthrough
Routed Packet
As the Dc-02 fabric has only l3vni stretched to the leaf switches, we can do packet walkthrough for routed packets only. Lets ping from Vlan-103-Pc-01 (10.91.103.11 - 50:5a:d1:00:3b:00) to Vlan-101-Pc-02 (10.91.101.12 - 50:1e:d1:00:3a:00).
Vlan-103-Pc-01:~# ping -c 5 10.91.101.12
PING 10.91.101.12 (10.91.101.12) 56(84) bytes of data.
64 bytes from 10.91.101.12: icmp_seq=1 ttl=62 time=6.68 ms
64 bytes from 10.91.101.12: icmp_seq=2 ttl=62 time=6.01 ms
64 bytes from 10.91.101.12: icmp_seq=3 ttl=62 time=5.73 ms
64 bytes from 10.91.101.12: icmp_seq=4 ttl=62 time=5.38 ms
64 bytes from 10.91.101.12: icmp_seq=5 ttl=62 time=5.96 ms
--- 10.91.101.12 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4008ms
Do not forget the rule of VXLAN routed packet forwarding - both the inner mac addresses will be replaced by respective leaf switches RMAC.
Dc-02-Leaf-07# show nve interface
Interface: nve1, State: Up, encapsulation: VXLAN
VPC Capability: VPC-VIP-Only [not-notified]
!!! RMAC used in inner mac address.
Local Router MAC: 5000.3100.1b08
Host Learning Mode: Control-Plane
Source-Interface: loopback1 (primary: 10.82.1.7, secondary: 0.0.0.0)
Dc-02-Leaf-06# show nve interface
Interface: nve1, State: Up, encapsulation: VXLAN
VPC Capability: VPC-VIP-Only [not-notified]
!!! RMAC used in inner mac address.
Local Router MAC: 5000.3300.1b08
Host Learning Mode: Control-Plane
Source-Interface: loopback1 (primary: 10.82.1.6, secondary: 0.0.0.0)
The capture the echo-request and echo-reply packet from Dc-02-Leaf-07 is given below -
02 - ICMP Echo-Request (Routed)
03 - ICMP Echo-Reply (Routed)
Now we have build two distinct data centers. But they do not talk with each other. In part - 09 we will enable VXLAN-Multisite with special leaf nodes called Border-Gateway. And we will achieve seamless extension of our L3VNI and L2VNI across data centers.
Comments
Post a Comment