VXLAN Journey - Part 03 - Explanation of Vrf/Multitenancy (L3VNI), RD and RT auto generation (32/64 bit ASN), Anycast Gateway
We have covered basic VXLAN implementation using BGP-EVPN control plane in Part 02. We will continue that journey in this blog by having a detail look at - Vrf/Multitenancy (L3VNI), RD and RT auto generation (32/64 bit ASN) and Anycast Gateway. We will still continue with just two switches connected back-to-back for simplicity. We need to understand the technology first before scaling up the topology.
L3VNI
We already know about L2VNI - which are our vlans. Let's assume our switches are gateway/router for two different vlans -
- VLAN (101) - L2VNI (100101) - Subnet (10.85.101.0/24)
- VLAN (102) - L2VNI (100102) - Subnet (10.85.102.0/24)
How we are going to route traffic between those subnets? Also vlans can belong to different customers - how we are going to isolate traffic between vlans of different customers? The magic answer is our classic concept of VRFs in a router. With different VRFs we will isolate traffic between different customers. In VXLAN each VRF is allocated a VNI - which is called L3VNI. This L3VNI with symmetric IRB provides routing between vlans within the VRF (intra-vrf routing).
As we are introducing the concept of VRF in VXLAN - we can easily understand why BGP's RD and RT (import/export) plays a very crucial role. The concept is exactly similar like MPLS. The process looks like below -
- A VRF will participate in BGP's IPv4 unicast address family.
- The VRF will make it routes unique using RD and attach proper RT-export community.
- These VRF specific IPv4 unicast routes will enter the switch's global BGP's EVPN address family. And will be advertised to it's BGP-EVPN neighbors.
- The receiving BGP-EVPN neighbor will install the routes in it's global BGP's EVPN address family.
- After that it will import the IPv4 unicast routes to it's own VRF's IPv4 unicast address family matching with the RT-import community.
For simplification I have just mentioned about subnets in above. The same process will be repeated for servers/hosts which are connected with our switch - L2VNI/Mac-Vrf.
Also I have made a statement before - in a VXLAN fabric - all the data-center separate switches becomes a single switch. Now we are getting the answer - individual switch will snoop on it's own mac and arp address tables; after that they will share it with all the other switches over BGP-EVPN control plane and they will become one big single distributed switch (every switch know each-others mac and arp address table).
RD and RT generation
RD and RT has same format; but different functions. Let's look at their size and formats first -
RD and RT are 64 bits values. The first 2 bytes (16 bits) are always reserved for type-code and we can only use the rest of the 6 bytes (48 bits). They have 3 standard based format which are -
 |
| 01 - BGP RD and RT Standard Types (ref 1) Looking at above standard formats - we can see a problem if we are using 32bit BGP autonomous system numbers. The vni field is 3 bytes in length, then we cannot use 4 bytes ASNs (total useable length 6 bytes). We cannot compensate on VNI field; instead we will compensate on ASN field if we are using 32 bits ASNs. Cisco NXOS can automatically generate RD and RT values which makes the network administrator's tasks a lot easier. NXOS auto generation follows below rules -
Last but not least - one can always go with manual RD/RT generation. Then we need to make sure we are generating them in a coherent manner and they remains the same for the whole fabric. Anycast Gateway Traditionally we have gateway for a IP subnet in a central-device like - firewall, router or layer-3 switch etc. VXLAN introduces a concept called - anycast gateway. Anycast Gateway means - the directly connected switch of a server is it's gateway. With this approach - we are moving the gateway functionally to any switch where it is required. We can call it by another name - Distributed Gateway. We achieve this by configuring the same virtual mac-address in the required switches and by making the vlan interface distributed in nature. And the rest is taken care by BGP-EVPN. Topology We need put our new found knowledge into practice. We will use the same topology from Part 02. We will generate RD/RT automatically, introduce VRF and Anycast Gateway, implement intra-vlan routing within the VRF.  Leaf-Sw-01 Configuration I will only explain the new features we are introducing in the topology. For basic VXLAN BGP-EVPN configuration, kindly read Part 02. ! !!! Enable required features. nv overlay evpn feature ospf feature bgp feature vn-segment-vlan-based feature nv overlay !!! Allows us to configure interface vlan in the switch. feature interface-vlan !!! Underlay OSPF configuration. router ospf UNDERLAY router-id 10.81.0.1 log-adjacency-changes detail ! interface loopback0 description Underlay-Routing-Loopback ip address 10.81.0.1/32 ip router ospf UNDERLAY area 0.0.0.0 ! interface loopback1 description Overlay-Vtep-Loopback ip address 10.81.1.1/32 ip router ospf UNDERLAY area 0.0.0.0 ! interface Ethernet1/1 description To_Leaf-Sw-02_e1/1 no switchport mtu 9216 medium p2p no ip redirects ip unnumbered loopback0 ip ospf network point-to-point ip router ospf UNDERLAY area 0.0.0.0 no shutdown ! interface Ethernet1/2 description To_Leaf-Sw-02_e1/2 no switchport mtu 9216 medium p2p no ip redirects ip unnumbered loopback0 ip ospf network point-to-point ip router ospf UNDERLAY area 0.0.0.0 no shutdown ! !!! Create the overlay vxlan tunnel interface. interface nve1 no shutdown description Vtep-Vxlan host-reachability protocol bgp source-interface loopback1 ! !!! Enable BGP-EVPN for the overlay. router bgp 64512 router-id 10.81.0.1 log-neighbor-changes neighbor 10.81.0.2 remote-as 64512 description Leaf-Sw-02 update-source loopback0 address-family l2vpn evpn send-community send-community extended ! !!! VRF/L3VNI configuration. !!! We will need three things - create the vrf, create a l3 vlan interface for the vrf and bgp configuration for the vrf. !!! Creating the Vrf, assign L3VNI, generate RD and RT for the VRF. vrf context Vrf-Blue description Vrf-Blue !!! L3VNI assignment to the Vrf. vni 100000 !!! Generate automatically RD. rd auto address-family ipv4 unicast !!! Generate automatically RD import/export for VPNv4 address family. route-target both auto !!! Generate automatically RD import/export for EVPN address family. route-target both auto evpn ! !!! Create vlan and vlan interface for the Vrf/L3VNI. vlan 100 name Vrf-Blue_L3-Vlan !!! Assign L3VNI to the vlan vn-segment 100000 ! !!! Create the L3 interface for the Vrf. interface Vlan100 description Vrf-Blue_L3-Interface no shutdown mtu 9216 vrf member Vrf-Blue !!! No ip address will be assigned to the interface. !!! ip forward means - enable routing over an interface without ip address. ip forward ! !!! Create a route map to redistribute directly connected Vrf interfaces into BGP. route-map Rmap_Tag-12345 permit 10 description Used-To-Redistribute-Connected-Routes-In-Vrf match tag 12345 ! !!! Configure BGP for the Vrf. router bgp 64512 vrf Vrf-Blue address-family ipv4 unicast !!! Deprecated-Command. Now this happens by default. !!! Only required if using very old version of NXOS. advertise l2vpn evpn !!! Redistribute connected route into bgp using the route-map created before. redistribute direct route-map Rmap_Tag-12345 !!! maximum-paths will be needed when we introduce spine switches. !!! 2 Spine switch - for ECMP we will need ibgp multipath. maximum-paths ibgp 2 ! !!! Configure the shared virtual mac-address for anycast gateways for VLAN/L2VNI. fabric forwarding anycast-gateway-mac 2020.0000.00aa !!! Create the vlan/L2VNI vlan 101 name Vrf-Blue_Vlan-101_10.85.101.0/24 vn-segment 100101 ! !!! Generate automatically RD and RT import/export for the L2VNI. evpn vni 100101 l2 rd auto route-target import auto route-target export auto ! !!! Create the vlan/L2VNI anycast gateway interface. interface Vlan101 description Vrf-Blue_Vlan-101_10.85.101.0/24_Anycast-Gw no shutdown mtu 9216 !!! Interface is a member of the VRF. vrf member Vrf-Blue !!! IP address is tagged with 12345, which will be redistributed into BGP with the route-map configured earlier. ip address 10.85.101.1/24 tag 12345 !!! Enable the anycast gateway functionality. fabric forwarding mode anycast-gateway ! !!! Configure the ports where clients are connected for vlan 101. interface Ethernet1/15 description To_Vlan-101-Pc-01 switchport access vlan 101 spanning-tree port type edge mtu 9216 ! !!! Activate the L3VNI and L2VNI over nve interface. interface nve1 !!! L3VNI activation. member vni 100000 associate-vrf member vni 100101 !!! L2VNI activation. ingress-replication protocol bgp ! Leaf-Sw-02 Configuration ! nv overlay evpn feature ospf feature bgp feature vn-segment-vlan-based feature nv overlay feature interface-vlan ! router ospf UNDERLAY router-id 10.81.0.2 log-adjacency-changes detail ! interface loopback0 description Underlay-Routing-Loopback ip address 10.81.0.2/32 ip router ospf UNDERLAY area 0.0.0.0 ! interface loopback1 description Overlay-Vtep-Loopback ip address 10.81.1.2/32 ip router ospf UNDERLAY area 0.0.0.0 ! interface Ethernet1/1 description To_Leaf-Sw-01_e1/1 no switchport mtu 9216 medium p2p no ip redirects ip unnumbered loopback0 ip ospf network point-to-point ip router ospf UNDERLAY area 0.0.0.0 no shutdown ! interface Ethernet1/2 description To_Leaf-Sw-01_e1/2 no switchport mtu 9216 medium p2p no ip redirects ip unnumbered loopback0 ip ospf network point-to-point ip router ospf UNDERLAY area 0.0.0.0 no shutdown ! interface nve1 no shutdown description Vtep-Vxlan host-reachability protocol bgp source-interface loopback1 ! router bgp 64512 router-id 10.81.0.2 log-neighbor-changes neighbor 10.81.0.1 remote-as 64512 description Leaf-Sw-01 update-source loopback0 address-family l2vpn evpn send-community send-community extended ! vrf context Vrf-Blue description Vrf-Blue vni 100000 rd auto address-family ipv4 unicast route-target both auto route-target both auto evpn ! vlan 100 name Vrf-Blue_L3-Vlan vn-segment 100000 ! interface Vlan100 description Vrf-Blue_L3-Interface no shutdown mtu 9216 vrf member Vrf-Blue ip forward ! route-map Rmap_Tag-12345 permit 10 description Used-To-Redistribute-Connected-Routes-In-Vrf match tag 12345 ! router bgp 64512 vrf Vrf-Blue address-family ipv4 unicast advertise l2vpn evpn redistribute direct route-map Rmap_Tag-12345 maximum-paths ibgp 2 ! fabric forwarding anycast-gateway-mac 2020.0000.00aa !!! Leaf-Sw-02 has two - VLANs/L2VNIs to test intra-vrf routing. vlan 101 name Vrf-Blue_Vlan-101_10.85.101.0/24 vn-segment 100101 ! vlan 102 name Vrf-Blue_Vlan-102_10.85.102.0/24 vn-segment 100102 ! evpn vni 100101 l2 rd auto route-target import auto route-target export auto vni 100102 l2 rd auto route-target import auto route-target export auto ! !!! Create the vlan/L2VNI anycast gateway interfaces. interface Vlan101 description Vrf-Blue_Vlan-101_10.85.101.0/24_Anycast-Gw no shutdown mtu 9216 vrf member Vrf-Blue ip address 10.85.101.1/24 tag 12345 fabric forwarding mode anycast-gateway ! interface Vlan102 description Vrf-Blue_Vlan-102_10.85.102.0/24_Anycast-Gw no shutdown mtu 9216 vrf member Vrf-Blue ip address 10.85.102.1/24 tag 12345 fabric forwarding mode anycast-gateway ! !!! Configure the ports where clients are connected for vlan 101. interface Ethernet1/15 description To_Vlan-101-Pc-01 switchport access vlan 101 spanning-tree port type edge mtu 9216 ! !!! Configure the ports where clients are connected for vlan 102. interface Ethernet1/14 description To_Vlan-102-Pc-01 switchport access vlan 102 spanning-tree port type edge mtu 9216 ! !!! Activate the L3VNI and L2VNI over nve interface. interface nve1 member vni 100000 associate-vrf member vni 100101 ingress-replication protocol bgp member vni 100102 ingress-replication protocol bgp ! Verification Let's start verifying our configuration - Leaf-Sw-01# show nve interface nve 1 detail Interface: nve1, State: Up, encapsulation: VXLAN VPC Capability: VPC-VIP-Only [not-notified] !!! Now we have a router-mac. Local Router MAC: 5000.0100.1b08 Host Learning Mode: Control-Plane Source-Interface: loopback1 (primary: 10.81.1.1, secondary: 0.0.0.0) Source Interface State: Up Virtual RMAC Advertisement: No NVE Flags: Interface Handle: 0x49000001 Source Interface hold-down-time: 180 Source Interface hold-up-time: 30 Remaining hold-down time: 0 seconds Virtual Router MAC: N/A Interface state: nve-intf-add-complete Fabric convergence time: 135 seconds Fabric convergence time left: 0 seconds Leaf-Sw-01# show nve peers Interface Peer-IP State LearnType Uptime Router-Mac --------- -------------------------------------- ----- --------- -------- ----------------- nve1 10.81.1.2 Up CP 1d04h 5000.0300.1b08 Leaf-Sw-01# show bgp l2vpn evpn summary BGP summary information for VRF default, address family L2VPN EVPN BGP router identifier 10.81.0.1, local AS number 64512 BGP table version is 254, L2VPN EVPN config peers 1, capable peers 1 16 network entries and 17 paths using 5200 bytes of memory BGP attribute entries [15/5520], BGP AS path entries [0/0] BGP community entries [0/0], BGP clusterlist entries [0/0] Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd !!! We have received 6 evpn-prefixes from Leaf-Sw-02. 10.81.0.2 4 64512 2291 2188 254 0 0 10:33:32 6 Neighbor T AS Type-1 Type-2 Type-3 Type-4 Type-5 !!! We have learned 3 rt-2 + 1 rt-3 + 2 rt-5 = 6 routes from Leaf-Sw-02. 10.81.0.2 I 64512 0 3 1 0 2 Leaf-Sw-01# show vrf VRF-Name VRF-ID State Reason !!! Vrf-Id is 4 for Vrf-Blue which will be used in auto RD for L3VNI. Vrf-Blue 4 Up -- default 1 Up -- management 2 Up -- !!! Verify VNI mapping for VRF/VLAN Leaf-Sw-01# show nve vni Codes: CP - Control Plane DP - Data Plane UC - Unconfigured SA - Suppress ARP S-ND - Suppress ND SU - Suppress Unknown Unicast Xconn - Crossconnect SAS-VLAN - vni associated to service vlan MS-IR - Multisite Ingress Replication HYB - Hybrid IRB mode Interface VNI Multicast-group State Mode Type [BD/VRF] Flags --------- -------- ----------------- ----- ---- ------------------ ----- !!! Vrf-Blue is L3VNI - 100000 nve1 100000 n/a Up CP L3 [Vrf-Blue] !!! Vlan 101 is L2VNI - 100101 and BUM traffic is using ingress replication/HER. nve1 100101 UnicastBGP Up CP L2 [101] !!! Verify RD and RT for VRF/L3VNI. Leaf-Sw-01# show bgp evi 100000 ----------------------------------------------- L3VNI ID : 100000 (L3-100000) !!! RD generated automatically with bgp-router-id:vrf-id. RD : 10.81.0.1:4 Prefixes (local/total) : 1/4 Created : Aug 17 15:06:57.389687 Last Oper Up/Down : Aug 17 15:30:37.467849 / never Enabled : Yes Associated IP-VRF : Vrf-Blue Address-family IPv4 Unicast Active Export RT list : 64512:100000 Active Import RT list : 64512:100000 Active EVPN Export RT list : !!! RT Export generated automatically with bgp-asn:l3vni. 64512:100000 Active EVPN Import RT list : !!! RT Import generated automatically with bgp-asn:l3vni. 64512:100000 Active MVPN Export RT list : 64512:100000 Active MVPN Import RT list : 64512:100000 !!! Verify RD and RT for VLAN/L2VNI. Leaf-Sw-01# show bgp evi 100101 ----------------------------------------------- !!! VLAN 101 = L2VNI 100101 L2VNI ID : 100101 (L2-100101) !!! RD generated automatically bgp-router-id:32767+101. RD : 10.81.0.1:32868 Prefixes (local/total) : 3/6 Created : Aug 17 14:03:00.055011 Last Oper Up/Down : Aug 17 14:03:00.055656 / never Enabled : Yes !!! Vlan 101 belongs to Vrf-Blue. Associated IP-VRF : Vrf-Blue !!! RD import/export generated automatically bgp-asn:l2vni. Active Export RT list : 64512:100101 Active Import RT list : 64512:100101 EVI-RT EC:64512:100101 !!! Verify the EVPN routes received from Leaf-Sw-01. !!! Detailed explanation of these routes will be provided in Part 04 of the blog. Leaf-Sw-01# show bgp l2vpn evpn BGP routing table information for VRF default, address family L2VPN EVPN BGP table version is 254, Local Router ID is 10.81.0.1 Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2 Network Next Hop Metric LocPrf Weight Path !!! 2 route-type 5 (prefix-route) received from Leaf-Sw-02 because of vlan 101 and vlan 102. !!! RD and RT for route-type 5 use L3VNI format. Route Distinguisher: 10.81.0.2:4 *>i[5]:[0]:[0]:[24]:[10.85.101.0]/224 10.81.1.2 0 100 0 ? *>i[5]:[0]:[0]:[24]:[10.85.102.0]/224 10.81.1.2 0 100 0 ? !!! 1 rt-3, 1 rt-2 (mac-route) and 1 rt-2 (mac+ip route) received for vlan 101. !!! RD and RT for route-type 2 and 3 use L2VNI format. Route Distinguisher: 10.81.0.2:32868 *>i[2]:[0]:[0]:[48]:[5077.ae00.0500]:[0]:[0.0.0.0]/216 10.81.1.2 100 0 i *>i[2]:[0]:[0]:[48]:[5077.ae00.0500]:[32]:[10.85.101.12]/272 10.81.1.2 100 0 i *>i[3]:[0]:[32]:[10.81.1.2]/88 10.81.1.2 100 0 i !!! 1 rt-2 (mac+ip route) received for vlan 102. !!! RD and RT for route-type 2 use L2VNI format. Route Distinguisher: 10.81.0.2:32869 *>i[2]:[0]:[0]:[48]:[5017.2e00.2b00]:[32]:[10.85.102.11]/272 10.81.1.2 100 0 i !!! Verify the Vrf-Blue IPv4 unicast routing table. Leaf-Sw-01# show ip route vrf Vrf-Blue IP Route Table for VRF "Vrf-Blue" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] '%<string>' in via output denotes VRF <string> 10.85.101.0/24, ubest/mbest: 1/0, attached *via 10.85.101.1, Vlan101, [0/0], 1d03h, direct, tag 12345 10.85.101.1/32, ubest/mbest: 1/0, attached *via 10.85.101.1, Vlan101, [0/0], 1d03h, local, tag 12345 10.85.101.11/32, ubest/mbest: 1/0, attached *via 10.85.101.11, Vlan101, [190/0], 00:20:45, hmm !!! Vlan-101-Pc-02 host route from Leaf-Sw-02. 10.85.101.12/32, ubest/mbest: 1/0 *via 10.81.1.2%default, [200/0], 00:46:26, bgp-64512, internal, tag 64512, segid: 100000 tunnelid: 0xa510102 encap: VXLAN !!! Vlan 102 prefix route from Leaf-Sw-02. 10.85.102.0/24, ubest/mbest: 1/0 *via 10.81.1.2%default, [200/0], 00:46:26, bgp-64512, internal, tag 64512, segid: 100000 tunnelid: 0xa510102 encap: VXLAN !!! Vlan-102-Pc-01 host route from Leaf-Sw-02. 10.85.102.11/32, ubest/mbest: 1/0 *via 10.81.1.2%default, [200/0], 00:46:26, bgp-64512, internal, tag 64512, segid: 100000 tunnelid: 0xa510102 encap: VXLAN As all the routes are available, we can do some ping test. !!! From Vlan-101-Pc-01 (Leaf-Sw-01) to Vlan-101-Pc-02 (Leaf-Sw-02) Vlan-101-Pc-01:~# ping -c 4 10.85.101.12 PING 10.85.101.12 (10.85.101.12) 56(84) bytes of data. 64 bytes from 10.85.101.12: icmp_seq=1 ttl=64 time=3.98 ms 64 bytes from 10.85.101.12: icmp_seq=2 ttl=64 time=3.41 ms 64 bytes from 10.85.101.12: icmp_seq=3 ttl=64 time=3.01 ms 64 bytes from 10.85.101.12: icmp_seq=4 ttl=64 time=3.05 ms --- 10.85.101.12 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3006ms !!! From Vlan-101-Pc-01 (Leaf-Sw-01) to Vlan-102-Pc-01 (Leaf-Sw-02) Vlan-101-Pc-01:~# ping -c 4 10.85.102.11 PING 10.85.102.11 (10.85.102.11) 56(84) bytes of data. 64 bytes from 10.85.102.11: icmp_seq=1 ttl=62 time=4.27 ms 64 bytes from 10.85.102.11: icmp_seq=2 ttl=62 time=3.40 ms 64 bytes from 10.85.102.11: icmp_seq=3 ttl=62 time=3.75 ms 64 bytes from 10.85.102.11: icmp_seq=4 ttl=62 time=3.34 ms --- 10.85.102.11 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3005ms In this blog intentionally I have not covered several things - route-type 2 (Mac+IP route) , route-type 5 (prefix-route), symmetric IRB, Router-MAC, intra-vrf vxlan packet walkthrough. Stay tuned for Part 04 of the blog for those. Reference |
Comments
Post a Comment