VXLAN Journey - Part 04 - rt-2 (mac+ip), rt-5 (prefix-route), router-mac, symmetric irb and intra-vrf packet walkthrough
In Part 03 - we have covered a lot of things related to single-site VXLAN implementation. But several things we have not explained. Let's explain them now.
Network Topology
Our topology is the same as previous blog -
 |
| 01 - Network Topology |
Route-type 2 (mac+ip)
We already know about rt-2 with which contains mac address of our connected hosts. Now we are seeing another type of rt-2 (mac+ip) which contains both mac and ip address of our connected hosts. This type of rt-2 will be generated if we have VRF/L3VNI only. Vlan 101 is configured on both Leaf-Sw-01 and Leaf-Sw-02. But Vlan 102 is configured on Leaf-Sw-02 only. Leaf-Sw-02 will send both combinations of rt-2 (mac) and rt-2 (mac+ip) to Leaf-Sw-01 for both the vlans.
Leaf-Sw-02# show bgp l2vpn evpn neighbors 10.81.0.1 advertised-routes
Peer 10.81.0.1 routes for address family L2VPN EVPN:
BGP table version is 426, Local Router ID is 10.81.0.2
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2
Network Next Hop Metric LocPrf Weight Path
!!! rt-2 routes for Vlan-101-Pc-02.
Route Distinguisher: 10.81.0.2:32868 (L2VNI 100101)
*>l[2]:[0]:[0]:[48]:[5077.ae00.0500]:[0]:[0.0.0.0]/216
10.81.1.2 100 32768 i
*>l[2]:[0]:[0]:[48]:[5077.ae00.0500]:[32]:[10.85.101.12]/272
10.81.1.2 100 32768 i
!!! rt-2 routes for Vlan-102-Pc-01.
Route Distinguisher: 10.81.0.2:32869 (L2VNI 100102)
*>l[2]:[0]:[0]:[48]:[5017.2e00.2b00]:[0]:[0.0.0.0]/216
10.81.1.2 100 32768 i
*>l[2]:[0]:[0]:[48]:[5017.2e00.2b00]:[32]:[10.85.102.11]/272
10.81.1.2 100 32768 i
Upon receiving those routes - Leaf-Sw-01, will check its VNI-database and find out vlan 102/L2VNI 100102 is not defined locally. It will install both types of rt-2 for vlan 101/L2VNI 100101. But for vlan 102/L2VNI 100102 - it will reject the rt-2 (mac) route; only install the rt-2 (mac+ip) route. The rt-2 (mac+ip) BGP-Update includes both the L2VNI and L3VNI. It also contains a BGP extended community called Router-Mac (RMAC) - more about that later in this blog. Below is a capture of BGP update for rt-2 (mac+ip) -
.png) |
| 02 - BGP Update RT-2 (MAC+IP) !!! rt-2 (mac) is installed in L2FM/MAC address-table. !!! As Vlan-102 is not defined in the Leaf-Sw-01, there is nothing in the mac-address table. Leaf-Sw-01# show mac address-table vlan 102 ERROR: Vlan does not exist !!! rt-2 (mac+ip) is installed in Vrf specific IPv4 unicast routing table. !!! As Vlan-102 is part of Vrf-Blue; host route for Vlan-102-Pc-01 will be in Vrf-Blue's IPv4 unicast routing table. Leaf-Sw-01# show ip route 10.85.102.11 vrf Vrf-Blue IP Route Table for VRF "Vrf-Blue" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] '%<string>' in via output denotes VRF <string> 10.85.102.11/32, ubest/mbest: 1/0 *via 10.81.1.2%default, [200/0], 21:24:52, bgp-64512, internal, tag 64512, segid: 100000 tunnelid: 0xa510102 encap: VXLAN |
With rt-5 (IP Prefix Route) - the switch basically advertises the IP subnets for which it is Anycast-Gateway in a VRF. Another use case is - if we receive routes/prefixes from another router which is not part of the fabric (for-example - vrf-lite handoff), those prefixes learned from external routers will be announced as rt-5 towards the vxlan fabric.
RT-5 routes are available when we have configured Vrf/L3VNI in the switches. In NLRI information it carries subnet and mask in two different fields. Also it carries the RMAC (router mac). Below is a capture of BGP update for rt-5 (prefix route) -
.png) |
| 03 - BGP Update RT-5 (Prefix Route) Router Mac (RMAC) Until now we have learned VXLAN just takes the original ethernet packet and encapsulates into a vxlan packet. It does not change anything in the original ethernet frame. But that is not true any more when Vrf/L3VNI is involved. For example - Ping from Vlan-101-Pc-01 (10.85.101.11) to Vlan-102-Pc-01 (10.85.102.11). The original/inner ethernet frame will be received by Leaf-Sw-01 (Eth1/15). The original packet src-mac-address is 50c0.6a00.0400 (Vlan-101-Pc-01) and dst-mac-address is 2020.0000.00aa (anycast-gw-mac for vlan 101). When Leaf-Sw-01 send the ping-request packet to Leaf-Sw-02 over a VXLAN tunnel; it will change both the src and dst mac-address of the original ethernet frame (inner ethernet frame of a vxlan packet). The src mac-address will be Leaf-Sw-01's rmac and dst mac-address will be Leaf-Sw-02's rmac. This is very important - when VXLAN does routing (inter-vlan routing within a Vrf) - both src and dst mac-address of the original frame (inner ethernet frame) is changed. Local switch already knows it's rmac and it also knows about remote switch's rmac from rt-2 (mac+ip) or rt-5 (prefix-route) BGP-EVPN advertisements. We will talk about this more in future blogs when the topology becomes more complex. Some basic verification commands for rmac - Leaf-Sw-01# show nve interface nve 1 detail Interface: nve1, State: Up, encapsulation: VXLAN VPC Capability: VPC-VIP-Only [not-notified] !!! Local switch Leaf-Sw-01 RMAC. Local Router MAC: 5000.0100.1b08 Host Learning Mode: Control-Plane Source-Interface: loopback1 (primary: 10.81.1.1, secondary: 0.0.0.0) Source Interface State: Up Virtual RMAC Advertisement: No NVE Flags: Interface Handle: 0x49000001 Source Interface hold-down-time: 180 Source Interface hold-up-time: 30 Remaining hold-down time: 0 seconds Virtual Router MAC: N/A Interface state: nve-intf-add-complete Fabric convergence time: 135 seconds Fabric convergence time left: 0 seconds Leaf-Sw-01# show bgp l2vpn evpn 10.85.102.11 BGP routing table information for VRF default, address family L2VPN EVPN Route Distinguisher: 10.81.0.2:32869 BGP routing table entry for [2]:[0]:[0]:[48]:[5017.2e00.2b00]:[32]:[10.85.102.11]/272, version 50 Paths: (1 available, best #1) Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW Advertised path-id 1 Path type: internal, path is valid, is best path, no labeled nexthop Imported to 2 destination(s) Imported paths list: Vrf-Blue L3-100000 AS-Path: NONE, path sourced internal to AS 10.81.1.2 (metric 41) from 10.81.0.2 (10.81.0.2) Origin IGP, MED not set, localpref 100, weight 0 Received label 100102 100000 !!! BGP communities for rt-2 carries RT (both L2VNI and L3VNI) and RMAC. Extcommunity: RT:64512:100000 RT:64512:100102 ENCAP:8 Router MAC:5000.0300.1b08 Path-id 1 not advertised to any peer Leaf-Sw-01# show nve peers peer-ip 10.81.1.2 detail Details of nve Peers: ---------------------------------------- Peer-Ip: 10.81.1.2 NVE Interface : nve1 Peer State : Up Peer Uptime : 08:04:15 !!! Remote switch Leaf-Sw-02's RMAC. Router-Mac : 5000.0300.1b08 Peer First VNI : 100101 Time since Create : 08:04:15 Configured VNIs : 100000,100101 Provision State : peer-add-complete Learnt CP VNIs : 100000,100101 vni assignment mode : SYMMETRIC Peer Location : N/A Group policy capable: no ---------------------------------------- Symmetric IRB VXLAN uses IRB (integrated routing and bridging) for forwarding packets in the fabric. We have asymmetric and symmetric irb. Symmetric IRB is always recommended for efficient packet forwarding. When I mention routing and bridging - this is purely between vnis - L2VNI/L3VNI (not between IP subnets/prefixes). In Asymmetric IRB - the ingress vtep does both routing and bridging. But egress vtep does bridging only. As ingress is doing two things and egress just does one thing - hence the name asymmetric.  04 - Asymmetric IRB In asymmetric irb if vlan 101 wants to communicate with vlan 102, the ingress vtep - Leaf-Sw-01 will do both routing and bridging, but egress vtep - Leaf-Sw-02 does only bridging. This has a very big disadvantage - every vlan/L2VNI must be available in both ingress and egress vtep. Symmetric IRB solves this problem using Vrfs/L3VNIs. Ingress and egress vtep does both routing and bridging.  04 - Symmetric IRB In this mode we do not need to create all the vlans/l2vnis in every vtep. Ingress vtep first does routing from l2vni --> l3vni; then bridging within l2vni. The egress vtep does the same thing (both routing and bridging) for the return traffic; hence the name symmetric. Intra-Vrf packet walkthrough Let's do a intra-vrf (Vrf-Blue) packet walkthrough between vlan 101 and vlan 102. We will send ping packet from Vlan-101-Pc-01 (10.85.101.11) to Vlan-102-Pc-02 (10.85.102.11). Leaf-Sw-01 will receive the packet in the interface eth1/15. There is nothing special about this packet, Vlan-101-Pc-01 knows it is trying to reach something outside of its subnet - so it will set the dst-mac-address to the anycast-gw-mac of Leaf-Sw-01.  06 - Ping Request Capture Leaf-Sw-01 eth1/15 Now Leaf-Sw-01 will encapsulate the ping request packet into VXLAN packet and sent it to Leaf-Sw-02. As we are using symmetric IRB - this packet will use L3VNI - 100000 (Vrf-Blue). Also as this is inter-vlan traffic within the same vrf; router-mac will come into play. Inner ethernet frame's both src and dst mac address will be removed by the Leaf-Sw-01. It will replace src-mac-address with it's own rmac and dst-mac-address to Leaf-Sw-02's rmac. Earlier in the blog we have shown the commands to verify the rmac addresses.  07 - Ping Request Capture Leaf-Sw-02 eth1/1 When Leaf-Sw-02 receives the ping request packet; it will now decapsulate the vxlan packet; create a normal ethernet frame and forward the packet to it's e1/14 interface where Vlan-102-Pc-01 is connected. Again rmac is in play here - the src-mac-address will be Leaf-Sw-02's rmac; it will not be the Leaf-Sw-02's anycast-gw-mac and dst-mac address will be Vlan-102-Pc-01's mac address.  08 - Ping Request Capture Leaf-Sw-02 eth1/14 For the ping reply packet - same packet flow will be repeated in opposite direction; of course again it will use proper rmac in the inner ethernet frame. Stay tuned for Part 05 - where I will introduce ARP Suppression for L2VNI's. This is particularly important when in future we introduce Spine switches in our topology - we will still be using using HER/Ingress-Replication for BUM traffic. ARP Suppression will optimize our BUM traffic for ARP protocol. |
Comments
Post a Comment