VXLAN Journey - Part 04 - rt-2 (mac+ip), rt-5 (prefix-route), router-mac, symmetric irb and intra-vrf packet walkthrough

In Part 03 - we have covered a lot of things related to single-site VXLAN implementation. But several things we have not explained. Let's explain them now.

Network Topology

Our topology is the same as previous blog -

01 - Network Topology

Route-type 2 (mac+ip)

We already know about rt-2 with which contains mac address of our connected hosts. Now we are seeing another type of rt-2 (mac+ip) which contains both mac and ip address of our connected hosts. This type of rt-2 will be generated if we have VRF/L3VNI only. Vlan 101 is configured on both Leaf-Sw-01 and Leaf-Sw-02. But Vlan 102 is configured on Leaf-Sw-02 only. Leaf-Sw-02 will send both combinations of rt-2 (mac) and rt-2 (mac+ip) to Leaf-Sw-01 for both the vlans.

Leaf-Sw-02# show bgp l2vpn evpn neighbors 10.81.0.1 advertised-routes 

Peer 10.81.0.1 routes for address family L2VPN EVPN:
BGP table version is 426, Local Router ID is 10.81.0.2
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2

Network            Next Hop                      Metric                     LocPrf     Weight    Path

!!! rt-2 routes for Vlan-101-Pc-02.
Route Distinguisher: 10.81.0.2:32868    (L2VNI 100101)
*>l[2]:[0]:[0]:[48]:[5077.ae00.0500]:[0]:[0.0.0.0]/216
                      10.81.1.2                                                     100       32768         i
*>l[2]:[0]:[0]:[48]:[5077.ae00.0500]:[32]:[10.85.101.12]/272
                      10.81.1.2                                                     100      32768          i

!!! rt-2 routes for Vlan-102-Pc-01.
Route Distinguisher: 10.81.0.2:32869    (L2VNI 100102)
*>l[2]:[0]:[0]:[48]:[5017.2e00.2b00]:[0]:[0.0.0.0]/216
                      10.81.1.2                                                     100      32768          i
*>l[2]:[0]:[0]:[48]:[5017.2e00.2b00]:[32]:[10.85.102.11]/272
                      10.81.1.2                                                     100      32768          i

Upon receiving those routes - Leaf-Sw-01, will check its VNI-database and find out vlan 102/L2VNI 100102 is not defined locally. It will install both types of rt-2 for vlan 101/L2VNI 100101. But for vlan 102/L2VNI 100102 - it will reject the rt-2 (mac) route; only install the rt-2 (mac+ip) route. The rt-2 (mac+ip) BGP-Update includes both the L2VNI and L3VNI. It also contains a BGP extended community called Router-Mac (RMAC) - more about that later in this blog. Below is a capture of BGP update for rt-2 (mac+ip) -


02 - BGP Update RT-2 (MAC+IP)


Route-type 5 (IP Prefix Route)

With rt-5 (IP Prefix Route) - the switch basically advertises the IP subnets for which it is Anycast-Gateway in a VRF. Another use case is - if we receive routes/prefixes from another router which is not part of the fabric (for-example - vrf-lite handoff), those prefixes learned from external routers will be announced as rt-5 towards the vxlan fabric.

RT-5 routes are available when we have configured Vrf/L3VNI in the switches. In NLRI information it carries subnet and mask in two different fields. Also it carries the RMAC (router mac). Below is a capture of BGP update for rt-5 (prefix route) -


03 - BGP Update RT-5 (Prefix Route)

Router Mac (RMAC)

Until now we have learned VXLAN just takes the original ethernet packet and encapsulates into a vxlan packet. It does not change anything in the original ethernet frame. But that is not true any more when Vrf/L3VNI is involved. For example - Ping from Vlan-101-Pc-01 (10.85.101.11) to Vlan-102-Pc-01 (10.85.102.11). The original/inner ethernet frame will be received by Leaf-Sw-01 (Eth1/15). The original packet src-mac-address is 50c0.6a00.0400 (Vlan-101-Pc-01) and dst-mac-address is 2020.0000.00aa (anycast-gw-mac for vlan 101). When Leaf-Sw-01 send the ping-request packet to Leaf-Sw-02 over a VXLAN tunnel; it will change both the src and dst mac-address of the original ethernet frame (inner ethernet frame of a vxlan packet). The src mac-address will be Leaf-Sw-01's rmac and dst mac-address will be Leaf-Sw-02's rmac. This is very important - when VXLAN does routing (inter-vlan routing within a Vrf) - both src and dst mac-address of the original frame (inner ethernet frame) is changed. Local switch already knows it's rmac and it also knows about remote switch's rmac from rt-2 (mac+ip) or rt-5 (prefix-route) BGP-EVPN advertisements. We will talk about this more in future blogs when the topology becomes more complex. Some basic verification commands for rmac - 

Leaf-Sw-01# show nve interface nve 1 detail 
Interface: nve1, State: Up, encapsulation: VXLAN
 VPC Capability: VPC-VIP-Only [not-notified]
 !!! Local switch Leaf-Sw-01 RMAC.
 Local Router MAC: 5000.0100.1b08
 Host Learning Mode: Control-Plane
 Source-Interface: loopback1 (primary: 10.81.1.1, secondary: 0.0.0.0)
 Source Interface State: Up
 Virtual RMAC Advertisement: No
 NVE Flags: 
 Interface Handle: 0x49000001
 Source Interface hold-down-time: 180
 Source Interface hold-up-time: 30
 Remaining hold-down time: 0 seconds
 Virtual Router MAC: N/A
 Interface state: nve-intf-add-complete
 Fabric convergence time: 135 seconds
 Fabric convergence time left: 0 seconds

Leaf-Sw-01# show bgp l2vpn evpn 10.85.102.11
BGP routing table information for VRF default, address family L2VPN EVPN
Route Distinguisher: 10.81.0.2:32869
BGP routing table entry for [2]:[0]:[0]:[48]:[5017.2e00.2b00]:[32]:[10.85.102.11]/272, version 50
Paths: (1 available, best #1)
Flags: (0x000202) (high32 00000000) on xmit-list, is not in l2rib/evpn, is not in HW

  Advertised path-id 1
  Path type: internal, path is valid, is best path, no labeled nexthop
             Imported to 2 destination(s)
             Imported paths list: Vrf-Blue L3-100000
  AS-Path: NONE, path sourced internal to AS
    10.81.1.2 (metric 41) from 10.81.0.2 (10.81.0.2)
      Origin IGP, MED not set, localpref 100, weight 0
      Received label 100102 100000
      !!! BGP communities for rt-2 carries RT (both L2VNI and L3VNI) and RMAC.
      Extcommunity: RT:64512:100000 RT:64512:100102 ENCAP:8 Router MAC:5000.0300.1b08

  Path-id 1 not advertised to any peer

Leaf-Sw-01# show nve peers peer-ip 10.81.1.2 detail 
Details of nve Peers:
----------------------------------------
Peer-Ip: 10.81.1.2
    NVE Interface       : nve1
    Peer State          : Up
    Peer Uptime         : 08:04:15
    !!! Remote switch Leaf-Sw-02's RMAC.
    Router-Mac          : 5000.0300.1b08
    Peer First VNI      : 100101
    Time since Create   : 08:04:15
    Configured VNIs     : 100000,100101
    Provision State     : peer-add-complete
    Learnt CP VNIs      : 100000,100101
    vni assignment mode : SYMMETRIC
    Peer Location       : N/A
    Group policy capable: no
----------------------------------------

Symmetric IRB

VXLAN uses IRB (integrated routing and bridging) for forwarding packets in the fabric. We have asymmetric and symmetric irb. Symmetric IRB is always recommended for efficient packet forwarding. When I mention routing and bridging - this is purely between vnis - L2VNI/L3VNI (not between IP subnets/prefixes).

In Asymmetric IRB - the ingress vtep does both routing and bridging. But egress vtep does bridging only. As ingress is doing two things and egress just does one thing - hence the name asymmetric.

04 - Asymmetric IRB

In asymmetric irb if vlan 101 wants to communicate with vlan 102, the ingress vtep - Leaf-Sw-01 will do both routing and bridging, but egress vtep - Leaf-Sw-02 does only bridging. This has a very big disadvantage - every vlan/L2VNI must be available in both ingress and egress vtep.

Symmetric IRB solves this problem using Vrfs/L3VNIs. Ingress and egress vtep does both routing and bridging.


04 - Symmetric IRB

In this mode we do not need to create all the vlans/l2vnis in every vtep. Ingress vtep first does routing from l2vni --> l3vni; then bridging within l2vni. The egress vtep does the same thing (both routing and bridging) for the return traffic; hence the name symmetric.

Intra-Vrf packet walkthrough

Let's do a intra-vrf (Vrf-Blue) packet walkthrough between vlan 101 and vlan 102.
We will send ping packet from Vlan-101-Pc-01 (10.85.101.11) to Vlan-102-Pc-02 (10.85.102.11).

Leaf-Sw-01 will receive the packet in the interface eth1/15. There is nothing special about this packet, Vlan-101-Pc-01 knows it is trying to reach something outside of its subnet - so it will set the dst-mac-address to the anycast-gw-mac of Leaf-Sw-01.

06 - Ping Request Capture Leaf-Sw-01 eth1/15

Now Leaf-Sw-01 will encapsulate the ping request packet into VXLAN packet and sent it to Leaf-Sw-02. As we are using symmetric IRB - this packet will use L3VNI - 100000 (Vrf-Blue). Also as this is inter-vlan traffic within the same vrf; router-mac will come into play. Inner ethernet frame's both src and dst mac address will be removed by the Leaf-Sw-01. It will replace src-mac-address with it's own rmac and dst-mac-address to Leaf-Sw-02's rmac. Earlier in the blog we have shown the commands to verify the rmac addresses.


07 - Ping Request Capture Leaf-Sw-02 eth1/1

When Leaf-Sw-02 receives the ping request packet; it will now decapsulate the vxlan packet; create a normal ethernet frame and forward the packet to it's e1/14 interface where Vlan-102-Pc-01 is connected. Again rmac is in play here - the src-mac-address will be Leaf-Sw-02's rmac; it will not be the Leaf-Sw-02's anycast-gw-mac and dst-mac address will be Vlan-102-Pc-01's mac address.


08 - Ping Request Capture Leaf-Sw-02 eth1/14

For the ping reply packet - same packet flow will be repeated in opposite direction; of course again it will use proper rmac in the inner ethernet frame.

Stay tuned for Part 05 - where I will introduce ARP Suppression for L2VNI's. This is particularly important when in future we introduce Spine switches in our topology - we will still be using using HER/Ingress-Replication for BUM traffic. ARP Suppression will optimize our BUM traffic for ARP protocol.






















Comments

Post a Comment

Popular posts from this blog

Fortigate firewall AAA Configuration for management with TACACS+ protocol and Cisco ISE

Image
In this blog I will write about how to implement AAA services in Fortigate firewalls using Cisco ISE as an authentication server with TACACS+ protocol. First good to know our limitations. AAA means - Authentication (supported by ISE TACACS+), Authorization (partially supported by ISE TACACS+; Fortigate implements Admin-Profiles; so what a user can/cannot do is defined locally in the firewall with "Admin Profiles"; ISE just instructs the firewall to allocate one of those profiles), Accounting (not supported; instead configure syslog to get accounting data, Fortigate firewalls generate syslog messages for configuration changes made by a user). Let's jump to our configuration. Our topology is very simple; one Fortigate firewall and Cisco ISE server connected with the same management network which looks like below -  01 - Network Topology We have a management network 192.168.199.0/24. The firewall is at  .230  and Cisco ISE is at  .49  IP address. We are running...
Read more

802.1x wired authentication with Huawei VRP Switch - (Unified Mode)

I found it very difficult to find the actual configuration steps that is required for 802.1x authentication in Huawei's VRP based switches. Huawei have two authentication modes - unified-mode and common-mode . I did not find any straight forward configuration guide on how to configure 802.1X in unified-mode . In today's blog article I will just try to address that. In this blog, I will just cover Huawei VRP based switches's configuration commands in unified-mode authentication for 802.1x. I will not present a network topology and work with that. If someone is interested about a full network setup and example, I recommend reading part I , part II and compare the configuration accordingly. Verify authentication mode First we will, check for current authentication mode the switch is running. If it is running "common-mode", we need to change it to "unified-mode" . Changing authentication mode requires a restart of the switch. [sw] display au...
Read more

Stacking switches Part - VI (Dell OS10 VLT - Virtual Link Trunking)

Image
In this blog I will configure MLAG (multi chassis link aggregation) for Dell OS10 switches. As every other vendor choses a fancy name of their implementation of MLAG; Dell is no exception to this. Dell calls their implementation VLT - Virtual Link Trunking . Let's look at our network topology - Two Dell OS10 switches will run peering between them and will run VLT/MLAG between them. Two Debian 10 linux machine will be connected with LACP bonding with both switches. They will simulate the client connection. Our topology looks like below - 01 - Dell OS10 VLT Topology Let's look at some terminology before configuration - VLT peer – The two switches participating in VLT are peer to each other. In a VLT domain a maximum of two switches are allowed. VLT interconnect (VLTi) – VLTi synchronizes state information between VLT peers. It synchronizes layer 2 and layer 3 control-plane information. These state/control-plane information includes MAC table, ARP table, IPv6 neighbors etc. VLT ...
Read more