VXLAN Journey - Part 05 - ARP Suppression - optimizing ARP traffic in VXLAN

What is ARP suppression? ARP suppression is a feature in VXLAN that reduces broadcast traffic by snooping Address Resolution Protocol (ARP) requests and replying from a local cache instead of flooding/broadcasting to the entire network.

Let's have a look at out topology again - 

01 - Network Topology

Currently we have a stretched vlan 101 between our switches. The underlay is using HER/ingress replication for BUM traffic replication. When Vlan-101-Pc-01 tries to communicate with Vlan-101-Pc-02, it will generate ARP request. Leaf-Switch-01 will encapsulate the traffic in a VXLAN packet and forward the packet to all the VXLAN/NVE peers from which it has received a rt-3 (IMET) route for vni 100101. Let's assume we have stretched the vlan 101 to 10 different leaf switches. Then the ingress vtep (Leaf-Sw-01) will create 9 duplicate packets for the ARP request and forward it over the overlay.

Now we will try to take a step back and optimize this ARP bum traffic duplication. Because of our MPBGP-EVPN, the Leaf-Sw-01 already knows the MAC and IP address of Vlan-101-Pc-02 from the rt-2 (mac+ip) route. We need a mechanism to tell the switch; transfer that information from MPBGP to it's special ARP cache table. Also when such ARP request comes, it will actively snoop the ARP packets from directly connected host and send the ARP reply from the local cache. And this process is called ARP Suppression.

We have some prerequisite for ARP Suppression to work. Firstly we need to enable TCAM craving for the switch if required (consult Cisco documentation and Software version for your switch model). Secondly the switch must have a local SVI for the vlan for which we want to enable ARP suppression. In other words, it needs to one of those vlans for which the switch is anycast-gateway. Lastly - all the leaf switches where we have stretched the vlan we need to enable the ARP Suppression.

Let's go through our full switch configurations again; with ARP suppression enabled (only settings related to ARP suppression will be explained) -

Leaf-Sw-01 Configuration 

hostname Leaf-Sw-01

nv overlay evpn
feature ospf
feature bgp
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay

fabric forwarding anycast-gateway-mac 2020.0000.00aa

vlan 100
  name Vrf-Blue_L3-Vlan
  vn-segment 100000
vlan 101
  name Vrf-Blue_Vlan-101_10.85.101.0/24
  vn-segment 100101

route-map Rmap_Tag-12345 permit 10
  description Used-To-Redistribute-Connected-Routes-In-Vrf
  match tag 12345 
vrf context Vrf-Blue
  description Vrf-Blue
  vni 100000
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn

interface Vlan100
  description Vrf-Blue_L3-Interface
  no shutdown
  mtu 9216
  vrf member Vrf-Blue
  ip forward

!!! We have SVI for Vlan 101.
!!! ARP suppression requirement. 
interface Vlan101
  description Vrf-Blue_Vlan-101_10.85.101.0/24_Anycast-Gw
  no shutdown
  mtu 9216
  vrf member Vrf-Blue
  ip address 10.85.101.1/24 tag 12345
  fabric forwarding mode anycast-gateway

interface nve1
  no shutdown
  description Vtep-Vxlan
  host-reachability protocol bgp
  source-interface loopback1
  member vni 100000 associate-vrf
  member vni 100101
    !!! Enable arp suppression for the l2vni.
    !!! Enable in all the Leaf switches where we have the vlan SVI.
    !!! ARP suppression requirement. 
    suppress-arp
    ingress-replication protocol bgp

interface Ethernet1/1
  description To_Leaf-Sw-02_e1/1
  no switchport
  mtu 9216
  medium p2p
  no ip redirects
  ip unnumbered loopback0
  ip ospf network point-to-point
  ip router ospf UNDERLAY area 0.0.0.0
  no shutdown

interface Ethernet1/2
  description To_Leaf-Sw-02_e1/2
  no switchport
  mtu 9216
  medium p2p
  no ip redirects
  ip unnumbered loopback0
  ip ospf network point-to-point
  ip router ospf UNDERLAY area 0.0.0.0
  no shutdown

interface Ethernet1/15
  description To_Vlan-101-Pc-01
  switchport access vlan 101
  spanning-tree port type edge
  mtu 9216

interface loopback0
  description Underlay-Routing-Loopback
  ip address 10.81.0.1/32
  ip router ospf UNDERLAY area 0.0.0.0

interface loopback1
  description Overlay-Vtep-Loopback
  ip address 10.81.1.1/32
  ip router ospf UNDERLAY area 0.0.0.0

router ospf UNDERLAY
  router-id 10.81.0.1
  log-adjacency-changes detail
  
router bgp 64512
  router-id 10.81.0.1
  log-neighbor-changes
  neighbor 10.81.0.2
    remote-as 64512
    description Leaf-Sw-02
    update-source loopback0
    address-family l2vpn evpn
      send-community
      send-community extended
  vrf Vrf-Blue
    address-family ipv4 unicast
      redistribute direct route-map Rmap_Tag-12345
      maximum-paths ibgp 2
evpn
  vni 100101 l2
    rd auto
    route-target import auto
    route-target export auto

Leaf-Sw-02 Configuration

hostname Leaf-Sw-02

nv overlay evpn
feature ospf
feature bgp
feature interface-vlan
feature vn-segment-vlan-based
feature nv overlay

fabric forwarding anycast-gateway-mac 2020.0000.00aa

vlan 100
  name Vrf-Blue_L3-Vlan
  vn-segment 100000
vlan 101
  name Vrf-Blue_Vlan-101_10.85.101.0/24
  vn-segment 100101
vlan 102
  name Vrf-Blue_Vlan-102_10.85.102.0/24
  vn-segment 100102

route-map Rmap_Tag-12345 permit 10
  description Used-To-Redistribute-Connected-Routes-In-Vrf
  match tag 12345 

vrf context Vrf-Blue
  description Vrf-Blue
  vni 100000
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn

interface Vlan100
  description Vrf-Blue_L3-Interface
  no shutdown
  mtu 9216
  vrf member Vrf-Blue
  ip forward

!!! We have SVI for Vlan 101.
!!! ARP suppression requirement. 
interface Vlan101
  description Vrf-Blue_Vlan-101_10.85.101.0/24_Anycast-Gw
  no shutdown
  mtu 9216
  vrf member Vrf-Blue
  ip address 10.85.101.1/24 tag 12345
  fabric forwarding mode anycast-gateway

interface Vlan102
  description Vrf-Blue_Vlan-102_10.85.102.0/24_Anycast-Gw
  no shutdown
  mtu 9216
  vrf member Vrf-Blue
  ip address 10.85.102.1/24 tag 12345
  fabric forwarding mode anycast-gateway

interface nve1
  no shutdown
  description Vtep-Vxlan
  host-reachability protocol bgp
  source-interface loopback1
  member vni 100000 associate-vrf
  member vni 100101
    !!! Enable arp suppression for the l2vni.
    !!! Enable in all the Leaf switches where we have the vlan SVI.
    !!! ARP suppression requirement. 
    suppress-arp
    ingress-replication protocol bgp
  member vni 100102
    ingress-replication protocol bgp

interface Ethernet1/1
  description To_Leaf-Sw-01_e1/1
  no switchport
  mtu 9216
  medium p2p
  no ip redirects
  ip unnumbered loopback0
  ip ospf network point-to-point
  ip router ospf UNDERLAY area 0.0.0.0
  no shutdown

interface Ethernet1/2
  description To_Leaf-Sw-01_e1/2
  no switchport
  mtu 9216
  medium p2p
  no ip redirects
  ip unnumbered loopback0
  ip ospf network point-to-point
  ip router ospf UNDERLAY area 0.0.0.0
  no shutdown

interface Ethernet1/14
  description To_Vlan-102-Pc-01
  switchport access vlan 102
  spanning-tree port type edge
  mtu 9216

interface Ethernet1/15
  description To_Vlan-101-Pc-02
  switchport access vlan 101
  spanning-tree port type edge
  mtu 9216

interface loopback0
  description Underlay-Routing-Loopback
  ip address 10.81.0.2/32
  ip router ospf UNDERLAY area 0.0.0.0

interface loopback1
  description Overlay-Vtep-Loopback
  ip address 10.81.1.2/32
  ip router ospf UNDERLAY area 0.0.0.0

router ospf UNDERLAY
  router-id 10.82.0.2
  log-adjacency-changes detail

router bgp 64512
  router-id 10.81.0.2
  log-neighbor-changes
  neighbor 10.81.0.1
    remote-as 64512
    description Leaf-Sw-01
    update-source loopback0
    address-family l2vpn evpn
      send-community
      send-community extended
  vrf Vrf-Blue
    address-family ipv4 unicast
      redistribute direct route-map Rmap_Tag-12345
      maximum-paths ibgp 2
evpn
  vni 100101 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 100102 l2
    rd auto
    route-target import auto
    route-target export auto

Verification

Now we will verify if the switch has build a ARP suppression cache from the MPBGP-EVPN routing information.

Leaf-Sw-01# show ip arp suppression topo-info 
ARP L2RIB Topology information
Topo-id  ARP-suppression mode(HMM SDB value)
100     ARP Suppression Disabled (ARP Suppression Disabled)
!!! For vlan 101 arp suppression is enabled.
101     L2/L3 ARP Suppression (L2/L3 ARP Suppression)

Leaf-Sw-01# show ip arp suppression-cache vlan 101

Flags: + - Adjacencies synced via CFSoE
       L - Local Adjacency
       R - Remote Adjacency
       L2 - Learnt over L2 interface
       PS - Added via L2RIB, Peer Sync
       RO - Dervied from L2RIB Peer Sync Entry

Ip Address      Age      Mac Address    Vlan Physical-ifindex         Flags    Remote Vtep Addrs

10.85.101.11    00:13:50 50c0.6a00.0400  101 Ethernet1/15        L
!!! Vlan-101-Pc-02 arp information is available in the cache.
!!! The flag is "R" - means connected remotely with vtep Leaf-Sw-01 (10.81.1.2).
10.85.101.12    00:13:50 5077.ae00.0500  101 (null)                   R        10.81.1.2

But remember these remote entries are available in the arp suppression-cache only. They are not available in the local ARP table for the the Vrf.

Leaf-Sw-01# show ip arp vrf Vrf-Blue 

Flags: * - Adjacencies learnt on non-active FHRP router
       + - Adjacencies synced via CFSoE
       # - Adjacencies Throttled for Glean
       CP - Added via L2RIB, Control plane Adjacencies
       PS - Added via L2RIB, Peer Sync
       RO - Re-Originated Peer Sync Entry
       D - Static Adjacencies attached to down interface

IP ARP Table for context Vrf-Blue
Total number of entries: 1
Address         Age          MAC Address       Interface       Flags
!!! ARP table shows information about directly connected hosts.
!!! Remotely connected hosts ARP info is available in suppression-cache only.
10.85.101.11 00:04:37  50c0.6a00.0400  Vlan101   

Packet walkthrough for ARP Suppression

Now we will generate some arp-request from Vlan-101-Pc-01 towards Vlan-101-Pc-02.

!!! Vlan-101-Pc-01 is asking for Vlan-101-Pc-02's mac-address.
Vlan-101-Pc-01:~# arping -b -c 5 10.85.101.12
ARPING 10.85.101.12 from 10.85.101.11 eth0
Unicast reply from 10.85.101.12 [50:77:AE:00:05:00]  1.832ms
Unicast reply from 10.85.101.12 [50:77:AE:00:05:00]  2.784ms
Unicast reply from 10.85.101.12 [50:77:AE:00:05:00]  1.872ms
Unicast reply from 10.85.101.12 [50:77:AE:00:05:00]  1.904ms
Unicast reply from 10.85.101.12 [50:77:AE:00:05:00]  1.937ms
Sent 5 probes (5 broadcast(s))
Received 5 response(s)

If we capture packet on Leaf-Sw-01's eth1/15 interface we can look at the ARP request packet. It is just a normal ARP request packet where Vlan-101-Pc-01 is asking for Vlan-101-Pc-02's mac address.

02 - ARP request from Vlan-101-Pc-01

But interesting is the ARP reply packet. The reply packet is generated by the switch locally using the ARP suppression cache. There will be no-more VXLAN encapsulated arp bum traffic traversing between the links where the switches are connected with each other (eth1/1 and eth1/2).

03 - ARP reply from Leaf-Sw-01 to Vlan-101-Pc-01

That's for today. Stay tuned for Part - 05, where we will go in depth how to configure a pair of vPC switches for VXLAN.






Comments

Post a Comment

Popular posts from this blog

Fortigate firewall AAA Configuration for management with TACACS+ protocol and Cisco ISE

Image
In this blog I will write about how to implement AAA services in Fortigate firewalls using Cisco ISE as an authentication server with TACACS+ protocol. First good to know our limitations. AAA means - Authentication (supported by ISE TACACS+), Authorization (partially supported by ISE TACACS+; Fortigate implements Admin-Profiles; so what a user can/cannot do is defined locally in the firewall with "Admin Profiles"; ISE just instructs the firewall to allocate one of those profiles), Accounting (not supported; instead configure syslog to get accounting data, Fortigate firewalls generate syslog messages for configuration changes made by a user). Let's jump to our configuration. Our topology is very simple; one Fortigate firewall and Cisco ISE server connected with the same management network which looks like below -  01 - Network Topology We have a management network 192.168.199.0/24. The firewall is at  .230  and Cisco ISE is at  .49  IP address. We are running...
Read more

802.1x wired authentication with Huawei VRP Switch - (Unified Mode)

I found it very difficult to find the actual configuration steps that is required for 802.1x authentication in Huawei's VRP based switches. Huawei have two authentication modes - unified-mode and common-mode . I did not find any straight forward configuration guide on how to configure 802.1X in unified-mode . In today's blog article I will just try to address that. In this blog, I will just cover Huawei VRP based switches's configuration commands in unified-mode authentication for 802.1x. I will not present a network topology and work with that. If someone is interested about a full network setup and example, I recommend reading part I , part II and compare the configuration accordingly. Verify authentication mode First we will, check for current authentication mode the switch is running. If it is running "common-mode", we need to change it to "unified-mode" . Changing authentication mode requires a restart of the switch. [sw] display au...
Read more

Stacking switches Part - VI (Dell OS10 VLT - Virtual Link Trunking)

Image
In this blog I will configure MLAG (multi chassis link aggregation) for Dell OS10 switches. As every other vendor choses a fancy name of their implementation of MLAG; Dell is no exception to this. Dell calls their implementation VLT - Virtual Link Trunking . Let's look at our network topology - Two Dell OS10 switches will run peering between them and will run VLT/MLAG between them. Two Debian 10 linux machine will be connected with LACP bonding with both switches. They will simulate the client connection. Our topology looks like below - 01 - Dell OS10 VLT Topology Let's look at some terminology before configuration - VLT peer – The two switches participating in VLT are peer to each other. In a VLT domain a maximum of two switches are allowed. VLT interconnect (VLTi) – VLTi synchronizes state information between VLT peers. It synchronizes layer 2 and layer 3 control-plane information. These state/control-plane information includes MAC table, ARP table, IPv6 neighbors etc. VLT ...
Read more