VXLAN Journey - Part 01 - What is VXLAN and how it works (Cisco Nexus Switch) - Basic data plane learning
For several years I have worked with Data-Center network based on VXLAN. Anyone suddenly start working with production DC-networks built on top VXLAN - with Spine-Leaf topology and Multi-Site - they will get lost easily. Also lack of good step by step written books from the vendors Cisco, Arista, Juniper etc. make it difficult to understand the technologies/protocols used for VXLAN in DC-network. Let's try to solve the problem - by building a DC-network from scratch and step by step using Cisco Nexus switches.
What is VXLAN? The most common answer is - tunnelling a layer-2 packets over layer-3 network. I define it another way - the whole DC is a fabric. A fabric consists of several switches. With VXLAN, the whole DC-Fabric becomes a single switch (of course - logically). We are eliminating three fundamental problems in DC -
- The network core becomes Spanning Tree Protocol (STP) free.
- All the physical links in the core are used Equal-cost multi-path routing (ECMP).
- The network becomes multi-tenancy/customer aware (VXLAN - 24 bit VNI).
What is actually - underlay and overlay? Overlay is the network where we will create the vxlan tunnels to encapsulate our layer-2 frame in a layer-3 packet. Then underlay is the network which will transport/forward the vxlan packet created by overlay.
Below is a basic diagram of how a vxlan packet looks like (reference 1)-
01 - VXLAN Packet Header
VXLAN will encapsulate the original layer-2 frame in a layer-3 packet. This layer-3 packet uses UDP (destination port 4789) and has a VXLAN header. The most important field in the VXLAN header is the 24-bit VNI. Above packet will be generated by overlay and forwarded through the network using the VNI. Our VLAN number becomes locally significant (within a single switch) and VNI number becomes globally significant (within the whole fabric/all switches). Encapsulation and decapsulation of VXLAN packet is done in the overlay. The network switch which can do vxlan encapsulation and decapsulation is called VXLAN Tunnel Endpoint (VTEP). Underlay will forward this packet from one switch to another switch using the outer IP and MAC header only.
Why not make a simple topology and see those benefits of VXLAN step by step !!!
 | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 02 - VXLAN - Data-Plane Topology The topology is very simple - we have two Cisco Nexus switches connected back-to-back (2 physical interfaces). Then we have one client connected with each switch (vlan 101). If we run a traditional network - one of the physical links (e1/1 and e1/2) in the switches will be active, the other one will be blocked by STP. As one link is blocked by STP, we are not utilizing all the links (no ECMP). And we cannot provide multi-tenancy just with Vlan alone (this is be discussed in detail later blog posts). Why I am calling the topology - Data-Plane only? As we want to understand how VXLAN works - we will not configure any control-plane routing protocols (OSPF, BGP, PIM etc.) to program our VXLAN data-plane. The less moving parts, the more easy to understand the concepts. Basic information about the topology -
|
Now we can start configuring our switches -
Leaf-Sw-01 Underlay Configuration -
!
!!! Underlay loopback0 which will be used to configure unnumbered physical interfaces.
interface loopback0
description Underlay-Routing-Loopback
ip address 10.81.0.1/32
!
!!! Inter-switch links are configured as layer-3 to eliminate STP.
interface Ethernet1/1
description To_Leaf-Sw-02_e1/1
no switchport
!!! change mtu to accommodate vxlan packet.
mtu 9216
!!! Cisco quirks to configure interface as unnumbered - medium must be p2p.
medium p2p
!!! Ip address is borrowed from loopback0.
ip unnumbered loopback0
no shutdown
!
interface Ethernet1/2
description To_Leaf-Sw-02_e1/2
no switchport
mtu 9216
medium p2p
ip unnumbered loopback0
no shutdown
!
!!! Overlay Vtep loopback (Vxlan tunnels will be established using loopback1).
interface loopback1
description Overlay-Vtep-Loopback
ip address 10.81.1.1/32
!
!!! Static routes for Vtep reachability.
!!! With dual static routes ECMP will be enabled over two physical interfaces.
ip route 10.81.1.2/32 Ethernet1/1 10.81.0.2 name Leaf-Sw-02-Vtep-Route-01
ip route 10.81.1.2/32 Ethernet1/2 10.81.0.2 name Leaf-Sw-02-Vtep-Route-02
!
Leaf-Sw-02 Underlay Configuration -
!
interface loopback0
description Underlay-Routing-Loopback
ip address 10.81.0.2/32
!
interface Ethernet1/1
description To_Leaf-Sw-01_e1/1
no switchport
mtu 9216
medium p2p
ip unnumbered loopback0
no shutdown
!
interface Ethernet1/2
description To_Leaf-Sw-01_e1/2
no switchport
mtu 9216
medium p2p
ip unnumbered loopback0
no shutdown
!
interface loopback1
description Overlay-Vtep-Loopback
ip address 10.81.1.2/32
!
ip route 10.81.1.1/32 Ethernet1/1 10.81.0.1 name Leaf-Sw-01-Vtep-Route-01
ip route 10.81.1.1/32 Ethernet1/2 10.81.0.1 name Leaf-Sw-01-Vtep-Route-02
!
We will verify that our underlay is working as layer-3 and we have achieve ECMP for the overlay.
Leaf-Sw-01# show ip route static
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
!!! Two static routes for Leaf-Sw-02's loopback1 using two different physical interfaces - ECMP.
10.81.1.2/32, ubest/mbest: 2/0
*via 10.81.0.2, Eth1/1, [1/0], 10:42:48, static
*via 10.81.0.2, Eth1/2, [1/0], 10:42:48, static
!!! Successful ping from Leaf-Sw-01's loopback1 to Leaf-Sw-02's loopback1.
Leaf-Sw-01# ping 10.81.1.2 source 10.81.1.1
PING 10.81.1.2 (10.81.1.2) from 10.81.1.1: 56 data bytes
64 bytes from 10.81.1.2: icmp_seq=0 ttl=254 time=5.14 ms
64 bytes from 10.81.1.2: icmp_seq=1 ttl=254 time=3.341 ms
64 bytes from 10.81.1.2: icmp_seq=2 ttl=254 time=1.555 ms
64 bytes from 10.81.1.2: icmp_seq=3 ttl=254 time=2.292 ms
64 bytes from 10.81.1.2: icmp_seq=4 ttl=254 time=1.513 ms
--- 10.81.1.2 ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 1.513/2.768/5.14 ms
Leaf-Sw-01 Overlay Configuration -
!
!!! Allows to map a VLAN to a VXLAN Network Identifier (VNI).
feature vn-segment-vlan-based
!
vlan 101
name Client-Test-Vlan
!!! Assigning a VNI to the Vlan.
vn-segment 100101
!
interface Ethernet1/15
!!! Port connected to Pc-01 is configured for vlan 101 in access mode.
switchport access vlan 101
!
!!! Allows to configure nve1 interface - which is the actual vxlan/vtep tunnel interface. Cisco calls it - nve (Network Virtualization Edge).
feature nv overlay
!
interface nve1
no shutdown
description Vtep-Vxlan
!!! Vtep interface borrows it's IP address from loopback1.
source-interface loopback1
!!! Enable vni 100101 (vlan 101) traffic tunneling over nve/vtep/vxlan interface.
member vni 100101
!!! Most important command. How to handle BUM traffic (Broadcast, Unknown Unicast, and Multicast).
!!! BUM traffic received by Leaf-Sw-01, for example - ARP traffic will be forwarded to Leaf-Sw-02 over nve interface.
!!! We are defining our nve neighbors statically as no dynamic routing protocol is involved. That's why we are calling our vxlan learning is data-plane based only.
ingress-replication protocol static
!!! Leaf-Sw-02 is the static peer.
peer-ip 10.81.1.2
!
Leaf-Sw-02 Overlay Configuration -
!
feature vn-segment-vlan-based
!
vlan 101
name Client-Test-Vlan
vn-segment 100101
!
interface Ethernet1/15
switchport access vlan 101
!
feature nv overlay
!
interface nve1
no shutdown
description Vtep-Vxlan
source-interface loopback1
member vni 100101
ingress-replication protocol static
!!! Leaf-Sw-01 is the static peer.
peer-ip 10.81.1.2
!
With above we have implemented VXLAN using the data-plane only for our topology.
Data-Plane Verification
Leaf-Sw-01# show interface nve1
nve1 is up
!!! Our nve interface is up with mtu 9216 and encapsulation vxlan.
admin state is up, Hardware: NVE
Description: Vtep-Vxlan
MTU 9216 bytes
Encapsulation VXLAN
Auto-mdix is turned off
RX
ucast: 7460 pkts, 715514 bytes - mcast: 1 pkts, 60 bytes
TX
ucast: 7461 pkts, 1088624 bytes - mcast: 0 pkts, 0 bytes
Leaf-Sw-01# show nve interface nve1 detail
Interface: nve1, State: Up, encapsulation: VXLAN
VPC Capability: VPC-VIP-Only [not-notified]
Local Router MAC: 5000.0100.1b08
!!! VXLAN is operating in Data-Plane mode.
Host Learning Mode: Data-Plane
Source-Interface: loopback1 (primary: 10.81.1.1, secondary: 0.0.0.0)
Source Interface State: Up
Virtual RMAC Advertisement: No
NVE Flags:
Interface Handle: 0x49000001
Source Interface hold-down-time: 180
Source Interface hold-up-time: 30
Remaining hold-down time: 0 seconds
Virtual Router MAC: N/A
Interface state: nve-intf-add-complete
Fabric convergence time: 135 seconds
Fabric convergence time left: 0 seconds
Leaf-Sw-01# show nve peers
Interface Peer-IP State LearnType Uptime Router-Mac
--------- -------------------------------------- ----- --------- -------- -----------------
nve1 10.81.1.2 Up DP 09:02:01 n/a
Leaf-Sw-01# show nve peers detail
Details of nve Peers:
----------------------------------------
!!! Leaf-Sw-02 is the peer.
Peer-Ip: 10.81.1.2
NVE Interface : nve1
Peer State : Up
Peer Uptime : 09:02:10
Router-Mac : n/a
Peer First VNI : 100101
Time since Create : 09:02:10
!!! Leaf-Sw-02 is tunneling VNI/VLAN - 100101/101 using VXLAN.
Configured VNIs : 100101
Provision State : peer-add-complete
Learnt CP VNIs : 100101
vni assignment mode : SYMMETRIC
Peer Location : N/A
Group policy capable: no
----------------------------------------
Ping packet walkthrough from Pc-01 to Pc-02
For successful ping from Pc-01 to Pc-02 - first Pc-01 will generate an ARP request packet to know the mac address of Pc-02. Pc-02 will send the ARP reply packet. After ARP reply packet is received by Pc-01, it will generate the ICMP request and will get an ICMP reply form the Pc-02.
root@Pc-01:~# ping 10.85.101.12
PING 10.85.101.12 (10.85.101.12) 56(84) bytes of data.
64 bytes from 10.85.101.12: icmp_seq=1 ttl=64 time=5.33 ms
64 bytes from 10.85.101.12: icmp_seq=2 ttl=64 time=3.23 ms
64 bytes from 10.85.101.12: icmp_seq=3 ttl=64 time=3.28 ms
64 bytes from 10.85.101.12: icmp_seq=4 ttl=64 time=2.99 ms
--- 10.85.101.12 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
Let's have a look at the ARP request packet -
 |
| 03 - ARP Request |
In above we can see our original layer-2 frame (blue color marked) is encapsulated in a layer-3 vxlan packet (red color marked). The inner frame's source mac is - 50:c0:6a:00:04:00 (Pc-01's mac address) and destination mac is - ffff.ffff.fff (broadcast mac address). The outer packet's source IP is 10.81.1.1 (Leaf-Sw-01's Vtep/NVE) and destination IP is 10.85.1.2 (Leaf-Sw-02 Vtep/NVE).
ARP reply packet is exactly similar - just direction is reversed as this time Leaf-Sw-02 is creating the packet/frame.
 |
| 04 - ARP Reply Now we can have a look at our ICMP request and reply unicast packet. These two packets are straight forward. ICMP request will be tunneled by Leaf-Sw-01 and ICMP reply will be tunneled by Leaf-Sw-02. |
05 - ICMP Request
06 - ICMP Reply
We can verify the mac address learning from the switches -
Leaf-Sw-01# show mac address-table vlan 101
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan,
(NA)- Not Applicable A - ESI Active Path, S - ESI Standby Path
TL - True Learned, PS - Peer Sync, RO - Re-originate
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
!!! Pc-02's mac address is learned over nve interface with next hop 10.81.1.2 - Leaf-Sw-02's NVE/Vtep ip address.
* 101 5077.ae00.0500 dynamic NA F F nve1(10.81.1.2)
!!! Pc-01's mac address is learned over directly connected interface e1/15.
* 101 50c0.6a00.0400 dynamic NA F F Eth1/15
Leaf-Sw-02(config)# show mac address-table vlan 101
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC, ~ - vsan,
(NA)- Not Applicable A - ESI Active Path, S - ESI Standby Path
TL - True Learned, PS - Peer Sync, RO - Re-originate
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
!!! Pc-02's mac address is learned over directly connected interface e1/15.
* 101 5077.ae00.0500 dynamic NA F F Eth1/15
!!! Pc-01's mac address is learned over nve interface with next hop 10.81.1.1 - Leaf-Sw-01's NVE/Vtep ip address.
* 101 50c0.6a00.0400 dynamic NA F F nve1(10.81.1.1)
Now with VXLAN we have solved two problems we mentioned earlier, our core links (inter-switch links) is not using STP any more and inter-switch links are ECMP aware.
With that we can wrap up today's blog.
Comments
Post a Comment