VXLAN Journey - Part 09 - Multi-Site VXLAN Anycast Border Gateway Configuration

In this blog we will configure special leaf nodes called border-gateways and prepare our data-centers for seamless l3vni (ip-vrf) and l2vni (mac-vrf) extension across sites.

Border-Gateways comes in two flavors - anycast and vpc. Anycast border-gateways are just standalone switches. vPC border-gateways are two border-gateways which are also vPC-pairs. We will configure the variant called - anycast border-gateway.

Our border-gateways and Spine switches topology is given below (Leafs and Spines connections are ignored intentionally; for that see previous blogs) - 

01 - Anycast Border-Gateway DCI/Fabric Topology

A border-gateway connects to Spine switches in the local site - those links are called fabric-links. It also connects to remote site's border-gateways - those links are called dci-links. In Cisco NXOS both dci and fabric links must be physical interface. Both dci and fabric links are also tracked. Tracking means if a border gateway either looses all of its fabric or dci links; then the vip interface - loopback100 will be brought down to isolate the bgw from multi-site traffic forwarding.

Limitation of anycast border-gateways - Cisco's Nexus anycast border-gateways has a big limitation. Any kind of layer-2 attachment is not supported in anycast border-gateways. In simple terms - configuring vlan svi (local vlan svi or anycast vlan svi) is not supported. If one has a need to create vlan svi (layer-2 attachment) in border-gateways - use vPC border-gateways or use service-leaf switch.

Our border-gateways are connected back-to-back in full mesh. Back-to-back means the border-gateways of different sites are connected with each other directly. There is no intermediate network, MPLS network or service provider between them. It is just connected directly over physical interface (eth1/4 or eth1/5) using a /30 ip network. Also site-local gateways are connected with each other with so called compensation dci-link (eth1/3).

Border-gateways will use bgp ipv4 unicast for underlay routing (ibgp and ebgp) and bgp l2vpn-evpn (ebgp) for overlay routing between each other. With ipv4 unicast border-gateways advertise the reachability of loopback0, loopback1 and loopback100 to each other. Then l2vpn-evpn is used to forward vxlan encapsulated traffic between sites.

The routing topology between border-gateways are shown below - 

02 - Bgws BGP Topology

In above intra-site border-gateway is connected with each other over eth1/3 interface- This is also a dci-link. But it will just run ibgp ipv4 unicast only - it will provide a backup routing path (for remote site's bgw's loopback0, loopback1 and loopback100) when a site-local bgw lost all of it's link towards remote site's bgws. Do not configure l2vpn-evpn over eth1/3 link; then you will bypass the spine-layer in forwarding (I have learned it the hard way by doing the mistake).

Virtual vtep ip address (loopback 100) - in border-gateways we will need one extra loopback ip address for vxlan forwarding. The same shared ip address is used in a site's all border gateways and usually configured as loopback100. An internal vxlan route is announced from the local site's bgws towards remote site's bgws using loopback100 (vip) as next-hop. Also externally learned vxlan route is announced from the local site's bgws to the spines (local site) using loopback100 (vip) as next-hop.

Bgws mask/suppress local evpn routes. They re-originate local evpn routes with their own identities (RD, next-hop) and announce only those re-originated routes towards remote bgws. The next-hop will be loopback 100. But bgws will use another second RD for re-origination. Cisco calls this feature dual-rd. It uses the format BGP_ASN/Site_Id:vrf_id (l3vni) or BGP_ASN/Site_Id:Vlan_Id+32767 (l2vni). This dual rd is generated automatically by NXOS. But in our case we need to manually configure it because we are using 32 bit ASN (limitation of RD field length - 6 bytes). But do not forget each site will use their own unique dual-rd value, but it will be shared between site-local border-gateways.

We are using below addressing in our topology -

Dc-01 BGP ASN - 4200000001

Dc-02 BGP ASN - 4200000002

Dc-01 ibgp dci-link - 10.81.2.X/24

Dc-02 ibgp dci-link - 10.82.2.X/24

Dc-01 and Dc-02 ebgp dci-link - 10.83.2.X/24

Dc-01 and Dc-02 vip loopback100 - 10.83.100.X/24

Dc-01 loopback 0 - 10.81.0.X/24

Dc-01 loopback 1 - 10.81.1.X/24

Dc-02 loopback 0 - 10.82.0.X/24

Dc-02 loopback 1 - 10.82.1.X/24

Dc-01 dual rd - 59905

Dc-02 dual rd - 59906

Lets start configuring our border-gateways. In this blog we will just activate border-gateways features. But we will not configure any l3vni/l2vni extension between our data centers, that will be done in part - 10.

Dc-01 Anycast Border-Gateway Configuration

Dc-01-Border-Gateway-03 Configuration

hostname Dc-01-Border-Gateway-03

nv overlay evpn

feature ospf

feature bgp

feature interface-vlan

feature vn-segment-vlan-based

feature nv overlay

!!! Enable multisite border-gateway feature.

!!! 4200000001 is site-id which is equal to our BGP-ASN.

evpn multisite border-gateway 4200000001

  !!! Loopback100 bring-up will be delayed for 300 seconds.

  !!! This is to ensure overlay/underlay control-plane routing converged first.

  delay-restore time 300

!!! Prefix list to select loopback0, loopback1 and loopback100.

ip prefix-list Pfx_Dc-02-Bgw_Ipv4_Out seq 10 permit 10.81.0.3/32 

ip prefix-list Pfx_Dc-02-Bgw_Ipv4_Out seq 20 permit 10.81.1.3/32 

ip prefix-list Pfx_Dc-02-Bgw_Ipv4_Out seq 30 permit 10.83.100.1/32

!!!! Outbound route-map which controls what is announced to remote-site's border-gateway using ebgp underlay.

route-map Rmap_Dc-02-Bgw_Ipv4_Out permit 10

  match ip address prefix-list Pfx_Dc-02-Bgw_Ipv4_Out

!!! Route-map to redistribute connected routes (loop0, loop1 and loop100) into bgp ipv4 unicast.

route-map Rmap_Tag-54321 permit 10

  match tag 54321 

interface nve1

  no shutdown

  description Vtep_Vxlan

  host-reachability protocol bgp

  source-interface loopback1

  !!! loopback 100 will be used as multi-site vip vtep address.

  multisite border-gateway interface loopback100

interface Ethernet1/1

  description To_Dc-01-Spine-01_e1/1

  no switchport

  mtu 9216

  medium p2p

  no ip redirects

  ip unnumbered loopback0

  ip router ospf Underlay area 0.0.0.0

  no shutdown

  !!! Connection to spine will be fabric-link and tracked.

  evpn multisite fabric-tracking

interface Ethernet1/2

  description To_Dc-01-Spine-02_e1/1

  no switchport

  mtu 9216

  medium p2p

  no ip redirects

  ip unnumbered loopback0

  ip router ospf Underlay area 0.0.0.0

  no shutdown

  !!! Connection to spine will be fabric-link and tracked.

  evpn multisite fabric-tracking

!!! Site-local border-gateways are connected with each-other.

!!! This is to provide backup path to remote-site's border-gateways loopbacks using ibgp.

interface Ethernet1/3

  description To_Dc-01-Border-Gateway-04_e1/3

  no switchport

  mtu 9216

  no ip redirects

  ip address 10.81.2.1/30

  no shutdown

  !!! Connection to site-local bgw will be dci-link and tracked.

  evpn multisite dci-tracking

interface Ethernet1/4

  description To_Dc-02-Border-Gateway-03_e1/4

  no switchport

  mtu 9216

  no ip redirects

  ip address 10.83.2.1/30

  no shutdown

  !!! Connection to site-remote bgw will be dci-link and tracked.

  evpn multisite dci-tracking

interface Ethernet1/5

  description To_Dc-02-Border-Gateway-04_e1/4

  no switchport

  mtu 9216

  no ip redirects

  ip address 10.83.2.5/30

  no shutdown

  !!! Connection to site-remote bgw will be dci-link and tracked.

  evpn multisite dci-tracking

interface loopback0

  description Underlay_Routing_Loopback

  ip address 10.81.0.3/32 tag 54321

  ip router ospf Underlay area 0.0.0.0

interface loopback1

  description Overlay_Vtep_Loopback

  ip address 10.81.1.3/32 tag 54321

  ip router ospf Underlay area 0.0.0.0

!!! Vtep's shared vip address.

!!! The same ip address will use used by all the bgws within the local site (Dc-01).

interface loopback100

  description Multisite_Overlay_Vtep_Loopback

  ip address 10.83.100.1/32 tag 54321

  ip router ospf Underlay area 0.0.0.0

router ospf Underlay

  router-id 10.81.0.3

  log-adjacency-changes detail

router bgp 4200000001

  router-id 10.81.0.3

  log-neighbor-changes

  !!! Manual dual rd generation.

  rd dual id 59905

  address-family ipv4 unicast

    !!! Redistribute the loopbacks into bgp ipv4 unicast (underlay).

    redistribute direct route-map Rmap_Tag-54321

    maximum-paths 64

    maximum-paths ibgp 64

  neighbor 10.81.0.1

    remote-as 4200000001

    description Dc-01-Spine-01_Evpn

    update-source loopback0

    address-family l2vpn evpn

      send-community

      send-community extended

  neighbor 10.81.0.2

    remote-as 4200000001

    description Dc-01-Spine-02_Evpn

    update-source loopback0

    address-family l2vpn evpn

      send-community

      send-community extended

  !!! Compensation ibgp ipv4 unicast (underlay) between site-local bgws.

  neighbor 10.81.2.2

    remote-as 4200000001

    description Dc-01-Border-Gateway-04_Ipv4_Unicast

    update-source Ethernet1/3

    address-family ipv4 unicast

      next-hop-self

  !!! Ebgp ipv4 unicast (underlay) between site-remote bgws.     

  neighbor 10.83.2.2

    remote-as 4200000002

    description Dc-02-Border-Gateway-03_Ipv4_Unicast

    update-source Ethernet1/4

    address-family ipv4 unicast

      !!! Route-map only local loopbacks are permitted.

      route-map Rmap_Dc-02-Bgw_Ipv4_Out out

      next-hop-self

  !!! Ebgp ipv4 unicast (underlay) between site-remote bgws.     

  neighbor 10.83.2.6

    remote-as 4200000002

    description Dc-02-Border-Gateway-04_Ipv4_Unicast

    update-source Ethernet1/5

    address-family ipv4 unicast

      !!! Route-map only local loopbacks are permitted.

      route-map Rmap_Dc-02-Bgw_Ipv4_Out out

      next-hop-self

  !!! Ebgp l2vpn-evpn (overlay) between site-remote bgws.       

  neighbor 10.82.0.3

    remote-as 4200000002

    description description Dc-02-Border-Gateway-03_Evpn

    update-source loopback0

    !!! Ebgp-multihop is required when neighborship is established over loopback interfaces.

    ebgp-multihop 5

    !!! Enable l2vpn-evpn route re-origination.

    peer-type fabric-external

    address-family l2vpn evpn

      send-community

      send-community extended

      !!! Incoming evpn-route's route-targets bgp_asn part will be replaced with local bgp_asn.

      !!! As we are using 32 bit asn and rt auto import and export; our ASN is always 23456.

      rewrite-evpn-rt-asn

  !!! Ebgp l2vpn-evpn (overlay) between site-remote bgws.       

  neighbor 10.82.0.4

    remote-as 4200000002

    description description Dc-02-Border-Gateway-04_Evpn

    update-source loopback0

    !!! Ebgp-multihop is required when neighborship is established over loopback interfaces.

    ebgp-multihop 5

    !!! Enable l2vpn-evpn route re-origination.

    peer-type fabric-external

    address-family l2vpn evpn

      send-community

      send-community extended

      !!! Incoming evpn-route's route-targets bgp_asn part will be replaced with local bgp_asn.

      !!! As we are using 32 bit asn and rt auto import and export; our ASN is always 23456.

      rewrite-evpn-rt-asn

Dc-01-Border-Gateway-04 Configuration

hostname Dc-01-Border-Gateway-04

nv overlay evpn

feature ospf

feature bgp

feature interface-vlan

feature vn-segment-vlan-based

feature nv overlay

evpn multisite border-gateway 4200000001

  delay-restore time 300

ip prefix-list Pfx_Dc-02-Bgw_Ipv4_Out seq 10 permit 10.81.0.4/32 

ip prefix-list Pfx_Dc-02-Bgw_Ipv4_Out seq 20 permit 10.81.1.4/32 

ip prefix-list Pfx_Dc-02-Bgw_Ipv4_Out seq 30 permit 10.83.100.1/32

route-map Rmap_Dc-02-Bgw_Ipv4_Out permit 10

  match ip address prefix-list Pfx_Dc-02-Bgw_Ipv4_Out

route-map Rmap_Tag-54321 permit 10

  match tag 54321 

interface nve1

  no shutdown

  description Vtep_Vxlan

  host-reachability protocol bgp

  source-interface loopback1

  multisite border-gateway interface loopback100

interface Ethernet1/1

  description To_Dc-01-Spine-01_e1/2

  no switchport

  mtu 9216

  medium p2p

  no ip redirects

  ip unnumbered loopback0

  ip router ospf Underlay area 0.0.0.0

  no shutdown

  evpn multisite fabric-tracking

interface Ethernet1/2

  description To_Dc-01-Spine-02_e1/2

  no switchport

  mtu 9216

  medium p2p

  no ip redirects

  ip unnumbered loopback0

  ip router ospf Underlay area 0.0.0.0

  no shutdown

  evpn multisite fabric-tracking

interface Ethernet1/3

  description To_Dc-01-Border-Gateway-03_e1/3

  no switchport

  mtu 9216

  no ip redirects

  ip address 10.81.2.2/30

  no shutdown

  evpn multisite dci-tracking

interface Ethernet1/4

  description To_Dc-02-Border-Gateway-03_e1/5

  no switchport

  mtu 9216

  no ip redirects

  ip address 10.83.2.9/30

  no shutdown

  evpn multisite dci-tracking

interface Ethernet1/5

  description To_Dc-02-Border-Gateway-04_e1/5

  no switchport

  mtu 9216

  no ip redirects

  ip address 10.83.2.13/30

  no shutdown

  evpn multisite dci-tracking

interface loopback0

  description Underlay_Routing_Loopback

  ip address 10.81.0.4/32 tag 54321

  ip router ospf Underlay area 0.0.0.0

interface loopback1

  description Overlay_Vtep_Loopback

  ip address 10.81.1.4/32 tag 54321

  ip router ospf Underlay area 0.0.0.0

interface loopback100

  description Multisite_Overlay_Vtep_Loopback

  ip address 10.83.100.1/32 tag 54321

  ip router ospf Underlay area 0.0.0.0

router ospf Underlay

  router-id 10.81.0.4

  log-adjacency-changes detail

router bgp 4200000001

  router-id 10.81.0.4

  log-neighbor-changes

  rd dual id 59905

  address-family ipv4 unicast

    redistribute direct route-map Rmap_Tag-54321

    maximum-paths 64

    maximum-paths ibgp 64

 

  neighbor 10.81.0.1

    remote-as 4200000001

    description Dc-01-Spine-01_Evpn

    update-source loopback0

    address-family l2vpn evpn

      send-community

      send-community extended

  neighbor 10.81.0.2

    remote-as 4200000001

    description Dc-01-Spine-02_Evpn

    update-source loopback0

    address-family l2vpn evpn

      send-community

      send-community extended

  neighbor 10.81.2.1

    remote-as 4200000001

    description Dc-01-Border-Gateway-03_Ipv4_Unicast

    update-source Ethernet1/3

    address-family ipv4 unicast

      next-hop-self

  

  neighbor 10.83.2.10

    remote-as 4200000002

    description Dc-02-Border-Gateway-03_Ipv4_Unicast

    update-source Ethernet1/4

    address-family ipv4 unicast

      route-map Rmap_Dc-02-Bgw_Ipv4_Out out

      next-hop-self

  

  neighbor 10.83.2.14

    remote-as 4200000002

    description Dc-02-Border-Gateway-04_Ipv4_Unicast

    update-source Ethernet1/5

    address-family ipv4 unicast

      route-map Rmap_Dc-02-Bgw_Ipv4_Out out

      next-hop-self

  

  neighbor 10.82.0.3

    remote-as 4200000002

    description description Dc-02-Border-Gateway-03_Evpn

    update-source loopback0

    ebgp-multihop 5

    peer-type fabric-external

    address-family l2vpn evpn

      send-community

      send-community extended

      rewrite-evpn-rt-asn

  neighbor 10.82.0.4

    remote-as 4200000002

    description description Dc-02-Border-Gateway-04_Evpn

    update-source loopback0

    ebgp-multihop 5

    peer-type fabric-external

    address-family l2vpn evpn

      send-community

      send-community extended

      rewrite-evpn-rt-asn

Dc-02 Anycast Border-Gateway Configuration

Dc-02-Border-Gateway-03 Configuration

hostname Dc-02-Border-Gateway-03

nv overlay evpn

feature ospf

feature bgp

feature interface-vlan

feature vn-segment-vlan-based

feature nv overlay

evpn multisite border-gateway 4200000002

  delay-restore time 300

ip prefix-list Pfx_Dc-01-Bgw_Ipv4_Out seq 10 permit 10.82.0.3/32 

ip prefix-list Pfx_Dc-01-Bgw_Ipv4_Out seq 20 permit 10.82.1.3/32 

ip prefix-list Pfx_Dc-01-Bgw_Ipv4_Out seq 30 permit 10.83.100.2/32 

route-map Rmap_Dc-01-Bgw_Ipv4_Out permit 10

  match ip address prefix-list Pfx_Dc-01-Bgw_Ipv4_Out 

route-map Rmap_Tag-54321 permit 10

  match tag 54321 

interface nve1

  no shutdown

  description Vtep_Vxlan

  host-reachability protocol bgp

  advertise virtual-rmac

  source-interface loopback1

  multisite border-gateway interface loopback100

interface Ethernet1/1

  description To_Dc-02-Spine-01_e1/1

  no switchport

  mtu 9216

  medium p2p

  no ip redirects

  ip unnumbered loopback0

  ip router ospf Underlay area 0.0.0.0

  no shutdown

  evpn multisite fabric-tracking

interface Ethernet1/2

  description To_Dc-02-Spine-02_e1/1

  no switchport

  mtu 9216

  medium p2p

  no ip redirects

  ip unnumbered loopback0

  ip router ospf Underlay area 0.0.0.0

  no shutdown

  evpn multisite fabric-tracking

interface Ethernet1/3

  description To_Dc-02-Border-Gateway-04_e1/3

  no switchport

  mtu 9216

  no ip redirects

  ip address 10.82.2.1/30

  no shutdown

  evpn multisite dci-tracking

interface Ethernet1/4

  description To_Dc-01-Border-Gateway-03_e1/4

  no switchport

  mtu 9216

  no ip redirects

  ip address 10.83.2.2/30

  no shutdown

  evpn multisite dci-tracking

interface Ethernet1/5

  description To_Dc-01-Border-Gateway-04_e1/4

  no switchport

  mtu 9216

  no ip redirects

  ip address 10.83.2.10/30

  no shutdown

  evpn multisite dci-tracking

interface loopback0

  description Underlay_Routing_Loopback

  ip address 10.82.0.3/32 tag 54321

  ip router ospf Underlay area 0.0.0.0

interface loopback1

  description Overlay_Vtep_Loopback

  ip address 10.82.1.3/32 tag 54321

  ip router ospf Underlay area 0.0.0.0

interface loopback100

  description Multisite_Overlay_Vtep_Loopback

  ip address 10.83.100.2/32 tag 54321

  ip router ospf Underlay area 0.0.0.0

router ospf Underlay

  router-id 10.82.0.3

  log-adjacency-changes detail

router bgp 4200000002

  router-id 10.82.0.3

  log-neighbor-changes

  rd dual id 59906

 

 address-family ipv4 unicast

    redistribute direct route-map Rmap_Tag-54321

    maximum-paths 64

    maximum-paths ibgp 64

  neighbor 10.82.0.1

    remote-as 4200000002

    description Dc-02-Spine-01_Evpn

    update-source loopback0

    address-family l2vpn evpn

      send-community

      send-community extended

  neighbor 10.82.0.2

    remote-as 4200000002

    description Dc-02-Spine-02_Evpn

    update-source loopback0

    address-family l2vpn evpn

      send-community

      send-community extended

  neighbor 10.82.2.2

    remote-as 4200000002

    description Dc-02-Border-Gateway-04_Ipv4_Unicast

    update-source Ethernet1/3

    address-family ipv4 unicast

      next-hop-self

  neighbor 10.83.2.1

    remote-as 4200000001

    description Dc-01-Border-Gateway-03_Ipv4_Unicast

    update-source Ethernet1/4

    address-family ipv4 unicast

      route-map Rmap_Dc-01-Bgw_Ipv4_Out out

      next-hop-self

  neighbor 10.83.2.9

    remote-as 4200000001

    description Dc-01-Border-Gateway-04_Ipv4_Unicast

    update-source Ethernet1/5

    address-family ipv4 unicast

      route-map Rmap_Dc-01-Bgw_Ipv4_Out out

      next-hop-self

  neighbor 10.81.0.3

    remote-as 4200000001

    description description Dc-01-Border-Gateway-03_Evpn

    update-source loopback0

    ebgp-multihop 5

    peer-type fabric-external

    address-family l2vpn evpn

      send-community

      send-community extended

      rewrite-evpn-rt-asn

  neighbor 10.81.0.4

    remote-as 4200000001

    description description Dc-01-Border-Gateway-04_Evpn

    update-source loopback0

    ebgp-multihop 5

    peer-type fabric-external

    address-family l2vpn evpn

      send-community

      send-community extended

      rewrite-evpn-rt-asn

Dc-02-Border-Gateway-04 Configuration

hostname Dc-02-Border-Gateway-04

nv overlay evpn

feature ospf

feature bgp

feature interface-vlan

feature vn-segment-vlan-based

feature nv overlay

evpn multisite border-gateway 4200000002

  delay-restore time 300

ip prefix-list Pfx_Dc-01-Bgw_Ipv4_Out seq 10 permit 10.82.0.3/32 

ip prefix-list Pfx_Dc-01-Bgw_Ipv4_Out seq 20 permit 10.82.1.3/32 

ip prefix-list Pfx_Dc-01-Bgw_Ipv4_Out seq 30 permit 10.83.100.2/32 

route-map Rmap_Dc-01-Bgw_Ipv4_Out permit 10

  match ip address prefix-list Pfx_Dc-01-Bgw_Ipv4_Out 

route-map Rmap_Tag-54321 permit 10

  match tag 54321 

interface nve1

  no shutdown

  description Vtep_Vxlan

  host-reachability protocol bgp

  advertise virtual-rmac

  source-interface loopback1

  multisite border-gateway interface loopback100

interface Ethernet1/1

  description To_Dc-02-Spine-01_e1/2

  no switchport

  mtu 9216

  medium p2p

  no ip redirects

  ip unnumbered loopback0

  ip router ospf Underlay area 0.0.0.0

  no shutdown

  evpn multisite fabric-tracking

interface Ethernet1/2

  description To_Dc-02-Spine-02_e1/2

  no switchport

  mtu 9216

  medium p2p

  no ip redirects

  ip unnumbered loopback0

  ip router ospf Underlay area 0.0.0.0

  no shutdown

  evpn multisite fabric-tracking

interface Ethernet1/3

  description To_Dc-02-Border-Gateway-03_e1/3

  no switchport

  mtu 9216

  no ip redirects

  ip address 10.82.2.2/30

  no shutdown

  evpn multisite dci-tracking

interface Ethernet1/4

  description To_Dc-01-Border-Gateway-03_e1/5

  no switchport

  mtu 9216

  no ip redirects

  ip address 10.83.2.6/30

  no shutdown

  evpn multisite dci-tracking

interface Ethernet1/5

  description To_Dc-01-Border-Gateway-04_e1/5

  no switchport

  mtu 9216

  no ip redirects

  ip address 10.83.2.14/30

  no shutdown

  evpn multisite dci-tracking

interface loopback0

  description Underlay_Routing_Loopback

  ip address 10.82.0.4/32 tag 54321

  ip router ospf Underlay area 0.0.0.0

interface loopback1

  description Overlay_Vtep_Loopback

  ip address 10.82.1.4/32 tag 54321

  ip router ospf Underlay area 0.0.0.0

interface loopback100

  description Multisite_Overlay_Vtep_Loopback

  ip address 10.83.100.2/32 tag 54321

  ip router ospf Underlay area 0.0.0.0

router ospf Underlay

  router-id 10.82.0.4

  log-adjacency-changes detail

router bgp 4200000002

  router-id 10.82.0.4

  log-neighbor-changes

  rd dual id 59906

  address-family ipv4 unicast

    redistribute direct route-map Rmap_Tag-54321

    maximum-paths 64

    maximum-paths ibgp 64

  

neighbor 10.82.0.1

    remote-as 4200000002

    description Dc-02-Spine-01_Evpn

    update-source loopback0

    address-family l2vpn evpn

      send-community

      send-community extended

  

  neighbor 10.82.0.2

    remote-as 4200000002

    description Dc-02-Spine-02_Evpn

    update-source loopback0

    address-family l2vpn evpn

      send-community

      send-community extended

  neighbor 10.82.2.1

    remote-as 4200000002

    description Dc-02-Border-Gateway-03_Ipv4_Unicast

    update-source Ethernet1/3

    address-family ipv4 unicast

      next-hop-self

  

  neighbor 10.83.2.5

    remote-as 4200000001

    description Dc-02-Border-Gateway-03_Ipv4_Unicast

    update-source Ethernet1/4

    address-family ipv4 unicast

      route-map Rmap_Dc-01-Bgw_Ipv4_Out out

      next-hop-self

  

  neighbor 10.83.2.13

    remote-as 4200000001

    description Dc-02-Border-Gateway-04_Ipv4_Unicast

    update-source Ethernet1/5

    address-family ipv4 unicast

      route-map Rmap_Dc-01-Bgw_Ipv4_Out out

      next-hop-self

  neighbor 10.81.0.3

    remote-as 4200000001

    description description Dc-01-Border-Gateway-03_Evpn

    update-source loopback0

    ebgp-multihop 5

    peer-type fabric-external

    address-family l2vpn evpn

      send-community

      send-community extended

      rewrite-evpn-rt-asn

  

  neighbor 10.81.0.4

    remote-as 4200000001

    description description Dc-01-Border-Gateway-04_Evpn

    update-source loopback0

    ebgp-multihop 5

    peer-type fabric-external

    address-family l2vpn evpn

      send-community

      send-community extended

      rewrite-evpn-rt-asn

Verification

Now our anycast border-gateway configurations are completed. 

Lets verify our underlay bgp routing - 

Dc-01-Border-Gateway-03# show ip bgp summary 

BGP summary information for VRF default, address family IPv4 Unicast

BGP router identifier 10.81.0.3, local AS number 4200000001

BGP table version is 136, IPv4 Unicast config peers 5, capable peers 3

10 network entries and 17 paths using 4368 bytes of memory

BGP attribute entries [4/1472], BGP AS path entries [1/6]

BGP community entries [0/0], BGP clusterlist entries [6/24]

Neighbor        V    AS               MsgRcvd    MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd

!!! ibgp to site-local border-gateway.

10.81.2.2       4 4200000001       3041       3027         136    0     0      00:31:07    8         

!!! ebgp to site-remote border-gateways.

10.83.2.2       4 4200000002      3065       3053         136    0     0      00:28:15    3         

10.83.2.6       4 4200000002      3068       3060         136    0    0       00:28:09   3         

Dc-01-Border-Gateway-03# show ip route bgp

IP Route Table for VRF "default"

'*' denotes best ucast next-hop

'**' denotes best mcast next-hop

'[x/y]' denotes [preference/metric]

'%<string>' in via output denotes VRF <string>

!!! Dc-02-Border-Gateway-03 loopback0.

10.82.0.3/32, ubest/mbest: 1/0

    *via 10.83.2.2, [20/0], 00:29:01, bgp-4200000001, external, tag 4200000002

!!! Dc-02-Border-Gateway-04 loopback0.

10.82.0.4/32, ubest/mbest: 1/0

    *via 10.83.2.6, [20/0], 00:28:55, bgp-4200000001, external, tag 4200000002

!!! Dc-02-Border-Gateway-03 loopback1.

10.82.1.3/32, ubest/mbest: 1/0

    *via 10.83.2.2, [20/0], 00:29:01, bgp-4200000001, external, tag 4200000002

!!! Dc-02-Border-Gateway-04 loopback1.

10.82.1.4/32, ubest/mbest: 1/0

    *via 10.83.2.6, [20/0], 00:28:55, bgp-4200000001, external, tag 4200000002

!!! Dc-02 Border-Gateway shared vip loopback100.

10.83.100.2/32, ubest/mbest: 2/0

    *via 10.83.2.2, [20/0], 00:29:01, bgp-4200000001, external, tag 4200000002

    *via 10.83.2.6, [20/0], 00:28:55, bgp-4200000001, external, tag 4200000002

Dc-01-Border-Gateway-03# show ip bgp 

BGP routing table information for VRF default, address family IPv4 Unicast

BGP table version is 136, Local Router ID is 10.81.0.3

Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best

Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist, I-injected

Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath, & - backup, 2 - best2

   Network                 Next Hop            Metric     LocPrf     Weight   Path

*>r10.81.0.3/32         0.0.0.0                  0        100        32768        ?

*>i10.81.0.4/32         10.81.2.2               0        100          0             ?

*>r10.81.1.3/32         0.0.0.0                  0        100        32768        ?

*>i10.81.1.4/32         10.81.2.2               0        100          0             ?

*>e10.82.0.3/32       10.83.2.2                0                       0      4200000002 ?

!!! Backup path to remote-site's loopbacks using ibgp underlay between local-site's border-gateways.

* i                            10.81.2.2                0        100          0     4200000002 ?

*>e10.82.0.4/32       10.83.2.6                0                       0     4200000002 ?

!!! Backup path to remote-site's loopbacks using ibgp underlay between local-site's border-gateways.

* i                            10.81.2.2                0        100          0    4200000002 ?

*>e10.82.1.3/32       10.83.2.2                0                       0    4200000002 ?

!!! Backup path to remote-site's loopbacks using ibgp underlay between local-site's border-gateways.

* i                            10.81.2.2                0        100          0   4200000002 ?

*>e10.82.1.4/32       10.83.2.6                0                       0   4200000002 ?

!!! Backup path to remote-site's loopbacks using ibgp underlay between local-site's border-gateways.

* i                            10.81.2.2                0        100          0  4200000002 ?

* i10.83.100.1/32     10.81.2.2                0        100          0          ?

*>r                          0.0.0.0                   0        100        32768     ?

*|e10.83.100.2/32     10.83.2.6              0                        0   4200000002 ?

*>e                          10.83.2.2               0                       0   4200000002 ?

!!! Backup path to remote-site's loopbacks using ibgp underlay between local-site's border-gateways.

* i                            10.81.2.2               0        100          0   4200000002 ?

We can have a look on our overlay l2vpn-evpn bgp -

Dc-01-Border-Gateway-03# show bgp l2vpn evpn summary 

BGP summary information for VRF default, address family L2VPN EVPN

BGP router identifier 10.81.0.3, local AS number 4200000001

BGP table version is 1255, L2VPN EVPN config peers 6, capable peers 4

9 network entries and 16 paths using 3900 bytes of memory

BGP attribute entries [15/5520], BGP AS path entries [0/0]

BGP community entries [0/0], BGP clusterlist entries [6/24]

Neighbor        V    AS               MsgRcvd    MsgSent   TblVer  InQ   OutQ   Up/Down  State/PfxRcd

!!! Site-local spines.

10.81.0.1       4 4200000001      3582        3125        1255    0       0      00:50:34       7         

10.81.0.2       4 4200000001      3583       3128         1255    0       0      00:50:32       7         

!!! Remote-site's border-gateways.

10.82.0.3       4 4200000002      3247       3089          1255    0      0      00:50:32       0         

10.82.0.4       4 4200000002      3263       3092          1255    0      0      00:50:31       0         

Neighbor        T        AS                 Type-1     Type-2     Type-3     Type-4     Type-5    

  

10.81.0.1       I    4200000001          0             4             2             1             0

  

10.81.0.2       I    4200000001          0             4             2             1             0

!!! Remote-site's border-gateways ebgp l2vpn-vpn; no prefix received as there is no l2vni/l3vni extension.  

10.82.0.3       E   4200000002          0              0             0             0            0      

10.82.0.4       E   4200000002          0              0             0             0            0     

At this stage we have successfully completed the anycast border-gateway configuration between two data-centers. But we are lacking l2vni/l2vni extension between then. In part - 10 we will implement l3vni (ip-vrf) extension between the two data-centers.