Abstract
Multi-tenant bare-metal cloud providers face a fundamentally different network architecture problem than hyperscalers running software-defined overlays on virtualized hosts. When tenants provision their own operating systems and kernel configurations directly on physical hardware, the network layer must provide isolation, mobility, and multi-tenancy without relying on a hypervisor shim to handle encapsulation. EVPN-VXLAN, combining the BGP-based control plane of EVPN (Ethernet VPN, RFC 7432) with the VXLAN data plane (RFC 7348), has emerged as the dominant fabric design for this use case, replacing proprietary VLAN-stitching schemes that struggled to scale beyond a few hundred tenants.
EVPN Control Plane Fundamentals
EVPN distributes MAC and IP reachability information via BGP extended communities, replacing the traditional flood-and-learn approach for VXLAN tunnels. In a spine-leaf topology, each leaf switch peers with the spines using iBGP (or eBGP with private ASNs in the Clos model popularized by Cumulus/NVIDIA and Arista). When a bare-metal host comes online, its leaf switch learns the host MAC via ARP suppression - the switch itself responds to ARP requests on behalf of the host using MAC-IP bindings distributed via EVPN Type 2 routes. This eliminates flooded ARP broadcasts across the entire VNI (VXLAN Network Identifier), which at thousands of tenant segments would generate substantial background traffic. EVPN Type 5 routes carry IP prefixes for inter-subnet routing, enabling distributed anycast gateways where every leaf switch acts as the default gateway for attached hosts - eliminating tromboning of inter-subnet traffic through a central router.
Tenant Isolation and VNI Allocation
Each tenant typically maps to one or more VNIs, with L2 VNIs providing Ethernet segment isolation and an L3 VNI carrying routed traffic within the tenant VRF. The VNI space (24 bits, roughly 16 million identifiers) is ample for multi-tenant use, but operational VNI lifecycle management - allocation, deallocation, propagation across the fabric on tenant provision/deprovision - requires automation. Equinix Metal and Oxide Computer Company have both published on their approaches to this problem; the common pattern is a network controller that programs leaf switches via Netconf/gNMI on tenant lifecycle events, maintaining VNI-to-tenant mapping in a central control plane that drives BGP policy.
BUM Traffic Handling
Broadcast, Unknown Unicast, and Multicast (BUM) traffic in VXLAN can use either ingress replication (IR) or multicast-based distribution. Ingress replication, where the originating VTEP copies BUM packets to every remote VTEP in the VNI, is operationally simpler - no multicast routing required in the underlay - and is universally supported. At large scale (hundreds of VTEPs per VNI), IR replication overhead grows linearly. Most bare-metal cloud providers with large-scale deployments use ingress replication for smaller tenant VNIs and accept the tradeoff; underlay multicast is deployed only when BUM traffic volume becomes a measurable fraction of fabric capacity.
Failure Modes and Fast Convergence
EVPN-VXLAN fabrics converge faster than legacy L2 spanning-tree designs - BFD on the BGP sessions between leaf and spine detects link failures in sub-second intervals, and EVPN withdraws the affected Type 2 and Type 5 routes promptly. The operational complexity shifts to ECMP hash polarization (ensuring traffic distributes across all spine uplinks under all packet size distributions) and underlay MTU sizing - VXLAN adds 50 bytes of overhead, requiring underlay MTU of at least 1600 on all fabric links to avoid fragmentation of jumbo-frame workloads. These are well-understood problems with established tooling, but they require explicit validation at fabric commissioning rather than discovery in production.