ACE Journal

eBPF-Based Load Balancing at the Edge

Who this is for: network and platform engineers running high-PPS edge nodes who want kernel-level L4 load balancing. The assumption is that you already terminate a lot of traffic on a small number of bare-metal or near-bare-metal hosts, you have profiled and found the load balancer itself on the hot path, and you are comfortable with a recent Linux kernel and the eBPF toolchain. If your traffic is modest or your bottleneck is elsewhere, a tuned userspace proxy is the simpler and more honest answer, and the “where it breaks” section says so plainly.

Where the packets actually go

Most load balancers people run, HAProxy, NGINX, Envoy, IPVS behind a netfilter ruleset, share a structural property: by the time they get to decide where a packet goes, the kernel has already done a lot of work on that packet’s behalf. It has allocated a socket buffer, walked it up through the receive path, matched it against conntrack and the netfilter hooks, and possibly copied it across the kernel-userspace boundary into a proxy process. Each of those steps is cheap in isolation. Multiply by a few hundred thousand packets per second on a single edge node and the load balancer’s own bookkeeping becomes a meaningful slice of the CPU budget, often the slice you notice first when you try to push a box harder.

XDP, the eXpress Data Path, moves the decision to the front of the line. An XDP program is eBPF bytecode the kernel verifies and JIT-compiles, then attaches to a network interface at the driver’s earliest receive hook, before the kernel allocates the sk_buff that represents the packet to the rest of the stack. The program sees the raw frame, makes a forwarding decision, and returns an action code. For a load balancer the decision is “which backend,” and the action is to rewrite and redirect. Because all of this happens before SKB allocation, the per-packet cost is dominated by the hash lookup and the header rewrite rather than by stack traversal. Figure 1 contrasts the two paths side by side.

flowchart TB
    subgraph trad[Traditional userspace / netfilter path]
        N1[NIC] --> S1[SKB allocation]
        S1 --> NF[netfilter / conntrack hooks]
        NF --> SK[socket layer]
        SK --> UP[userspace proxy<br/>copy across boundary]
        UP --> B1[backend]
    end
    subgraph xdp[XDP path]
        N2[NIC] --> XH[XDP hook<br/>before SKB alloc]
        XH --> MAP[BPF map lookup<br/>backend pool + hash]
        MAP --> RD[XDP_REDIRECT]
        RD --> B2[backend]
    end

Figure 1. Traditional netfilter/userspace path vs XDP path: XDP acts before SKB allocation, skipping socket, conntrack, and proxy stages that the traditional path traverses on every packet.

The contrast in the diagram is the whole argument. The traditional path does real, useful work at each stage, and for the great majority of services that work is invisible because the service is nowhere near the packet rate where it matters. The XDP path collapses the decision into a map lookup and a redirect that run before the expensive stages even begin. The cost you save is not any single stage. It is the cumulative tax of carrying every packet up through machinery it does not need just to be told where to go.

How an XDP load balancer makes its decision

An XDP program returns one of a small set of action codes, and three of them matter for load balancing:

Here is a deliberately minimal XDP program that shows the shape of the decision. It parses Ethernet and IPv4, hashes the source address to pick a slot, looks the backend up in a map, and redirects. It is not Katran, it omits checksum fixups and IPv6 and the encapsulation a real LB needs, but it is correct in structure and it verifies.

#include <linux/bpf.h>
#include <linux/if_ether.h>
#include <linux/ip.h>
#include <bpf/bpf_helpers.h>

/* The backend lookup table and pool maps are defined in the next listing. */
extern struct bpf_map_def backend_lookup;
extern struct bpf_map_def backend_pool;

SEC("xdp")
int xdp_lb(struct xdp_md *ctx)
{
    void *data     = (void *)(long)ctx->data;
    void *data_end = (void *)(long)ctx->data_end;

    struct ethhdr *eth = data;
    if ((void *)(eth + 1) > data_end)
        return XDP_PASS;
    if (eth->h_proto != __constant_htons(ETH_P_IP))
        return XDP_PASS;

    struct iphdr *ip = (void *)(eth + 1);
    if ((void *)(ip + 1) > data_end)
        return XDP_PASS;

The eBPF verifier requires every pointer dereference to be preceded by an explicit bounds check against data_end; a program that skips any check is rejected at load time, which is why these guards are non-negotiable. With the headers validated, the backend map lookup and redirect follow.

    /* Maglev lookup: hash the source into the precomputed table,
       then resolve the table slot to a backend index. */
    __u32 slot = ip->saddr % MAGLEV_TABLE_SIZE;
    __u32 *idx = bpf_map_lookup_elem(&backend_lookup, &slot);
    if (!idx)
        return XDP_PASS;

    struct backend *be = bpf_map_lookup_elem(&backend_pool, idx);
    if (!be || be->state != BACKEND_HEALTHY)
        return XDP_DROP;

    /* A real LB rewrites L2/L3 (or encapsulates) toward be->ifindex
       and fixes checksums here before redirecting. */
    return bpf_redirect(be->ifindex, 0);
}

char _license[] SEC("license") = "GPL";

The bounds checks before every dereference are not optional defensive style. The eBPF verifier rejects the program at load time unless it can prove every packet access stays inside [data, data_end), which is one of the reasons XDP programs are safe to run at this privileged a position. The verifier is also why these programs stay small and loop-free in spirit: it has to be able to reason about them.

The backend pool and consistent hashing

The data the program reads lives in BPF maps, kernel-resident key/value tables that both the XDP program and userspace can access. A load balancer uses at least two: an array map holding the backend pool, and an array map holding the Maglev lookup table that maps a hash slot to a backend index. Here is a representative definition.

#define MAGLEV_TABLE_SIZE 65537   /* prime, per the Maglev paper */
#define MAX_BACKENDS      4096

enum backend_state {
    BACKEND_HEALTHY   = 0,
    BACKEND_DRAINING  = 1,
    BACKEND_DOWN      = 2,
};

struct backend {
    __u32 ifindex;        /* redirect target */
    __u32 daddr;          /* backend address (encap or DNAT target) */
    __u16 state;          /* enum backend_state */
    __u16 weight;
};

/* Maglev table: slot -> backend index. Precomputed in userspace. */
struct {
    __uint(type, BPF_MAP_TYPE_ARRAY);
    __uint(max_entries, MAGLEV_TABLE_SIZE);
    __type(key, __u32);
    __type(value, __u32);
} backend_lookup SEC(".maps");

/* Backend pool: index -> backend record. */
struct {
    __uint(type, BPF_MAP_TYPE_ARRAY);
    __uint(max_entries, MAX_BACKENDS);
    __type(key, __u32);
    __type(value, struct backend);
} backend_pool SEC(".maps");

Maglev hashing, the scheme Katran uses, exists to solve a specific problem with naive consistent hashing: when the backend set changes, you want as few existing flows as possible to be remapped to a different backend, and you want the load to stay even. Maglev precomputes a fixed-size lookup table, the prime MAGLEV_TABLE_SIZE above, in which each slot points at a backend, arranged so that adding or removing one backend disturbs only a small fraction of slots. The XDP program does a single array lookup per packet, which is why it is fast, and the cleverness all lives in how userspace fills the table.

For traffic that needs session affinity, a single flow pinned to one backend for its lifetime, you add a third map, a hash map keyed by the connection 5-tuple, that records the chosen backend on the first packet and is consulted on every subsequent packet of that flow. This is the connection-tracking, or conntrack, table, and it is where the memory math gets real. A per-connection entry costs roughly 64 to 128 bytes once you account for the key, the value, and the hash map’s own overhead. A table sized for one million concurrent connections therefore locks down on the order of 64 to 128 MB of unswappable kernel memory.

The practical guidance is to size the conntrack map to your observed peak concurrent connection count plus about 20 percent headroom, and to monitor map insertion failures, the bpf_map_full-style errors, as a first-class metric. A BPF hash map that hits max_entries does not grow. New flows that cannot get an entry fail closed, which manifests as dropped or reset connections that are maddening to diagnose if you are not already watching the map’s fill level. Size it deliberately, alarm on it, and treat the map size as a capacity decision rather than a default you copied.

The control plane: health checks live in userspace

An XDP program cannot open a TCP connection to a backend to see if it is alive. It runs in a constrained, verifier-checked context with no ability to block or do I/O. So all of the liveness logic lives in a userspace daemon, and the daemon’s only job, as far as the dataplane is concerned, is to keep the BPF maps reflecting reality. It polls the backends, recomputes the Maglev table when the healthy set changes, and writes the maps. Figure 2 shows that split between the userspace control plane and the kernel dataplane.

flowchart LR
    subgraph user[userspace]
        HD[health daemon] -->|poll| BE[(backends)]
        BE -->|status| HD
        HD -->|recompute| MG[Maglev table builder]
    end
    subgraph kern[kernel]
        BL[(backend_lookup map)]
        BP[(backend_pool map)]
        XP[XDP program]
        XP -->|lookup| BL
        XP -->|lookup| BP
    end
    MG -->|atomic update| BL
    HD -->|atomic update| BP

Figure 2. Control plane / dataplane split: the userspace health daemon polls backends and atomically updates the BPF maps; the XDP program reads those maps in-kernel without blocking or I/O.

The reason map updates do not disrupt live traffic is that they are atomic at the element level. Updating a single backend’s state, or replacing the lookup table entry by entry, does not require detaching and reattaching the XDP program, and it does not drop packets in flight. The userspace tooling for this is libbpf from C or Go, or bpftool for one-off operations and debugging. Here is the shape of a map update from a Go control plane using the cilium/ebpf library, marking a backend down:

// be is the eBPF map handle for backend_pool, loaded at startup.
func evictBackend(be *ebpf.Map, idx uint32) error {
    var rec backendRecord
    if err := be.Lookup(idx, &rec); err != nil {
        return fmt.Errorf("lookup backend %d: %w", idx, err)
    }
    rec.State = backendDown
    // Update is atomic per element; the XDP program sees either
    // the old or the new value, never a torn write.
    return be.Update(idx, &rec, ebpf.UpdateExist)
}

For quick inspection without writing a program, bpftool reads and writes the same maps from the shell:

# List loaded maps and find the backend pool by name.
$ bpftool map show name backend_pool
217: array  name backend_pool  flags 0x0
        key 4B  value 12B  max_entries 4096  memlock 65536B

# Dump current entries (key is the backend index).
$ bpftool map dump name backend_pool

# Mark backend index 3 down (state field) by writing the value bytes.
$ bpftool map update name backend_pool key 3 0 0 0 value <12 bytes>

The gap between a backend failing and the daemon evicting it from the map is the blast radius of this design, and it is worth naming precisely. During that window, packets for flows that hash to the dead backend are dropped or reset, because the dataplane still believes the backend is healthy. Shrinking the window means polling faster, which costs control-plane CPU and adds backend-side probe load, so there is a real tradeoff rather than a free win. A common pattern is a hybrid interval: probe backends flagged as degraded aggressively, say every 500 ms, and probe healthy backends more lazily, say every 5 s, weighting toward whichever backends have shown recent errors. That keeps probe overhead low in the common case while tightening the eviction window exactly where failures are most likely.

Where XDP sits in a real edge stack

The single most important architectural fact about an XDP load balancer is what it does not do. It is layer 4. It steers packets by address and port. It does not terminate TLS, it does not parse HTTP, it cannot route on a path or a header or a cookie, and it cannot do an application-layer health check that asks a backend for a 200 OK on /healthz. Those are all real and common requirements, and the answer is not to bolt them onto XDP. The answer is to put XDP where its strengths are and let a layer 7 proxy do the layer 7 work. Figure 3 shows the resulting two-layer topology.

flowchart LR
    Internet((internet)) --> XDP[XDP L4 LB<br/>per edge node]
    XDP --> E1[Envoy / NGINX<br/>L7 proxy]
    XDP --> E2[Envoy / NGINX<br/>L7 proxy]
    E1 --> S1[service A]
    E1 --> S2[service B]
    E2 --> S1
    E2 --> S2

Figure 3. Two-layer edge topology: XDP handles L4 distribution and early packet drop at line rate; L7 proxies terminate TLS, route by path and header, and do application-aware health checks.

In this topology XDP is the front door. It absorbs the raw packet rate, including volumetric DDoS that you want dropped as early and as cheaply as possible, and it spreads connections across a fleet of layer 7 proxies by consistent hashing. The proxies terminate TLS, speak HTTP, and do the path-based and header-based routing that applications actually need. Each layer does what it is good at: XDP gives you cheap, line-rate L4 distribution and a place to drop garbage before it costs you anything, and the proxies give you the rich L7 behavior that no amount of eBPF cleverness will turn an XDP program into. The boundary is clean precisely because you are not asking either layer to do the other’s job.

It helps to place XDP load balancing against the alternatives honestly, because for many teams one of the alternatives is the right call.

Against IPVS and netfilter, the in-kernel L4 load balancing that has been in Linux for years and underpins, for example, kube-proxy in its IPVS mode, XDP’s edge is purely about where in the receive path the work happens. IPVS is mature, well understood, and entirely adequate for a great many workloads. XDP wins when you are packet-rate bound and the savings from acting before SKB allocation actually show up in your profiles. If you are not packet-rate bound, IPVS gives you most of the benefit of in-kernel L4 with none of the eBPF operational learning curve.

Against userspace proxies, Envoy and HAProxy, the comparison is not really apples to apples, and that is the point. Those proxies operate at layer 7 and give you TLS, HTTP routing, retries, circuit breaking, and rich observability. XDP gives you none of that. The right framing is not “XDP versus Envoy” but “XDP in front of Envoy,” as the topology above shows. You reach for XDP when the L4 distribution layer itself has become the bottleneck, not as a replacement for the L7 features your application depends on.

On the hashing choice, Maglev is one option among a few. Classic ring (consistent) hashing places backends on a hash ring and walks clockwise to the next one, which is simple and well known but can distribute unevenly without a lot of virtual nodes, and rebalancing touches more flows than Maglev does. Rendezvous (highest-random-weight) hashing computes a hash per (key, backend) pair and picks the maximum, which gives excellent disruption properties and no precomputed table, at the cost of being O(number of backends) per lookup rather than O(1). Maglev’s bet is to spend memory on a precomputed lookup table in exchange for an O(1) per-packet lookup and bounded disruption when the backend set changes, which is exactly the right bet for a dataplane that has to make a decision on every single packet at line rate. For a control plane that looks up rarely, rendezvous hashing’s simplicity may win.

Where it breaks

XDP load balancing is a sharp tool, and the failure modes are as specific as the benefits.

It is layer 4, full stop. If your requirements include TLS termination, HTTP routing, or application-aware health checks, XDP does not do them and should not be made to. Reaching for clever workarounds here is how you end up with an unmaintainable program that the verifier barely accepts and that nobody on the team can debug. Use a proxy for L7.

BPF map memory pressure is silent until it isn’t. The conntrack and lookup maps are locked, unswappable kernel memory. Undersize the conntrack map and new flows fail closed under load, which looks like random connection failures rather than an obvious out-of-memory event. The fix is discipline, not cleverness: size to peak plus headroom and alarm on map-full conditions.

The health-check eviction gap is intrinsic. Because liveness lives in a userspace daemon, there is always a window between a backend dying and the dataplane learning about it, during which flows to that backend are lost. You can shrink the window with faster probing but you cannot close it, and faster probing costs CPU and backend load. Design for the gap rather than pretending it away.

Observability and debugging are genuinely harder. A packet that an XDP program drops never appears in tcpdump’s usual view, never hits conntrack, and never shows up in the netfilter counters operators reflexively reach for. You debug with bpftool, BPF tracepoints, perf events, and the program’s own statistics maps, which is a different and less familiar toolkit. Budget for the learning curve, and instrument the program with counters from day one so that “where did the packet go” has an answer.

Below a few hundred thousand packets per second per core, it is usually not worth it. The whole value proposition is amortizing away stack-traversal cost at high packet rates. Roughly speaking, below the 500 Kpps-per-core neighborhood a well-tuned userspace proxy or IPVS will serve you fine and will be far simpler to operate. These figures are illustrative rules of thumb, not a benchmark, and your own profile is the only number that matters. Do not adopt eBPF complexity for a workload that does not have the packet rate to pay it back.

Kernel version and verifier constraints are real. XDP and the eBPF features a real LB relies on, certain map types, bpf_redirect_map, native driver XDP support, have minimum kernel versions and, for the best performance, depend on the NIC driver implementing native XDP rather than the slower generic path. The verifier also bounds what your program can express, and a refactor that a compiler accepts can still be rejected at load time. Pin your kernel and driver expectations explicitly and treat the verifier as part of your build.

Further reading

Start with the kernel’s own BPF and XDP documentation [1] and the libbpf project [2], which together cover the program model, the map types, and the loader you will actually use. For a production reference architecture, Katran’s repository and design notes [3] show how Meta assembled the pieces this article sketches, and Cilium’s eBPF datapath documentation [4] shows the same ideas inside a Kubernetes networking layer. The Maglev paper [5] is the primary source for the hashing scheme and is worth reading for the table-construction algorithm alone. For the broader eBPF landscape, the ebpf.io overview [6] and the BPF and XDP reference guide [7] are good orientation. The bpftool documentation [8] covers map inspection and the debugging workflow, the AF_XDP and XDP action documentation in the kernel tree [9] covers redirect targets in detail, and for the userspace control plane in Go the cilium/ebpf library [10] is the most widely used option. For positioning against the incumbent in-kernel approach, the IPVS documentation [11] is the right comparison point.

Written with AI assistance, editorially reviewed.

References

[1] The Linux Kernel, “BPF Documentation.” https://docs.kernel.org/bpf/

[2] libbpf, “libbpf: eBPF library.” https://github.com/libbpf/libbpf

[3] Meta, “Katran: A high performance layer 4 load balancer.” https://github.com/facebookincubator/katran

[4] Cilium, “eBPF Datapath.” https://docs.cilium.io/en/stable/network/ebpf/

[5] D. E. Eisenbud et al., “Maglev: A Fast and Reliable Software Network Load Balancer,” USENIX NSDI, 2016. https://www.usenix.org/conference/nsdi16/technical-sessions/presentation/eisenbud

[6] eBPF, “What is eBPF? An Introduction and Deep Dive.” https://ebpf.io/what-is-ebpf/

[7] Cilium, “BPF and XDP Reference Guide.” https://docs.cilium.io/en/stable/reference-guides/bpf/

[8] The Linux Kernel, “bpftool documentation.” https://docs.kernel.org/bpf/bpftool.html

[9] The Linux Kernel, “AF_XDP.” https://docs.kernel.org/networking/af_xdp.html

[10] Cilium, “cilium/ebpf: eBPF library for Go.” https://github.com/cilium/ebpf

[11] The Linux Virtual Server Project, “IPVS Software.” http://www.linuxvirtualserver.org/software/ipvs.html

[12] IO Visor Project, “XDP (eXpress Data Path).” https://www.iovisor.org/technology/xdp