ACE Journal

Branch prediction microarchitecture after Spectre mitigations

Who this is for: systems engineers, performance engineers, and architects working on Linux kernel internals, JVM tuning, hypervisor design, or latency-sensitive server applications. You should be comfortable reading x86 assembly and running perf stat. Familiarity with the basic Spectre narrative is assumed; this article is about what changed in the microarchitecture and what overhead remains.

The profile that broke in 2018

You are running a JVM-based service. Before January 2018, it is saturating its cores at around 85 percent, call-site prediction is near-perfect for the virtual dispatch paths the JIT has warmed, and p99 latency is stable. After the kernel patches land, the same binary on the same hardware runs noticeably slower. Not catastrophically, but measurably: somewhere between 5 and 20 percent throughput regression depending on how much of the workload flows through indirect branches and system calls.

The root cause is not a software bug you can fix by recompiling. It is a change to how the CPU’s indirect branch predictor is allowed to behave, forced by the discovery that the predictor’s shared state is exploitable as a side channel. Understanding what was changed, why, and what the microarchitecture now looks like is how you reason about what the cost actually is and where to look for it.

Mechanics: predictor structures and how they speculate

A modern out-of-order CPU does not wait to know where a branch goes before fetching instructions from the target. It guesses, fetches speculatively, executes speculatively, and squashes the work if the guess was wrong. That squash costs roughly 15-20 cycles on a typical x86 microarchitecture for a mispredicted branch. The structures doing the guessing are the following.

Branch target buffer (BTB). The BTB is an address-keyed cache. Given the program counter of an indirect branch (a call [rax], a virtual dispatch, a PLT trampoline), it returns the target address that was last observed at that PC. It is essentially a direct-mapped or set-associative table indexed by the low bits of the PC. Aliasing is a known quality issue: two different indirect branches that map to the same BTB set evict each other’s entries and cause mispredictions.

Return address stack (RAS). The RAS is a small hardware stack, typically 16-64 entries deep, that tracks the return addresses pushed by call instructions. On a ret, the CPU pops the top of the RAS rather than reading from memory or waiting to decode. It is extremely accurate for normal call stacks but wraps around on deep recursion or unusual calling conventions.

Direction predictors (TAGE and perceptron). For conditional branches (taken or not taken), modern predictors use tabular history methods. TAGE (tagged geometric history length predictor) maintains multiple tables of counters indexed by the branch PC XORed with different lengths of the global history shift register. Perceptron-based predictors use a similar multi-history-length structure but compute a weighted sum. Both achieve very high accuracy on workloads with regular pattern structure.

The speculation pipeline looks like the following (Figure 1 shows the fetch-to-execute flow):

flowchart LR
    A[Fetch PC] --> B{BTB / RAS hit?}
    B -->|yes| C[Speculative fetch at predicted target]
    B -->|no| D[Stall until decode resolves]
    C --> E[Speculative execute]
    E --> F{Actual target matches?}
    F -->|yes| G[Commit]
    F -->|no| H[Squash pipeline\npenalty ~15-20 cycles]

Figure 1. Speculative fetch and execute path for an indirect branch. The critical path is the BTB lookup latency: a miss or stall here is a pipeline bubble even before a misprediction is confirmed.

Spectre variant 2 (branch target injection, CVE-2017-5715) exploits the BTB specifically. An attacker running in user space can execute a carefully crafted sequence of indirect branches that trains the BTB to associate a chosen prediction with a branch PC that happens to be used by the kernel. When the kernel later executes that branch speculatively, it fetches and transiently executes instructions at the attacker-chosen address, reading secrets into microarchitectural state (caches, execution ports) that the attacker can then read via a timing side channel, even though the architectural execution was squashed.

Worked example: measuring misprediction cost

The following C snippet creates a pointer call through a function-pointer array that is difficult to predict because the call-site PC is shared by multiple callee targets. It is a stripped-down analog of the virtual dispatch patterns in JVMs and database engines.

#include <stdlib.h>

typedef void (*fn_t)(void);
static void nop_a(void) {}
static void nop_b(void) {}

void run_indirect(int n, fn_t *table, int *indices) {
    for (int i = 0; i < n; i++) {
        table[indices[i]]();   /* indirect call; BTB must predict target */
    }
}

Compile with gcc -O2 -o bench bench.c and measure with:

# r04c0: retired indirect calls (Skylake/Cascade Lake; event codes differ on other microarchitectures)
# r00c5: indirect branch mispredictions
perf stat -e branches,branch-misses,\
cycles,instructions,\
r04c0,r00c5 \
-- ./bench

The relevant counter on modern Intel is INDIRECT_BRANCH_MISPREDICTIONS (event 0xC5, umask 0x00). On AMD it surfaces as ExRetIndirectBranchMispred (0x00D1). A high ratio of mispredictions to indirect calls on a polymorphic call site indicates that the BTB cannot hold stable predictions for all the targets, either because of aliasing or because the prediction history encodes a pattern longer than the predictor can track.

The assembly at the call site (Intel syntax) looks like this:

; rax = table + indices[i] * 8
mov     eax, DWORD PTR [rcx+rsi*4]   ; load index
lea     rax, [rbx+rax*8]              ; compute table[index] address
call    QWORD PTR [rax]               ; indirect call - BTB target lookup

Without mitigations the CPU speculates at the BTB-predicted target on the call [rax]. With retpoline in effect, this call site is rewritten by the compiler into a trampoline that never actually speculates at the predicted target (the speculation is redirected to a pause-and-loop sequence). The BTB lookup still happens, but the speculated path is harmless. The cost is that the real call target is loaded from memory and a return is used to reach it, adding latency, defeating the BTB’s purpose, and increasing branch misprediction rates.

The hard part: how Spectre mitigations interact with predictor state

The mitigations deployed since 2018 fall into three categories, and each interacts with the predictor structures differently (Figure 2 maps the mitigations to the structures they protect).

graph TD
    subgraph "Predictor structures"
        BTB[BTB - indirect target]
        RAS[RAS - return address]
        DIR[Direction tables - TAGE/perceptron]
    end
    subgraph "Mitigations"
        RETPOLINE[retpoline]
        IBRS[IBRS / eIBRS]
        IBPB[IBPB]
        STIBP[STIBP]
        BTI[Arm BTI]
    end
    RETPOLINE -->|replaces indirect calls| BTB
    IBRS -->|restricts guest-to-host speculation| BTB
    eIBRS -->|privilege-tag isolation| BTB
    IBPB -->|flushes all indirect prediction state| BTB
    IBPB -->|flushes| RAS
    STIBP -->|blocks sibling HT thread from poisoning| BTB
    BTI -->|marks valid indirect targets| BTB

Figure 2. Mapping of Spectre variant 2 mitigations to the predictor structures they protect. IBPB is the only mechanism that flushes both BTB and RAS state; the others are isolation or replacement strategies.

Retpoline (return-trampoline). Retpoline is a compiler transform, enabled by -mindirect-branch=thunk in GCC and Clang, that replaces every indirect branch with a sequence that redirects speculation to an infinite pause loop via a misspeculated ret. The key insight is that a ret speculatively targets the RAS top, which the attacker cannot control for a user-kernel attack. Retpoline eliminates BTB-based speculation entirely for the protected call sites. The cost is measurable: every indirect call becomes a call-to-thunk-plus-ret, adding several cycles of fixed overhead and increasing branch-misprediction events because the ret at the end mispredicts on the first call to a new callee.

IBRS (indirect branch restricted speculation). IBRS is a model-specific register (MSR) write that tells the CPU to restrict speculation so that indirect branches in less-privileged execution cannot influence more-privileged speculation. Early IBRS required writing the MSR on every kernel entry and exit. On the first-generation silicon (Skylake, Broadwell) this was expensive: the MSR write itself took hundreds of cycles and the performance regression for syscall-heavy workloads was severe enough that some kernels defaulted to retpoline instead.

eIBRS (enhanced IBRS). Intel’s answer to the MSR-write overhead was eIBRS, available in microcode updates starting around 2018 and baked into Alder Lake (Golden Cove, 2021) in hardware. eIBRS works by tagging BTB entries with the privilege level at which they were written. A user-mode entry cannot match against a kernel-mode BTB lookup, so the isolation is persistent and the per-syscall MSR write is no longer needed. The performance cost of eIBRS relative to unmitigated hardware is low on current silicon (a few percent on typical workloads), because the predictor table size was also increased to compensate for the effective halving of capacity between privilege domains.

IBPB (indirect branch predictor barrier). IBPB flushes all BTB and RAS state on the core. It is the nuclear option: after an IBPB, no previously written predictor entry is visible. IBPB is issued on process context switches between mutually untrusted processes (when prctl(PR_SET_SPECULATION_CTRL, ...) with PR_SPEC_DISABLE is active, or when the scheduler determines the outgoing and incoming processes are in different security domains). On current x86 microarchitectures, IBPB takes hundreds of cycles. A system doing thousands of context switches per second on a busy server with strict process isolation can see IBPB as a meaningful fraction of scheduler overhead.

STIBP (single thread indirect branch predictors). On hyperthreaded cores, two logical CPUs share the BTB hardware. STIBP restricts that sharing, preventing a sibling hyperthread from poisoning the BTB of its partner. The cost is reduced predictor sharing between the two logical cores, which slightly increases BTB miss rates for workloads where the two threads run related code.

The privilege-partitioning introduced by eIBRS has a concrete capacity cost (Figure 3). If a BTB has N physical entries and is split into user-mode and kernel-mode domains, each domain has access to roughly N/2 entries before aliasing begins. For a JVM with a large polymorphic call graph, fewer effective BTB entries means more aliasing and more mispredictions. Post-2019 microarchitectures compensate by building larger BTBs (Intel’s Golden Cove core has a substantially larger BTB than Skylake), but “larger to compensate for partitioning” is not the same as “back to where you were.”

graph LR
    subgraph "Pre-Spectre: single shared BTB pool"
        P1[All N entries available\nto any privilege level]
    end
    subgraph "Post-eIBRS: privilege-tagged BTB"
        K[~N/2 entries tagged\nkernel mode]
        U[~N/2 entries tagged\nuser mode]
    end
    P1 -. partition .-> K
    P1 -. partition .-> U

Figure 3. BTB capacity before and after eIBRS privilege tagging. The physical table grows in post-2019 silicon to compensate, but workloads with large indirect call graphs still see reduced effective capacity in each domain.

In virtualized environments the situation is worse. A VMM-to-guest boundary introduces a third domain (hypervisor, guest-kernel, guest-user), which means two privilege-level splits rather than one. VM exits are already expensive because they involve a mode transition, MSR saves and restores, and TLB management. Post-Spectre, they also include predictor state management (either IBPB or equivalent per-exit flushing), adding to already-high exit costs in systems with nested page tables (Intel VT-x + EPT, AMD-V + NPT).

Integration and day-to-day: measuring and choosing mitigations

The Linux kernel exposes the active mitigation state for each Spectre variant via /sys/devices/system/cpu/vulnerabilities/. On a modern Intel system with eIBRS and microcode updates applied, you typically see:

$ cat /sys/devices/system/cpu/vulnerabilities/spectre_v2
Mitigation: Enhanced / Automatic IBRS; IBPB: conditional; RSB filling; PBRSB-eIBRS: SW sequence

The conditional IBPB qualifier means the kernel is not issuing IBPB on every context switch, only on transitions where it determines isolation is required. You can adjust this policy with boot parameters:

# Disable IBPB on kernel entry (reduces overhead, accepts residual risk):
spectre_v2_user=off

# Force IBPB on every context switch (maximum isolation, higher overhead):
spectre_v2_user=on

# Full mitigation disable (testing / benchmarking only):
mitigations=off

To measure how much IBPB is actually costing you on a running system, use perf to count IBPB-related events and context-switch rates together:

# r00ae: IBPB-triggered flushes on Intel Skylake/Cascade Lake (MSR write count, platform-specific; event codes differ on other microarchitectures)
perf stat -e context-switches,\
cpu-migrations,\
r00ae,\
branch-misses,\
indirect_branch_mispredictions \
-a sleep 10

A high ratio of branch misses to total branches, combined with a high context-switch rate, is the signature of an IBPB-dominated overhead profile. The actionable intervention is to reduce unnecessary cross-process isolation. If all the processes on a host are owned by the same security principal, spectre_v2_user=prctl lets individual processes opt into IBPB protection without imposing it universally.

For kernel builds, retpoline remains the default on x86 when eIBRS is not available or not trusted. Building with CONFIG_RETPOLINE=y (the default for distro kernels since mid-2018) instruments all indirect branches in the kernel image. The compiler annotation __noretpoline allows individual hot paths to opt out when the call target is guaranteed safe; the scheduler’s call * to the idle thread is a canonical example.

The Spectre mitigations landscape in early 2026 covers hardware, firmware, and software layers, and the tradeoffs are different depending on which generation of silicon you are running.

Hardware redesigns (eIBRS, Zen 4 predictor isolation). The cleanest fix is microarchitectural isolation that removes the shared-state side channel without software involvement. Intel’s eIBRS, available in microcode for Skylake-generation and later, provides privilege-level tagging. AMD’s Zen 4 (Genoa, Raphael, 2022) introduced similar isolation in the indirect branch predictor hardware. For new deployments on current silicon, the software overhead of early IBRS is mostly gone, but the capacity-partitioning tax remains.

Retpoline as compiler mitigation. Retpoline is still the right answer for kernels running on pre-eIBRS hardware and for compiling user-space code that handles untrusted inputs and runs at elevated privilege (hypervisors, container runtimes). Its overhead is best characterized as “a fixed per-indirect-call surcharge” rather than a global penalty, and workloads with few indirect branches see little impact.

Arm’s BTI and context-ID tagging. On Arm, Branch Target Identification (BTI, introduced with ARMv8.5) works differently. It marks valid indirect branch targets at the instruction encoding level using bti instructions in the prologue of each callable function. The CPU raises a fault if speculation lands on an unmarked address. BTI mitigates ROP-style exploitation but is not a full Spectre variant 2 fix by itself. Arm’s Cortex-X4 and Neoverse N2 implement context-ID tagging in the return address stack and indirect branch predictor as the primary Spectre-v2 mitigation, providing isolation without the MSR-write approach of x86.

BHI (branch history injection). BHI (CVE-2022-0001/CVE-2022-0002, disclosed March 2022) extends the Spectre variant 2 threat model to the branch history register (BHR), the global history used by direction predictors. An attacker can manipulate the BHR to influence which aliased predictor entry is used, bypassing eIBRS’s privilege-level tag in some configurations. Intel’s response is a software mitigation (BHI_DIS_S, available as a microcode patch and kernel boot parameter) and a forthcoming hardware fix. This shows that the predictor-as-attack-surface story is not fully closed even on current hardware.

Where it breaks

High-frequency cross-process scheduling. Any system that runs many mutually untrusted processes on the same core and relies on IBPB for isolation will pay the flush cost on every context switch. Serverless compute platforms running customer functions, container orchestrators with strict multi-tenant isolation, and CI systems running arbitrary code on shared hardware all fall into this category. The overhead is not abstract: on a core doing thousands of context switches per second, IBPB flushing accumulates to a visible fraction of available cycles.

Interpreters and JITs with large polymorphic call graphs. CPython’s eval loop, the JVM’s megamorphic call sites, and database engines that dispatch query operators through function pointers all exercise the BTB intensively. With privilege-partitioning in effect, the effective BTB capacity for each execution domain is smaller. On Skylake-era hardware still deployed in many data centers, this is still a regression relative to pre-mitigation behavior.

Hypervisors with high VM-exit rates. A VMM that takes frequent VM exits (for device emulation, hypercall handling, or nested virtualization) pays predictor management overhead on each exit. The cost depends on whether the kernel is issuing a full IBPB or relying on eIBRS isolation. For workloads that are exit-heavy by nature (emulated I/O, MMIO trapping), the predictor overhead is a fixed tax on every exit and is difficult to amortize.

BHI on eIBRS-only deployments. Systems relying on eIBRS without the BHI mitigation applied remain theoretically vulnerable to branch history injection attacks that bypass the privilege-level tag. The BHI_DIS_S mitigation carries its own (smaller) overhead. Systems that disabled retpoline specifically because eIBRS was available may need to revisit their mitigation stack.

Heterogeneous core topologies. On processors with asymmetric core designs (efficiency cores and performance cores on the same die, as in Intel’s Alder Lake and later), the BTB and RAS implementations may differ between core types. Workloads that migrate between P-cores and E-cores may encounter different misprediction rates and different effective mitigation behaviors depending on which scheduler binds them. The Linux kernel handles this correctly for the documented mitigation flags, but performance profiling must account for the core type when comparing results.

Mitigation state invisibility. It is common on long-lived deployments to find that the effective mitigation state has drifted from what was intended: a microcode update that did not apply cleanly, a kernel upgrade that changed defaults, a container runtime that set process isolation flags differently than expected. The /sys/devices/system/cpu/vulnerabilities/ interface shows the system’s view; spectre-meltdown-checker (a third-party shell script) provides a more detailed audit. Neither is a substitute for measuring actual performance with perf counters on a representative workload.

Further reading

Start with the original Spectre paper [1] for the threat model, then the Google Project Zero blog post on Spectre variant 2 [2] for the attack mechanics. Intel’s deep dive on IBRS and retpoline [3] explains the MSR semantics and the original retpoline construction. The retpoline whitepaper by Paul Turner at Google [4] is the definitive reference on the compiler transform. For eIBRS, read Intel’s Security Advisory INTEL-SA-00074 and its updates [5]. The BHI disclosure and Intel’s mitigation guidance [6] covers the post-eIBRS extension to the threat model. For Arm-specific mitigations including BTI and context-ID tagging, Arm’s Cortex-A series software mitigation guide [7] is the starting point. The Linux kernel documentation on Spectre mitigations [8] covers the boot parameters and their tradeoffs. Brendan Gregg’s perf CPU performance analysis guide [9] is the practical reference for measuring branch mispredictions and scheduler overhead. For predictor microarchitecture (TAGE and perceptron predictors), the original TAGE paper by Seznec and Michaud [10] and the PPM-based predictor history by Jimenez and Lin [11] provide the algorithmic foundations. For a current Intel microarchitecture reference on Golden Cove’s predictor, the Intel Optimization Reference Manual [12] covers the high-level structures.

Written with AI assistance, editorially reviewed.

References

[1] P. Kocher et al., “Spectre attacks: exploiting speculative execution,” in Proc. IEEE S&P 2019, pp. 1-19. https://spectreattack.com/spectre.pdf

[2] J. Horn, “Reading privileged memory with a side-channel,” Google Project Zero, Jan. 2018. https://googleprojectzero.blogspot.com/2018/01/reading-privileged-memory-with-side.html

[3] Intel Corporation, “Retpoline: a branch target injection mitigation,” White Paper, Jan. 2018. https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/technical-documentation/retpoline-a-branch-target-injection-mitigation.html

[4] P. Turner, “Retpoline: a software construct for preventing branch-target-injection,” Google, Jan. 2018. https://support.google.com/faqs/answer/7625886

[5] Intel Corporation, “Intel Security Advisory INTEL-SA-00088 (Spectre variant 2 and related updates).” https://www.intel.com/content/www/us/en/security-center/advisory/intel-sa-00088.html

[6] Intel Corporation, “Branch History Injection and Intra-mode Branch Target Injection / CVE-2022-0001, CVE-2022-0002,” Mar. 2022. https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/advisory-guidance/branch-history-injection.html

[7] Arm Limited, “Speculation barriers,” Arm Developer documentation. https://developer.arm.com/documentation/102816/0100/Speculation-barriers

[8] Linux Kernel Documentation, “Spectre side channels.” https://www.kernel.org/doc/html/latest/admin-guide/hw-vuln/spectre.html

[9] B. Gregg, “Linux perf Examples.” https://www.brendangregg.com/perf.html

[10] A. Seznec and P. Michaud, “A case for (partially) TAgged GEometric history length branch prediction,” J. Instruction-Level Parallelism, vol. 8, 2006. https://jilp.org/vol8/v8paper1.pdf

[11] D. A. Jimenez and C. Lin, “Dynamic branch prediction with perceptrons,” in Proc. HPCA 2001, pp. 197-206. https://doi.org/10.1109/HPCA.2001.903263

[12] Intel Corporation, “Intel 64 and IA-32 Architectures Optimization Reference Manual,” 2023. https://www.intel.com/content/www/us/en/developer/articles/technical/intel-sdm.html