Abstract

NVIDIA’s tensor cores, first introduced in Volta in 2017, have undergone four generations of architectural revision driven almost entirely by the demands of large-scale neural network training. The evolution from FP16-only matrix multiplication in Volta to the Hopper generation’s FP8 with per-tensor and per-block scaling in H100 reflects a broader industry shift: precision is now a first-class architectural variable, chosen per layer and per operator rather than fixed for the entire model. This article traces the microarchitectural changes across generations and examines how competing approaches from AMD and Intel are shaping the mixed-precision landscape as of late 2025.

Volta to Ampere - Building the Foundation

Volta’s tensor cores operated on 4x4x4 matrix multiply-accumulate (MMA) tiles in FP16, with FP32 accumulation. The key insight was that FP16 inputs with FP32 internal accumulation preserved enough numerical range for gradient descent without the overflow problems that pure FP16 accumulation caused in early experiments. Turing added INT8 and INT4 paths for inference, but training remained FP16-dominant. Ampere’s A100 introduced structural float (TF32), which truncates the FP32 mantissa to 10 bits (matching FP16 precision) while retaining the full FP32 exponent range. TF32 training requires no model changes and provides a 10x throughput improvement over FP32 on A100, making it a pragmatic drop-in for many production training pipelines. Ampere also introduced BF16 support, which trades mantissa precision for exponent range - better suited to the large dynamic range of gradient distributions than FP16.

Hopper’s FP8 and the Scaling Problem

The H100’s Transformer Engine, introduced with Hopper in 2022 and widely deployed through 2025, added FP8 in two formats: E4M3 (4 exponent bits, 3 mantissa) for forward pass activations where dynamic range requirements are modest, and E5M2 (5 exponent bits, 2 mantissa) for backward pass gradients where range matters more than precision. The challenge with FP8 is that its representable range is narrow enough that naive per-tensor scaling causes overflow in a meaningful fraction of training steps. NVIDIA’s solution is per-tensor delayed scaling, where the scaling factor for each tensor is computed from the maximum absolute value observed in the prior iteration. This works well for stable training but can produce spikes when loss landscapes shift sharply. Per-block scaling - applying separate scale factors to 16x16 or 32x32 sub-tiles of a weight matrix - is more robust but requires hardware support to track and apply the per-block metadata efficiently. H100’s Transformer Engine supports per-tensor scaling in hardware; per-block scaling is implemented partially in software in the current cuDNN and Transformer Engine library releases, with full hardware support targeted for future generations.

AMD and Intel’s Approaches

AMD’s MI300X, shipping in volume for AI training as of early 2025, implements FP8 in a similar E4M3/E5M2 scheme and adds matrix cores with native BF16 support at higher throughput than the previous MI250X. AMD’s ROCm software stack has gained parity with CUDA’s mixed-precision training libraries for most standard transformer architectures, though niche operators still require porting effort. Intel’s Gaudi 3, released in mid-2025, takes a different tack: rather than exposing raw precision knobs to the programmer, Gaudi 3’s matrix multiplication units select precision dynamically based on hardware-computed estimates of the input tensor’s effective dynamic range. This approach simplifies the programming model but makes the effective compute throughput harder to predict and reason about from the software side. Early benchmarks from MLCommons suggest Gaudi 3 achieves competitive throughput on LLM pre-training workloads, though the training efficiency in token-per-joule terms varies more across workloads than either H100 or MI300X.

Implications for Model Architecture

The availability of cheap FP8 compute is beginning to influence model design rather than just training recipes. Researchers at Meta AI and Google DeepMind have published work in 2025 showing that transformers designed from scratch with FP8-aware weight initialization schemes and careful placement of layer normalization train stably in FP8 for the full run, without the mixed-precision fallback for numerically sensitive layers that current frameworks require. If these techniques generalize, the next generation of large models may assume FP8 as the native precision in the same way the previous generation assumed FP16 - pushing the hardware generation after Hopper to support even lower precision (FP6, FP4) with the same robustness.