Abstract
The systolic array, revived by Google’s TPUv1 in 2016, has become the default compute structure for matrix-multiply-heavy AI inference workloads. But the original TPU design made specific dataflow choices, weight-stationary with weights preloaded into the array and activations streaming through, that suited large-batch datacenter inference. Edge accelerators operate under fundamentally different constraints: low batch sizes (often one), tight power envelopes under 5 W, and irregular network shapes from quantized and pruned models. These constraints have pushed edge silicon designers toward alternative dataflow strategies and hybrid architectures, and the silicon shipping in 2024-2025 reflects several distinct bets on which direction is correct.
Dataflow Taxonomy Refresher
The three canonical dataflows for systolic arrays are weight-stationary (WS), output-stationary (OS), and row-stationary (RS). In WS, filter weights remain fixed in the array’s register files and input activations pass through; this minimizes weight reads from SRAM but requires large arrays to amortize weight load latency, favoring large batch sizes. In OS, each processing element accumulates a complete output activation before writing it out, minimizing writes to global buffers at the cost of re-reading inputs. RS, used in MIT’s Eyeriss architecture, assigns each PE a row of a filter and streams inputs diagonally, balancing reads across all three data types. For edge batch-1 inference, WS performs poorly because the weight load overhead dominates with no batching to amortize it. This has driven commercial designs toward OS and RS variants.
Commercial Designs in 2024-2025
Apple’s Neural Engine in the M-series and A-series chips uses an architecture that Apple has not disclosed in detail but that third-party analyses, including the work from Anandtech’s architecture teardowns and academic reverse-engineering efforts, suggest is closest to an output-stationary variant with heavy use of activation compression to reduce SRAM bandwidth. Qualcomm’s Hexagon NPU in the Snapdragon 8 Elite, announced in late 2024, exposes a vector-VLIW front-end feeding a systolic back-end, allowing the compiler to mix spatial and temporal tiling strategies depending on layer shape. MediaTek’s APU 790 in the Dimensity 9400 takes a different approach: a smaller systolic array augmented by a separate sparse-MAC unit that handles the non-zero activations from ReLU-pruned layers without wasting cycles on zero multiplications.
The Sparse Activation Problem
The common thread across these designs is handling sparsity. Quantized INT4 and INT8 models running on-device after magnitude pruning often have 40-60% activation sparsity on convolutional layers. A standard dense systolic array still computes those multiplications and discards the result. The sparse-MAC units in designs like MediaTek’s and in Arm’s Ethos-U85 use a combination of bitmask-driven skipping and compressed activation storage formats. The challenge is that the skipping logic adds area and the latency benefit is workload-dependent. For transformers with softmax attention, where sparsity is less predictable than in ReLU networks, the sparse path often falls back to the dense array anyway.
Compiler-Hardware Co-Design
The shift in edge dataflow has raised the importance of tile-size scheduling in compilers. TensorFlow Lite’s XNNPack backend and the Apache TVM relay-to-target pipeline both expose tiling parameters that interact with the specific dataflow of the underlying array. A weight-stationary tiling schedule on an output-stationary hardware array can halve utilization. MLCommons’ MLPerf Inference Edge benchmark suite has become the de facto pressure test for this interaction, and the 2024 edge results show a 3-4x spread in performance-per-watt among devices with similar peak TOPS ratings, primarily attributable to compiler-hardware dataflow alignment.