Abstract

Neural Architecture Search (NAS) has long been criticized for its computational extravagance: early approaches required thousands of GPU-hours to evaluate candidate architectures. Hardware-aware NAS addresses a different problem. It steers the search process not just toward high accuracy but toward architectures that are efficient on a specific target hardware device. By mid-2025, latency proxy methods, which predict hardware performance from architectural features without fully training or even executing each candidate, have become the dominant mechanism for making hardware-aware NAS tractable. This article covers the design of effective latency proxies, the mismatch between proxy rankings and measured hardware performance, and current approaches that address this mismatch.

Latency Proxies and Their Limitations

A latency proxy maps an architecture description to a predicted runtime on a target device, without requiring the architecture to be compiled, deployed, and benchmarked. The simplest proxies are lookup-table based: operator latencies for common layer types (convolution, attention, depthwise separable convolution) are measured once on the target hardware, and total predicted latency is the sum of per-operator estimates. This approach is cheap and generalizes across architectures sharing the same operator vocabulary.

Lookup-table proxies fail in several well-documented ways. They ignore memory access patterns that depend on the sequence of operations rather than individual operators. On hardware with limited on-chip memory (mobile NPUs, DSPs), operations that fit in cache when run in isolation may cause expensive DRAM traffic when chained in a specific order. Additionally, operator fusion, where a backend compiler merges adjacent operations into a single kernel, can change realized latency by 2-5x compared to the sum-of-parts estimate. Architectures that benefit heavily from fusion look misleadingly expensive under lookup-table proxies.

Learning-Based and Compiler-Aware Proxies

More sophisticated proxies learn a regression model from a sample of (architecture, measured latency) pairs. Microsoft’s NAS tools (incorporated into ONNX Runtime’s optimization stack) and Google’s Vizier platform both support learned latency predictors that take operator-level features as input. These predictors generalize to unseen architectures within the search space, with error rates in the 10-20% range for architectures that remain within the distribution of the training sample.

A more principled approach couples the proxy to the compiler backend directly. Instead of approximating latency from architecture features, the proxy runs the compiler’s cost model, a lower-fidelity but analytically grounded estimate of compute and memory cost that the compiler itself uses for optimization decisions. This approach is tightly tied to a specific compiler (TVM, XLA, or MLIR-based backends) but is more robust to fusion and memory layout effects. Teams at OctoML and in the TVM community have demonstrated that compiler-cost-model proxies outperform lookup-table proxies on devices where fusion coverage is high.

Multi-Device and Pareto-Front Search

A practical complication is that most production pipelines target multiple hardware backends simultaneously, a single model may need to run on server GPUs, mobile devices, and edge accelerators. Optimizing for a single device’s proxy can produce architectures that are inefficient on others. Multi-objective NAS, which constructs a Pareto front across accuracy and latency on multiple devices, is computationally heavier but avoids this trap. Once-for-All (OFA) networks from MIT CSAIL address this by training a single supernetwork from which device-specific subnetworks can be extracted without retraining, with device-specific latency proxies used to select the extraction configuration.

Remaining Challenges

Proxy accuracy degrades most severely for novel operators (FlashAttention variants, chunked attention, custom CUDA kernels) that are not covered by the proxy’s training distribution. Hardware evolution also outdates proxies quickly: a proxy trained on an A100 does not transfer to an H100 without recalibration, despite the architectural similarity. Building proxies that are robust to hardware generation changes, or that require minimal recalibration, is an open problem that limits the long-term value of expensive proxy training efforts.