Abstract

Mechanistic interpretability seeks to reverse-engineer neural network computations into human-readable algorithms: identifying which components implement which sub-functions, how information is routed through the residual stream, and what abstractions individual neurons or heads represent. Transformer attention heads are a natural unit of analysis because their weight matrices are interpretable as low-rank linear maps, and their outputs are readable as position-weighted combinations of value vectors. Work from Anthropic, Redwood Research, and DeepMind over the past three years has moved from proof-of-concept circuit analyses in toy models to partial circuits in models with billions of parameters, though significant gaps remain between what can be characterized in small models and what scales.

Attention Heads as Functional Units

Early mechanistic work decomposed GPT-2 small into circuits: subgraphs of attention heads and MLP layers that together implement specific input-output behaviors on narrow tasks such as indirect object identification (Wang et al., 2022) and greater-than comparison (Hanna et al., 2023). Heads were found to specialize into coarse functional categories - induction heads that copy from earlier context, duplicate token heads that suppress repeated information, and name mover heads that move entity representations to output positions. These categories are not imposed by architecture but emerge from training. The analysis method, activation patching (also called causal tracing), measures how much a component contributes to a specific output by replacing its activations with those from a clean forward pass and observing the change in logit difference.

Scaling to Larger Models

Circuit analysis in GPT-2 small (117M parameters) benefited from the model being small enough to enumerate heads systematically. In models with tens of billions of parameters, exhaustive enumeration is infeasible. Automated circuit discovery tools such as ACDC (Conmy et al.) prune the computational graph by iteratively ablating edges that do not contribute above a threshold, yielding sparser circuits without full enumeration. Neel Nanda’s TransformerLens library has become a standard research infrastructure layer for activation hooking and patching in PyTorch, lowering the barrier for running mechanistic analyses on open-weight models including Llama and Mistral families. Sparse dictionary learning applied to MLP activations (Bricken et al., Anthropic) reveals that individual neurons are often polysemantic - responding to multiple unrelated features - while sparse basis directions in learned dictionaries correspond to more monosemantic concepts. This extends the circuit picture to MLP layers, which had been treated as largely opaque relative to attention.

Interpretability and Alignment

The longer-term motivation for mechanistic interpretability is alignment: if we can characterize what circuits implement, we can look for circuits that correspond to deceptive or dangerous behaviors before they manifest at inference time. Current work remains far from this goal in practice. The gap between characterizing narrow circuits in controlled prompts and auditing a general-purpose model for safety-relevant behaviors is substantial. However, intermediate progress - understanding how factual retrieval, in-context learning, and chain-of-thought steps are implemented mechanistically - contributes to building a more structured understanding of what large models actually compute, rather than treating them purely as black boxes to be probed with behavioral tests.

Open Challenges

Attribution of behaviors to circuits is complicated by superposition, where the model encodes more features than it has dimensions by using near-orthogonal directions. Interventions designed to test one circuit can inadvertently disrupt superposed features that share the same activations. Additionally, circuits identified for one prompt distribution often fail to generalize to slightly different phrasings of the same task, suggesting that what appears to be a general-purpose circuit may be a narrow pattern specific to training distribution statistics.