Abstract

Catastrophic forgetting, the tendency of a neural network to lose performance on previously learned tasks when trained on new data, remains one of the most persistent unsolved problems in deep learning. For vision models, the problem has practical urgency: production systems routinely need to incorporate new object categories, adapt to domain shift, or absorb annotated data from new sensors without the cost of full retraining. Continual learning (CL) research has produced a broad taxonomy of approaches, including regularization-based methods, architectural expansion, and rehearsal-based replay. By mid-2025, the field has reached rough consensus on where each approach succeeds and where it fails, though no method has achieved satisfactory performance across the full spectrum of CL scenarios. This article surveys the landscape with emphasis on vision-specific findings.

Why Vision Models Forget

Catastrophic forgetting in neural networks traces to the shared parameterization across tasks. When gradient updates for a new task shift weights, the representations that supported prior tasks are overwritten. The severity depends on how much the new task’s gradient landscape overlaps with the old task’s critical weight directions. For vision transformers, attention heads and MLP layers in later blocks carry the most task-specific information; early blocks encode lower-level features that transfer more stably. This layer-specific structure suggests that freezing or regularizing later layers more aggressively while allowing early layers to adapt is a natural mitigation, and empirically it outperforms uniform regularization strategies.

Regularization-Based Approaches

Elastic Weight Consolidation (EWC) and its successors compute a measure of weight importance (approximated via the Fisher information matrix) for each task and add a penalty term that resists large changes to important weights during subsequent training. For small task sequences (fewer than 10 tasks), EWC-family methods provide effective forgetting control at low memory cost. For longer task sequences, the approximation quality of the Fisher degrades and the penalty terms can become contradictory across many tasks, leading to plasticity collapse, where the model becomes so constrained that it cannot learn new tasks well either. Synaptic Intelligence (SI) and Memory-Aware Synapses (MAS) are variants that compute importance online rather than per-task but share the same long-sequence fragility.

Rehearsal and Generative Replay

Rehearsal methods store a small buffer of past data and interleave it with new task data during training. Dark Experience Replay (DER++), from University of Bologna, augments stored samples with soft logit targets from the model at the time of storage, providing richer supervisory signal than hard labels alone. For vision classification benchmarks (Split-CIFAR, Split-ImageNet variants), DER++ consistently outperforms regularization-only methods when buffer sizes are above a few hundred examples per class.

Generative replay, where a generative model produces synthetic examples of past tasks instead of storing real data, avoids privacy concerns but introduces a new failure mode: if the generative model itself is subject to forgetting, the quality of generated past-task samples degrades over time. Diffusion-based replay is receiving attention in 2025 as a potential improvement over GAN-based replay due to more stable training, though the generation latency of diffusion models adds overhead to each training step.

Architectural Expansion Methods

Methods like PackNet and Progressive Neural Networks allocate network capacity to new tasks by pruning or adding parameters, ensuring old task weights are protected by construction. These methods largely solve forgetting but at the cost of growing model size, which is impractical for long task sequences or deployment on fixed-memory hardware. Parameter isolation via structured sparsity, where tasks are allocated sparse subnetworks of a fixed-size model, is a more compact variant explored in recent work from continual learning groups at Mila and EPFL. Prompt-based CL for vision transformers, using learnable task-specific prompt tokens that are frozen after training, is emerging as a low-overhead architectural approach that avoids full parameter growth.

Outlook

No single approach dominates across all CL scenarios. Regularization methods work at low memory cost for short sequences. Rehearsal is reliable when buffer storage is permissible. Architectural methods provide correctness guarantees but scale poorly. Hybrid approaches combining small rehearsal buffers with lightweight regularization appear to offer the best practical tradeoff for production vision systems as of mid-2025.