Abstract

Federated learning is often presented as the privacy-respecting alternative to centralized training, and differential privacy (DP) is the mathematical guarantee that makes that claim defensible. But DP guarantees degrade with every training round, every composition of queries, and every downstream fine-tuning step applied to the resulting model. In production deployments, privacy budget exhaustion is treated inconsistently: some teams track epsilon carefully; many do not. The result is models that carry a DP label but whose actual guarantees have silently collapsed.

How Budget Exhaustion Occurs

Every gradient update clipped and noised under DP consumes a slice of the epsilon budget. In federated settings, the number of rounds is often set by convergence criteria rather than privacy planners, which means teams discover they have burned through epsilon after the fact, not before. Composition theorems, including the moments accountant method and Renyi DP accounting used in Google’s TensorFlow Privacy and OpenMined’s PySyft, give tighter bounds than naive composition but still require that someone actually runs the accountant at each round boundary. When federated jobs are checkpointed and resumed, or when a base model is fine-tuned by a second team with a fresh DP wrapper, the global epsilon stack is often not propagated, creating a double-counting error that inflates the apparent privacy guarantee.

Downstream Fine-Tuning as a Hidden Risk

The specific failure mode that drew scrutiny in late 2025 is sequential fine-tuning: a federated model trained under epsilon = 8 is released to a downstream team, which applies a second round of DP-SGD with its own epsilon = 4 claim. Without explicit tracking that the base model already consumed budget from the same data distribution, the effective epsilon for the combined system could be substantially higher. This is not hypothetical; the US National Institute of Standards and Technology’s AI Risk Management Framework (AI RMF 1.0) flags composition across organizational boundaries as an unresolved governance gap, and the gap remains unresolved.

Practical Mitigations and Honest Accounting

The minimum defensible practice is to treat epsilon as a property of the dataset, not the training run. Tools such as Google’s DP Accounting library and Apple’s published differential privacy infrastructure allow budget to be registered against a data source identifier and checked before any new training job accesses that source. Model cards should surface the epsilon and delta values that actually apply to the released checkpoint, including all composition steps, not the nominal per-round budget. Audit logs that record every gradient computation touching a given dataset are a precondition for any honest accounting. Without them, DP labels on federated models are assertions, not guarantees.