Stochastic Resetting Mitigates Latent Gradient Bias of SGD from Label Noise
arXiv:2406.00396 · doi:10.1088/2632-2153/adbc46
Abstract
Giving up and starting over may seem wasteful in many situations such as searching for a target or training deep neural networks (DNNs). Our study, though, demonstrates that resetting from a checkpoint can significantly improve generalization performance when training DNNs with noisy labels. In the presence of noisy labels, DNNs initially learn the general patterns of the data but then gradually memorize the corrupted data, leading to overfitting. By deconstructing the dynamics of stochastic gradient descent (SGD), we identify the behavior of a latent gradient bias induced by noisy labels, which harms generalization. To mitigate this negative effect, we apply the stochastic resetting method to SGD, inspired by recent developments in the field of statistical physics achieving efficient target searches. We first theoretically identify the conditions where resetting becomes beneficial, and then we empirically validate our theory, confirming the significant improvements achieved by resetting. We further demonstrate that our method is both easy to implement and compatible with other methods for handling noisy labels. Additionally, this work offers insights into the learning dynamics of DNNs from an interpretability perspective, expanding the potential to analyze training methods through the lens of statistical physics.
arXiv comment30 pages, 14 figures
References in corpus (30)
- Machine learning and the physical sciences
- Diffusion with Stochastic Resetting
- Intermittent search strategies
- Stochastic Resetting and Applications
- Diffusion with Optimal Resetting
- First Passage Under Restart
- Optimal stochastic restart renders fluctuations in first passage times universal
- Experimental realization of diffusion with stochastic resetting
- Diffusion under time-dependent resetting
- Diffusion in a potential landscape with stochastic resetting
- Three Factors Influencing Minima in SGD
- Diffusion with resetting in arbitrary spatial dimension
- Diffusion with stochastic resetting at power-law times
- Search with home returns provides advantage under high uncertainty
- Péclet number governs transition to acceleratory restart in drift-diffusion
- Optimizing persistent random searches
- Diffusion with resetting in a logarithmic potential
- Path-integral formalism for stochastic resetting: Exactly solved examples and shortcuts to confinement
- Stochastic resetting in underdamped Brownian motion
- Stochastic resetting in interacting particle systems: A review
- Do We Train on Test Data? Purging CIFAR of Near-Duplicates
- Random acceleration process under stochastic resetting
- Dynamical Regimes of Diffusion Models
- Optimal non-Markovian search strategies with n-step memory
- Stochastic Resetting for Enhanced Sampling
- Stochastic gradient descent introduces an effective landscape-dependent regularization favoring flat solutions
- Mitigating long queues and waiting times with service resetting
- Sampling with flows, diffusion and autoregressive neural networks: A spin-glass perspective
- Self-trapping self-repelling random walks
- Unexpected advantages of exploitation for target searches in complex networks