Asymmetric Heavy Tails and Implicit Bias in Gaussian Noise Injections

Abstract

Gaussian noise injections (GNIs) are a family of simple and widely-used regularisation methods for training neural networks, where one injects additive or multiplicative Gaussian noise to the network activations at every iteration of the optimisation algorithm, which is typically chosen as stochastic gradient descent (SGD). In this paper we focus on the so-called `implicit effect' of GNIs, which is the effect of the injected noise on the dynamics of SGD. We show that this effect induces an asymmetric heavy-tailed noise on SGD gradient updates. In order to model this modified dynamics, we first develop a Langevin-like stochastic differential equation that is driven by a general family of asymmetric heavy-tailed noise. Using this model we then formally prove that GNIs induce an `implicit bias', which varies depending on the heaviness of the tails and the level of asymmetry. Our empirical results confirm that different types of neural networks trained with GNIs are well-modelled by the proposed dynamics and that the implicit effect of these injections induces a bias that degrades the performance of networks.

arXiv commentMain paper of 12 pages, followed by appendix

References in corpus (22)

• Scikit-learn: Machine Learning in Python
• An overview of gradient descent optimization algorithms
• The fundamental solution of the space-time fractional diffusion equation
• A Class of Second Order Difference Approximation for Solving Space Fractional Diffusion Equations
• Certified Adversarial Robustness via Randomized Smoothing
• Variational Dropout and the Local Reparameterization Trick
• Three Factors Influencing Minima in SGD
• Non-asymptotic convergence analysis for the Unadjusted Langevin Algorithm
• Analyzing noise in autoencoders and deep networks
• Stationary states in Langevin dynamics under asymmetric Lévy noises
• Moving Beyond Sub-Gaussianity in High-Dimensional Statistics: Applications in Covariance Estimation and Linear Regression
• The Heavy-Tail Phenomenon in SGD
• Understanding Priors in Bayesian Neural Networks at the Unit Level
• Multiplicative noise and heavy tails in stochastic optimization
• Global Convergence of Stochastic Gradient Hamiltonian Monte Carlo for Non-Convex Stochastic Optimization: Non-Asymptotic Performance Bounds and Momentum-Based Acceleration
• A Tail-Index Analysis of Stochastic Gradient Noise in Deep Neural Networks
• Fractional Underdamped Langevin Dynamics: Retargeting SGD with Momentum under Heavy-Tailed Gradient Noise
• The Binomial Coefficient for Negative Arguments
• Noisin: Unbiased Regularization for Recurrent Neural Networks
• Features modeling with an $\alpha$-stable distribution: Application to pattern recognition based on continuous belief functions
• Stationary States in Bistable System Driven by Lévy Noise
• Fractional moment-preserving initialization schemes for training deep neural networks

Cited by in corpus (1)

• Hausdorff Dimension, Heavy Tails, and Generalization in Neural Networks