Explicit Regularisation in Gaussian Noise Injections
arXiv:2007.07368
Abstract
We study the regularisation induced in neural networks by Gaussian noise injections (GNIs). Though such injections have been extensively studied when applied to data, there have been few studies on understanding the regularising effect they induce when applied to network activations. Here we derive the explicit regulariser of GNIs, obtained by marginalising out the injected noise, and show that it penalises functions with high-frequency components in the Fourier domain; particularly in layers closer to a neural network's output. We show analytically and empirically that such regularisation produces calibrated classifiers with large classification margins.
References in corpus (21)
- On Calibration of Modern Neural Networks
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Certified Adversarial Robustness via Randomized Smoothing
- Variational Dropout and the Local Reparameterization Trick
- On Exact Computation with an Infinitely Wide Neural Net
- Exponential expressivity in deep neural networks through transient chaos
- Dropout Training as Adaptive Regularization
- Robust Large Margin Deep Neural Networks
- On Lazy Training in Differentiable Programming
- Three Factors Influencing Minima in SGD
- On the Spectral Bias of Neural Networks
- Empirical Analysis of the Hessian of Over-Parametrized Neural Networks
- Sharp Minima Can Generalize For Deep Nets
- Norm-Based Capacity Control in Neural Networks
- Analyzing noise in autoencoders and deep networks
- Practical Gauss-Newton Optimisation for Deep Learning
- A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks
- The Implicit and Explicit Regularization Effects of Dropout
- Noisin: Unbiased Regularization for Recurrent Neural Networks
- Dropout: Explicit Forms and Capacity Control
- Tangent Space Separability in Feedforward Neural Networks