Adding noise to the input of a model trained with a regularized objective

Abstract

Regularization is a well studied problem in the context of neural networks. It is usually used to improve the generalization performance when the number of input samples is relatively small or heavily contaminated with noise. The regularization of a parametric model can be achieved in different manners some of which are early stopping (Morgan and Bourlard, 1990), weight decay, output smoothing that are used to avoid overfitting during the training of the considered model. From a Bayesian point of view, many regularization techniques correspond to imposing certain prior distributions on model parameters (Krogh and Hertz, 1991). Using Bishop's approximation (Bishop, 1995) of the objective function when a restricted type of noise is added to the input of a parametric function, we derive the higher order terms of the Taylor expansion and analyze the coefficients of the regularization terms induced by the noisy input. In particular we study the effect of penalizing the Hessian of the mapping function with respect to the input in terms of generalization performance. We also show how we can control independently this coefficient by explicitly penalizing the Jacobian of the mapping function on corrupted inputs.

Cited by in corpus (20)

• Deep Learning using Linear Support Vector Machines
• Dropout Training as Adaptive Regularization
• Regularization for Deep Learning: A Taxonomy
• Long-time predictive modeling of nonlinear dynamical systems using neural networks
• Dropout as a Low-Rank Regularizer for Matrix Factorization
• Improving Pre-trained Language Model Fine-tuning with Noise Stability Regularization
• Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like process
• Efficient ConvNets for Analog Arrays
• On Classification of Distorted Images with Deep Convolutional Neural Networks
• On architectural choices in deep learning: From network structure to gradient convergence and parameter estimation
• Stochastic gradient descent with random learning rate
• Binary Stochastic Filtering: feature selection and beyond
• An Analysis of Dropout for Matrix Factorization
• Convergence rates for pretraining and dropout: Guiding learning parameters using network structure
• Shakeout: A New Approach to Regularized Deep Neural Network Training
• Efficient Modelling Across Time of Human Actions and Interactions
• Block-Cyclic Stochastic Coordinate Descent for Deep Neural Networks
• Noise Stability Regularization for Improving BERT Fine-tuning
• Regularization by Misclassification in ReLU Neural Networks
• Information based Deep Clustering: An experimental study