Noisy Activation Functions

Abstract

Common nonlinear activation functions used in neural networks can cause training difficulties due to the saturation behavior of the activation function, which may hide dependencies that are not visible to vanilla-SGD (using first order gradients only). Gating mechanisms that use softly saturating activation functions to emulate the discrete switching of digital logic circuits are good examples of this. We propose to exploit the injection of appropriate noise so that the gradients may flow easily, even if the noiseless application of the activation function would yield zero gradient. Large noise will dominate the noise-free gradient and allow stochastic gradient descent toexplore more. By adding noise only to the problematic parts of the activation function, we allow the optimization procedure to explore the boundary between the degenerate (saturating) and the well-behaved parts of the activation function. We also establish connections to simulated annealing, when the amount of noise is annealed down, making it easier to optimize hard objective functions. We find experimentally that replacing such saturating activation functions by noisy variants helps training in many contexts, yielding state-of-the-art or competitive results on different datasets and task, especially when training seems to be the most difficult, e.g., when curriculum learning is necessary to obtain good results.

References in corpus (11)

• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Teaching Machines to Read and Comprehend
• Recurrent Neural Network Regularization
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Visualizing and Understanding Recurrent Networks
• A Simple Way to Initialize Recurrent Networks of Rectified Linear Units
• Learning to Execute
• Adding Gradient Noise Improves Learning for Very Deep Networks
• Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition
• Estimating or Propagating Gradients Through Stochastic Neurons
• Training Recurrent Neural Networks by Diffusion

Cited by in corpus (25)

• Deep Gated Recurrent and Convolutional Network Hybrid Model for Univariate Time Series Classification
• Neural Vector Spaces for Unsupervised Information Retrieval
• Neural Machine Translation with External Phrase Memory
• Deep Convolutional Neural Network Design Patterns
• Dynamic Neural Turing Machine with Soft and Hard Addressing Schemes
• Inflation Model Selection meets Dark Radiation
• Noisy Softmax: Improving the Generalization Ability of DCNN via Postponing the Early Softmax Saturation
• Using deep Residual Networks to search for galaxy-Lyα emitter lens candidates based on spectroscopic-selection
• Activation Ensembles for Deep Neural Networks
• Improving Deep Neural Network with Multiple Parametric Exponential Linear Units
• Structured Convolution Matrices for Energy-efficient Deep learning
• Stability of Internal States in Recurrent Neural Networks Trained on Regular Languages
• Neural Networks with A La Carte Selection of Activation Functions
• Learning to Generate Wikipedia Summaries for Underserved Languages from Wikidata
• Simplified Stochastic Feedforward Neural Networks
• Mollifying Networks
• Energy Confused Adversarial Metric Learning for Zero-Shot Image Retrieval and Clustering
• Towards Non-saturating Recurrent Units for Modelling Long-term Dependencies
• Neural Networks with Activation Networks
• Detection of Glottal Closure Instants from Raw Speech using Convolutional Neural Networks
• Learning Simpler Language Models with the Differential State Framework
• Deep-Learning Inversion Method for the Interpretation of Noisy Logging-While-Drilling Resistivity Measurements
• NeuralDP Differentially private neural networks by design
• Minimalist Regression Network with Reinforced Gradients and Weighted Estimates: a Case Study on Parameters Estimation in Automated Welding
• Diagnostic Visualization for Deep Neural Networks Using Stochastic Gradient Langevin Dynamics