Estimating or Propagating Gradients Through Stochastic Neurons

Abstract

Stochastic neurons can be useful for a number of reasons in deep learning models, but in many cases they pose a challenging problem: how to estimate the gradient of a loss function with respect to the input of such stochastic neurons, i.e., can we "back-propagate" through these stochastic neurons? We examine this question, existing approaches, and present two novel families of solutions, applicable in different settings. In particular, it is demonstrated that a simple biologically plausible formula gives rise to an an unbiased (but noisy) estimator of the gradient with respect to a binary stochastic neuron firing probability. Unlike other estimators which view the noise as a small perturbation in order to estimate gradients by finite differences, this estimator is unbiased even without assuming that the stochastic perturbation is small. This estimator is also interesting because it can be applied in very general settings which do not allow gradient back-propagation, including the estimation of the gradient with respect to future rewards, as required in reinforcement learning setups. We also propose an approach to approximating this unbiased but high-variance estimator by learning to predict it using a biased estimator. The second approach we propose assumes that an estimator of the gradient can be back-propagated and it provides an unbiased estimator of the gradient, but can only work with non-linearities unlike the hard threshold, but like the rectifier, that are not flat for all of their range. This is similar to traditional sigmoidal units but has the advantage that for many inputs, a hard decision (e.g., a 0 output) can be produced, which would be convenient for conditional computation and achieving sparse representations and sparse gradients.

References in corpus (2)

• Improving neural networks by preventing co-adaptation of feature detectors
• Gradient learning in spiking neural networks by dynamic perturbation of conductances

Cited by in corpus (28)

• Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
• Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
• Variational Dropout and the Local Reparameterization Trick
• Deep Generative Stochastic Networks Trainable by Backprop
• A Practical Survey on Faster and Lighter Transformers
• Deep Learning of Representations: Looking Forward
• Deep Neural Decision Trees
• Improving Deep Neural Networks with Probabilistic Maxout Units
• Noisy Activation Functions
• Pruning and Quantization for Deep Neural Network Acceleration: A Survey
• DFTerNet: Towards 2-bit Dynamic Fusion Networks for Accurate Human Activity Recognition
• Getting a CLUE: A Method for Explaining Uncertainty Estimates
• Robust Training of Vector Quantized Bottleneck Models
• Techniques for Learning Binary Stochastic Feedforward Neural Networks
• Efficient Gradient-Based Inference through Transformations between Bayes Nets and Neural Nets
• Noisy Softmax: Improving the Generalization Ability of DCNN via Postponing the Early Softmax Saturation
• Word2Bits - Quantized Word Vectors
• Opportunities and Challenges of Generative-AI in Finance
• Deep Directed Generative Autoencoders
• Quantization of Deep Neural Networks for Accumulator-constrained Processors
• Bounding the Test Log-Likelihood of Generative Models
• SGAD: Soft-Guided Adaptively-Dropped Neural Network
• Unleashing Transformers: Parallel Token Prediction with Discrete Absorbing Diffusion for Fast High-Resolution Image Generation from Vector-Quantized Codes
• Latent Iterative Refinement for Modular Source Separation
• Understanding Dropout: Training Multi-Layer Perceptrons with Auxiliary Independent Stochastic Neurons
• GSNs : Generative Stochastic Networks
• Training highly effective connectivities within neural networks with randomly initialized, fixed weights
• Differentiable Spline Approximations