MuProp: Unbiased Backpropagation for Stochastic Neural Networks

Abstract

Deep neural networks are powerful parametric models that can be trained efficiently using the backpropagation algorithm. Stochastic neural networks combine the power of large parametric functions with that of graphical models, which makes it possible to learn very complex distributions. However, as backpropagation is not directly applicable to stochastic networks that include discrete sampling operations within their computational graph, training such networks remains difficult. We present MuProp, an unbiased gradient estimator for stochastic networks, designed to make this task easier. MuProp improves on the likelihood-ratio estimator by reducing its variance using a control variate based on the first-order Taylor expansion of a mean-field network. Crucially, unlike prior attempts at using backpropagation for training stochastic networks, the resulting estimator is unbiased and well behaved. Our experiments on structured output prediction and discrete latent variable modeling demonstrate that MuProp yields consistently good performance across a range of difficult tasks.

Published as a conference paper at ICLR 2016

References in corpus (7)

• Sequence to Sequence Learning with Neural Networks
• Recurrent Models of Visual Attention
• Gradient Estimation Using Stochastic Computation Graphs
• The Optimal Reward Baseline for Gradient-Based Reinforcement Learning
• Variational Bayesian Inference with Stochastic Search
• Reinforcement Learning Neural Turing Machines - Revised
• Techniques for Learning Binary Stochastic Feedforward Neural Networks

Cited by in corpus (25)

• SNAS: Stochastic Neural Architecture Search
• Boundary-Seeking Generative Adversarial Networks
• DeepIoT: Compressing Deep Neural Network Structures for Sensing Systems with a Compressor-Critic Framework
• DVAE++: Discrete Variational Autoencoders with Overlapping Transformations
• TD-Regularized Actor-Critic Methods
• Stochastic Generative Hashing
• ACtuAL: Actor-Critic Under Adversarial Learning
• Straight-Through Estimator as Projected Wasserstein Gradient Flow
• Direct Optimization through $\arg \max$ for Discrete Variational Auto-Encoder
• Improved Gradient-Based Optimization Over Discrete Distributions
• Joint Stochastic Approximation learning of Helmholtz Machines
• GO Gradient for Expectation-Based Objectives
• Simplified Stochastic Feedforward Neural Networks
• Iterative Refinement of the Approximate Posterior for Directed Belief Networks
• Stochastic Sequential Neural Networks with Structured Inference
• Joint Stochastic Approximation and Its Application to Learning Discrete Latent Variable Models
• Probabilistic Mixture-of-Experts for Efficient Deep Reinforcement Learning
• Backprop-Q: Generalized Backpropagation for Stochastic Computation Graphs
• A unified view of likelihood ratio and reparameterization gradients and an optimal importance sampling scheme
• New Tricks for Estimating Gradients of Expectations
• Invertible Gaussian Reparameterization: Revisiting the Gumbel-Softmax
• Inherent Weight Normalization in Stochastic Neural Networks
• Learning Discrete Energy-based Models via Auxiliary-variable Local Exploration
• A Fourier View of REINFORCE
• KF-LAX: Kronecker-factored curvature estimation for control variate optimization in reinforcement learning