Monte Carlo Gradient Estimation in Machine Learning

Abstract

This paper is a broad and accessible survey of the methods we have at our disposal for Monte Carlo gradient estimation in machine learning and across the statistical sciences: the problem of computing the gradient of an expectation of a function with respect to parameters defining the distribution that is integrated; the problem of sensitivity analysis. In machine learning research, this gradient problem lies at the core of many learning problems, in supervised, unsupervised and reinforcement learning. We will generally seek to rewrite such gradients in a form that allows for Monte Carlo estimation, allowing them to be easily and efficiently used and analysed. We explore three strategies--the pathwise, score function, and measure-valued gradient estimators--exploring their historical development, derivation, and underlying assumptions. We describe their use in other fields, show how they are related and can be combined, and expand on their possible generalisations. Wherever Monte Carlo gradient estimators have been derived and deployed in the past, important advances have followed. A deeper and more widely-held understanding of this problem will lead to further advances, and it is these advances that we wish to support.

62 pages

References in corpus (32)

• Auto-Encoding Variational Bayes
• Continuous control with deep reinforcement learning
• Variational Inference with Normalizing Flows
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• Learning Continuous Control Policies by Stochastic Value Gradients
• A General Framework for the Parametrization of Hierarchical Models
• Deep Generative Stochastic Networks Trainable by Backprop
• Learning in Implicit Generative Models
• Gradient Estimation Using Stochastic Computation Graphs
• The Optimal Reward Baseline for Gradient-Based Reinforcement Learning
• REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models
• Implicit Reparameterization Gradients
• Variational Bayesian Inference with Stochastic Search
• Measuring Sample Quality with Stein's Method
• Automated Variational Inference in Probabilistic Programming
• Maximizing acquisition functions for Bayesian optimization
• Pathwise Derivatives Beyond the Reparameterization Trick
• Operator Variational Inference
• Gradient Estimators for Implicit Models
• A Spectral Approach to Gradient Estimation for Implicit Distributions
• Reducing Reparameterization Gradient Variance
• Deep Amortized Inference for Probabilistic Programs
• Efficient Gradient-Based Inference through Transformations between Bayes Nets and Neural Nets
• Stochastic Backpropagation through Mixture Density Distributions
• Counterfactual Prediction with Deep Instrumental Variables Networks
• Fast Second-Order Stochastic Backpropagation for Variational Inference
• Credit Assignment Techniques in Stochastic Computation Graphs
• Total stochastic gradient algorithms and applications in reinforcement learning
• Reparameterization Gradient for Non-differentiable Models
• GO Gradient for Expectation-Based Objectives
• New Tricks for Estimating Gradients of Expectations
• The Epic Story of Maximum Likelihood

Cited by in corpus (36)

• Learning Discrete Structures for Graph Neural Networks
• Yao.jl: Extensible, Efficient Framework for Quantum Algorithm Design
• Lifelong Generative Modeling
• Meta-Learning to Communicate: Fast End-to-End Training for Fading Channels
• Learning Generalisable Omni-Scale Representations for Person Re-Identification
• DADA: Differentiable Automatic Data Augmentation
• Ab-initio study of interacting fermions at finite temperature with neural canonical transformation
• Amortized Variational Inference: A Systematic Review
• Graph Deep Learning for Time Series Forecasting
• Adjoint DSMC for nonlinear Boltzmann equation constrained optimization
• Learning Neural Event Functions for Ordinary Differential Equations
• Gradient Estimation with Stochastic Softmax Tricks
• Intensity-Free Learning of Temporal Point Processes
• ADEV: Sound Automatic Differentiation of Expected Values of Probabilistic Programs
• Robust, Accurate Stochastic Optimization for Variational Inference
• A Physical Model for Microstructural Characterization and Segmentation of 3D Tomography Data
• ITENE: Intrinsic Transfer Entropy Neural Estimator
• Hierarchical Indian Buffet Neural Networks for Bayesian Continual Learning
• Efficient Marginalization of Discrete and Structured Latent Variables via Sparsity
• Flexible Prior Elicitation via the Prior Predictive Distribution
• Optimal Baseline Corrections for Off-Policy Contextual Bandits
• Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planning
• The Differentiable Cross-Entropy Method
• Probabilistic Programming with Programmable Variational Inference
• Telescoping Density-Ratio Estimation
• Order Matters: Probabilistic Modeling of Node Sequence for Graph Generation
• Loaded DiCE: Trading off Bias and Variance in Any-Order Score Function Estimators for Reinforcement Learning
• Combining Pessimism with Optimism for Robust and Efficient Model-Based Deep Reinforcement Learning
• A unified view of likelihood ratio and reparameterization gradients and an optimal importance sampling scheme
• Joint Sampling and Optimisation for Inverse Rendering
• Amortized Population Gibbs Samplers with Neural Sufficient Statistics
• Scaling Bayesian inference of mixed multinomial logit models to very large datasets
• GO Hessian for Expectation-Based Objectives
• Differentiable Greedy Submodular Maximization: Guarantees, Gradient Estimators, and Applications
• Approximation Based Variance Reduction for Reparameterization Gradients
• Importance sampling-based gradient method for dimension reduction in Poisson log-normal model