GO Hessian for Expectation-Based Objectives

Abstract

An unbiased low-variance gradient estimator, termed GO gradient, was proposed recently for expectation-based objectives ${\mathbb{E}}_{q_{\mathbf{\gamma }}(\mathbf{y})}[f(\mathbf{y})]$, where the random variable (RV) $\mathbf{y}$ may be drawn from a stochastic computation graph with continuous (non-reparameterizable) internal nodes and continuous/discrete leaves. Upgrading the GO gradient, we present for ${\mathbb{E}}_{q_{\mathbf{\gamma }}(\mathbf{y})}[f(\mathbf{y})]$ an unbiased low-variance Hessian estimator, named GO Hessian. Considering practical implementation, we reveal that GO Hessian is easy-to-use with auto-differentiation and Hessian-vector products, enabling efficient cheap exploitation of curvature information over stochastic computation graphs. As representative examples, we present the GO Hessian for non-reparameterizable gamma and negative binomial RVs/nodes. Based on the GO Hessian, we design a new second-order method for ${\mathbb{E}}_{q_{\mathbf{\gamma }}(\mathbf{y})}[f(\mathbf{y})]$, with rigorous experiments conducted to verify its effectiveness and efficiency.

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Beta-Negative Binomial Process and Poisson Factor Analysis
• Gradient Estimation Using Stochastic Computation Graphs
• Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
• Monte Carlo Gradient Estimation in Machine Learning
• On Nonconvex Optimization for Machine Learning: Gradients, Stochasticity, and Saddle Points
• Second-Order Optimization for Non-Convex Machine Learning: An Empirical Study
• Stochastic Recursive Variance-Reduced Cubic Regularization Methods
• The Poisson Gamma Belief Network
• Total stochastic gradient algorithms and applications in reinforcement learning