Efficient Learning of Generative Models via Finite-Difference Score Matching
arXiv:2007.03317
Abstract
Several machine learning applications involve the optimization of higher-order derivatives (e.g., gradients of gradients) during training, which can be expensive in respect to memory and computation even with automatic differentiation. As a typical example in generative modeling, score matching (SM) involves the optimization of the trace of a Hessian. To improve computing efficiency, we rewrite the SM objective and its variants in terms of directional derivatives, and present a generic strategy to efficiently approximate any-order directional derivative with finite difference (FD). Our approximation only involves function evaluations, which can be executed in parallel, and no gradient computations. Thus, it reduces the total computational cost while also improving numerical stability. We provide two instantiations by reformulating variants of SM objectives into the FD forms. Empirically, we demonstrate that our methods produce results comparable to the gradient-based counterparts while being much more computationally efficient.
arXiv commentNeurIPS 2020
References in corpus (29)
- Adam: A Method for Stochastic Optimization
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- Auto-Encoding Variational Bayes
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Pixel Recurrent Neural Networks
- NICE: Non-linear Independent Components Estimation
- Generative Modeling by Estimating Gradients of the Data Distribution
- Which Training Methods for GANs do actually Converge?
- A Learned Representation For Artistic Style
- Reinforcement Learning with Deep Energy-Based Policies
- Sum-Product Networks: A New Deep Architecture
- Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
- Wasserstein Auto-Encoders
- A Theory of Generative ConvNet
- WAIC, but Why? Generative Ensembles for Robust Anomaly Detection
- Lagging Inference Networks and Posterior Collapse in Variational Autoencoders
- Density Estimation in Infinite Dimensional Exponential Families
- Sliced Score Matching: A Scalable Approach to Density and Score Estimation
- Maximum Entropy Generators for Energy-Based Models
- Detecting Out-of-Distribution Inputs to Deep Generative Models Using Typicality
- Gradient Estimators for Implicit Models
- A Spectral Approach to Gradient Estimation for Implicit Distributions
- Deep Energy Estimator Networks
- Gradient-free Hamiltonian Monte Carlo with Efficient Kernel Exponential Families
- SUMO: Unbiased Estimation of Log Marginal Probability for Latent Variable Models
- Understanding and Stabilizing GANs' Training Dynamics with Control Theory
- Bi-level Score Matching for Learning Energy-based Latent Variable Models
- A Wasserstein Minimum Velocity Approach to Learning Unnormalized Models