Bi-level Score Matching for Learning Energy-based Latent Variable Models

Abstract

Score matching (SM) provides a compelling approach to learn energy-based models (EBMs) by avoiding the calculation of partition function. However, it remains largely open to learn energy-based latent variable models (EBLVMs), except some special cases. This paper presents a bi-level score matching (BiSM) method to learn EBLVMs with general structures by reformulating SM as a bi-level optimization problem. The higher level introduces a variational posterior of the latent variables and optimizes a modified SM objective, and the lower level optimizes the variational posterior to fit the true posterior. To solve BiSM efficiently, we develop a stochastic optimization algorithm with gradient unrolling. Theoretically, we analyze the consistency of BiSM and the convergence of the stochastic algorithm. Empirically, we show the promise of BiSM in Gaussian restricted Boltzmann machines and highly nonstructural EBLVMs parameterized by deep convolutional neural networks. BiSM is comparable to the widely adopted contrastive divergence and SM methods when they are applicable; and can learn complex EBLVMs with intractable posteriors to generate natural images.

References in corpus (24)

• Adam: A Method for Stochastic Optimization
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Categorical Reparameterization with Gumbel-Softmax
• Semi-Supervised Learning with Deep Generative Models
• NICE: Non-linear Independent Components Estimation
• Generative Modeling by Estimating Gradients of the Data Distribution
• Deep Equilibrium Models
• Triple Generative Adversarial Nets
• A Connection between Generative Adversarial Networks, Inverse Reinforcement Learning, and Energy-Based Models
• Meta-Learning with Implicit Gradients
• Bilevel Programming for Hyperparameter Optimization and Meta-Learning
• Implicit Generation and Generalization in Energy-Based Models
• Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One
• Sliced Score Matching: A Scalable Approach to Density and Score Estimation
• Interpretation and Generalization of Score Matching
• Minimum Probability Flow Learning
• Deep Energy Estimator Networks
• Graphical Generative Adversarial Networks
• Variational approximations using Fisher divergence
• Variational Noise-Contrastive Estimation
• Learning Implicit Generative Models by Teaching Explicit Ones
• To Relieve Your Headache of Training an MRF, Take AdVIL
• Efficient Learning of Generative Models via Finite-Difference Score Matching

Cited by in corpus (1)

• Efficient Learning of Generative Models via Finite-Difference Score Matching