Deep Energy Estimator Networks

Abstract

Density estimation is a fundamental problem in statistical learning. This problem is especially challenging for complex high-dimensional data due to the curse of dimensionality. A promising solution to this problem is given here in an inference-free hierarchical framework that is built on score matching. We revisit the Bayesian interpretation of the score function and the Parzen score matching, and construct a multilayer perceptron with a scalable objective for learning the energy (i.e. the unnormalized log-density), which is then optimized with stochastic gradient descent. In addition, the resulting deep energy estimator network (DEEN) is designed as products of experts. We present the utility of DEEN in learning the energy, the score function, and in single-step denoising experiments for synthetic and high-dimensional data. We also diagnose stability problems in the direct estimation of the score function that had been observed for denoising autoencoders.

References in corpus (3)

• Interpretation and Generalization of Score Matching
• Generalizing Hamiltonian Monte Carlo with Neural Networks
• Learning Deep Energy Models: Contrastive Divergence vs. Amortized MLE

Cited by in corpus (33)

• Generative Modeling by Estimating Gradients of the Data Distribution
• Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models
• Sliced Score Matching: A Scalable Approach to Density and Score Estimation
• Interacting particle solutions of Fokker-Planck equations through gradient-log-density estimation
• WaveGrad: Estimating Gradients for Waveform Generation
• Learning the Stein Discrepancy for Training and Evaluating Energy-Based Models without Sampling
• Improved Contrastive Divergence Training of Energy Based Models
• ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICA
• How to Train Your Energy-Based Model for Regression
• Learning Energy-Based Models by Diffusion Recovery Likelihood
• Solving Linear Inverse Problems Using the Prior Implicit in a Denoiser
• Energy-Based Imitation Learning
• Learning deep kernels for exponential family densities
• Neural Empirical Bayes
• On approximating $\nabla f$ with neural networks
• Improved Autoregressive Modeling with Distribution Smoothing
• Flow Contrastive Estimation of Energy-Based Models
• Efficient Learning of Generative Models via Finite-Difference Score Matching
• Bi-level Score Matching for Learning Energy-based Latent Variable Models
• Regularizing Model-Based Planning with Energy-Based Models
• Autoregressive Score Matching
• Distribution Aware Active Learning
• Building Minimal and Reusable Causal State Abstractions for Reinforcement Learning
• Provable Robust Classification via Learned Smoothed Densities
• General Probabilistic Surface Optimization and Log Density Estimation
• Variational (Gradient) Estimate of the Score Function in Energy-based Latent Variable Models
• Arbitrary Conditional Distributions with Energy
• Iterative Refinement in the Continuous Space for Non-Autoregressive Neural Machine Translation
• Learning and Inference in Imaginary Noise Models
• Deep PDF: Probabilistic Surface Optimization and Density Estimation
• Unnormalized Variational Bayes
• Automatic design of novel potential 3CL${}^{\text{pro}}$ and PL${}^{\text{pro}}$ inhibitors
• A Wasserstein Minimum Velocity Approach to Learning Unnormalized Models