On Energy-Based Models with Overparametrized Shallow Neural Networks
arXiv:2104.07531
Abstract
Energy-based models (EBMs) are a simple yet powerful framework for generative modeling. They are based on a trainable energy function which defines an associated Gibbs measure, and they can be trained and sampled from via well-established statistical tools, such as MCMC. Neural networks may be used as energy function approximators, providing both a rich class of expressive models as well as a flexible device to incorporate data structure. In this work we focus on shallow neural networks. Building from the incipient theory of overparametrized neural networks, we show that models trained in the so-called "active" regime provide a statistical advantage over their associated "lazy" or kernel regime, leading to improved adaptivity to hidden low-dimensional structure in the data distribution, as already observed in supervised learning. Our study covers both maximum likelihood and Stein Discrepancy estimators, and we validate our theoretical results with numerical experiments on synthetic data.
References in corpus (14)
- Auto-Encoding Variational Bayes
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
- Generative Modeling by Estimating Gradients of the Data Distribution
- A Mean Field View of the Landscape of Two-Layers Neural Networks
- Measuring Sample Quality with Stein's Method
- Implicit Generation and Generalization in Energy-Based Models
- A Kernel Test of Goodness of Fit
- Measuring Sample Quality with Kernels
- When Do Neural Networks Outperform Kernel Methods?
- Learning the Stein Discrepancy for Training and Evaluating Energy-Based Models without Sampling
- Nonparametric Density Estimation under Adversarial Losses
- Generative Modeling with Denoising Auto-Encoders and Langevin Sampling
- Adaptive Monte Carlo augmented with normalizing flows
- Quantifying the Benefit of Using Differentiable Learning over Tangent Kernels