A Mathematical Framework for Learning Probability Distributions

Abstract

The modeling of probability distributions, specifically generative modeling and density estimation, has become an immensely popular subject in recent years by virtue of its outstanding performance on sophisticated data such as images and texts. Nevertheless, a theoretical understanding of its success is still incomplete. One mystery is the paradox between memorization and generalization: In theory, the model is trained to be exactly the same as the empirical distribution of the finite samples, whereas in practice, the trained model can generate new samples or estimate the likelihood of unseen samples. Likewise, the overwhelming diversity of distribution learning models calls for a unified perspective on this subject. This paper provides a mathematical framework such that all the well-known models can be derived based on simple principles. To demonstrate its efficacy, we present a survey of our results on the approximation error, training error and generalization error of these models, which can all be established based on this framework. In particular, the aforementioned paradox is resolved by proving that these models enjoy implicit regularization during training, so that the generalization error at early-stopping avoids the curse of dimensionality. Furthermore, we provide some new results on landscape analysis and the mode collapse phenomenon.

arXiv commentfixed typos

References in corpus (52)

• Deep Learning in Neural Networks: An Overview
• Auto-Encoding Variational Bayes
• Conditional Generative Adversarial Nets
• A Simple Framework for Contrastive Learning of Visual Representations
• Denoising Diffusion Probabilistic Models
• WaveNet: A Generative Model for Raw Audio
• Bootstrap your own latent: A new approach to self-supervised Learning
• PaLM: Scaling Language Modeling with Pathways
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• Variational Inference with Normalizing Flows
• Score-Based Generative Modeling through Stochastic Differential Equations
• Normalizing Flows: An Introduction and Review of Current Methods
• Zero-Shot Text-to-Image Generation
• Generative Modeling by Estimating Gradients of the Data Distribution
• Energy-based Generative Adversarial Network
• Which Training Methods for GANs do actually Converge?
• Wasserstein GAN
• Recent Advances in Recurrent Neural Networks
• Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks
• Generative Moment Matching Networks
• A Variational Approach to Enhanced Sampling and Free Energy Calculations
• Parallel WaveNet: Fast High-Fidelity Speech Synthesis
• TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up
• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• On Gradient Descent Ascent for Nonconvex-Concave Minimax Problems
• Neural networks-based variationally enhanced sampling
• Neural Network Renormalization Group
• On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport
• On Unifying Deep Generative Models
• Nonparametric Density Estimation for High-Dimensional Data - Algorithms and Applications
• Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow
• Efficient sampling of high-dimensional free energy landscapes using adaptive reinforced dynamics
• Machine Learning from a Continuous Viewpoint
• A Priori Estimates of the Population Risk for Residual Networks
• Concentration in unbounded metric spaces and algorithmic stability
• Monge-Ampère Flow for Generative Modeling
• Towards GAN Benchmarks Which Require Generalization
• Overview frequency principle/spectral bias in deep learning
• Building Normalizing Flows with Stochastic Interpolants
• On the Approximation Properties of Random ReLU Features
• Understanding Overparameterization in Generative Adversarial Networks
• Generalization Error of GAN from the Discriminator's Perspective
• Variational Inference: A Unified Framework of Generative Models and Some Revelations
• Alleviating Mode Collapse in GAN via Diversity Penalty Module
• Generalization and Memorization: The Bias Potential Model
• Unifying Generative Models with GFlowNets and Beyond
• Generative Modeling with Optimal Transport Maps
• The Slow Deterioration of the Generalization Error of the Random Feature Model
• Optimal Transport Based Generative Autoencoders
• Learning Optimal Flows for Non-Equilibrium Importance Sampling
• Making Method of Moments Great Again? -- How can GANs learn distributions