A likelihood approach to nonparametric estimation of a singular distribution using deep generative models
arXiv:2105.04046
Abstract
We investigate statistical properties of a likelihood approach to nonparametric estimation of a singular distribution using deep generative models. More specifically, a deep generative model is used to model high-dimensional data that are assumed to concentrate around some low-dimensional structure. Estimating the distribution supported on this low-dimensional structure, such as a low-dimensional manifold, is challenging due to its singularity with respect to the Lebesgue measure in the ambient space. In the considered model, a usual likelihood approach can fail to estimate the target distribution consistently due to the singularity. We prove that a novel and effective solution exists by perturbing the data with an instance noise, which leads to consistent estimation of the underlying distribution with desirable convergence rates. We also characterize the class of distributions that can be efficiently estimated via deep generative models. This class is sufficiently general to contain various structured distributions such as product distributions, classically smooth distributions and distributions supported on a low-dimensional manifold. Our analysis provides some insights on how deep generative models can avoid the curse of dimensionality for nonparametric distribution estimation. We conduct a thorough simulation study and real data analysis to empirically demonstrate that the proposed data perturbation technique improves the estimation performance significantly.
arXiv comment42 pages, 13 figures, 1 table
References in corpus (19)
- Auto-Encoding Variational Bayes
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Empirical Evaluation of Rectified Activations in Convolutional Network
- Variational Inference with Normalizing Flows
- Generative Modeling by Estimating Gradients of the Data Distribution
- MMD GAN: Towards Deeper Understanding of Moment Matching Network
- Markov Chain Monte Carlo and Variational Inference: Bridging the Gap
- Adaptive Bayesian multivariate density estimation with Dirichlet mixtures
- Approximation and Convergence Properties of Generative Adversarial Learning
- Adaptive Approximation and Generalization of Deep Neural Network with Intrinsic Dimensionality
- On the Discrimination-Generalization Tradeoff in GANs
- Reinterpreting Importance-Weighted Autoencoders
- Deep ReLU network approximation of functions on a manifold
- Approximability of Discriminators Implies Diversity in GANs
- How Well Generative Adversarial Networks Learn Distributions
- On the rate of convergence of fully connected very deep neural network regression estimates
- Structure-adaptive manifold estimation
- Statistical guarantees for generative models without domination
- Reweighted Expectation Maximization