High-dimensional Asymptotics of VAEs: Threshold of Posterior Collapse and Dataset-Size Dependence of Rate-Distortion Curve
arXiv:2309.07663 · doi:10.1088/1742-5468/adde3e
Abstract
In variational autoencoders (VAEs), the variational posterior often collapses to the prior, known as posterior collapse, which leads to poor representation learning quality. An adjustable hyperparameter beta has been introduced in VAEs to address this issue. This study sharply evaluates the conditions under which the posterior collapse occurs with respect to beta and dataset size by analyzing a minimal VAE in a high-dimensional limit. Additionally, this setting enables the evaluation of the rate-distortion curve of the VAE. Our results show that, unlike typical regularization parameters, VAEs face "inevitable posterior collapse" beyond a certain beta threshold, regardless of dataset size. Moreover, the dataset-size dependence of the derived rate-distortion curve suggests that relatively large datasets are required to achieve a rate-distortion curve with high rates. These findings robustly explain generalization behavior observed in various real datasets with highly non-linear VAEs.
arXiv comment25 pages, 7 figures
References in corpus (25)
- Adam: A Method for Stochastic Optimization
- Auto-Encoding Variational Bayes
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
- Rank-Sparsity Incoherence for Matrix Decomposition
- NVAE: A Deep Hierarchical Variational Autoencoder
- Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models
- Statistical Mechanics of Support Vector Networks
- Generalisation error in learning with random features and the hidden manifold model
- Thermodynamics of Restricted Boltzmann Machines and related learning dynamics
- The Gaussian equivalence of generative models for learning with shallow neural networks
- Phase Transitions in Sparse PCA
- Simple and Effective VAE Training with Calibrated Decoders
- Generalization error in high-dimensional perceptrons: Approaching Bayes error with convex optimization
- The committee machine: Computational to statistical gaps in learning a two-layers neural network
- Lossy Compression with Gaussian Diffusion
- Rate Distortion For Model Compression: From Theory To Practice
- Evaluating Lossy Compression Rates of Deep Generative Models
- An Information-Theoretic Justification for Model Pruning
- The dynamics of representation learning in shallow, non-linear autoencoders
- Statistical-mechanical study of deep Boltzmann machine given weight parameters after training by singular value decomposition
- High-dimensional Asymptotics of Denoising Autoencoders
- Analysis of feature learning in weight-tied autoencoders via the mean field lens
- Posterior Collapse of a Linear Latent Variable Model
- Multi-Rate VAE: Train Once, Get the Full Rate-Distortion Curve