High-dimensional Asymptotics of VAEs: Threshold of Posterior Collapse and Dataset-Size Dependence of Rate-Distortion Curve
arXiv:2309.07663 · doi:10.1088/1742-5468/adde3e
Abstract
In variational autoencoders (VAEs), the variational posterior often collapses to the prior, known as posterior collapse, which leads to poor representation learning quality. An adjustable hyperparameter beta has been introduced in VAEs to address this issue. This study sharply evaluates the conditions under which the posterior collapse occurs with respect to beta and dataset size by analyzing a minimal VAE in a high-dimensional limit. Additionally, this setting enables the evaluation of the rate-distortion curve of the VAE. Our results show that, unlike typical regularization parameters, VAEs face "inevitable posterior collapse" beyond a certain beta threshold, regardless of dataset size. Moreover, the dataset-size dependence of the derived rate-distortion curve suggests that relatively large datasets are required to achieve a rate-distortion curve with high rates. These findings robustly explain generalization behavior observed in various real datasets with highly non-linear VAEs.
25 pages, 7 figures
References in corpus (10)
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Lossy Compression with Gaussian Diffusion
- Evaluating Lossy Compression Rates of Deep Generative Models
- An Information-Theoretic Justification for Model Pruning
- The dynamics of representation learning in shallow, non-linear autoencoders
- Statistical-mechanical study of deep Boltzmann machine given weight parameters after training by singular value decomposition
- High-dimensional Asymptotics of Denoising Autoencoders
- Analysis of feature learning in weight-tied autoencoders via the mean field lens
- Posterior Collapse of a Linear Latent Variable Model
- Multi-Rate VAE: Train Once, Get the Full Rate-Distortion Curve