Quantization-Based Regularization for Autoencoders
arXiv:1905.11062
Abstract
Autoencoders and their variations provide unsupervised models for learning low-dimensional representations for downstream tasks. Without proper regularization, autoencoder models are susceptible to the overfitting problem and the so-called posterior collapse phenomenon. In this paper, we introduce a quantization-based regularizer in the bottleneck stage of autoencoder models to learn meaningful latent representations. We combine both perspectives of Vector Quantized-Variational AutoEncoders (VQ-VAE) and classical denoising regularization methods of neural networks. We interpret quantizers as regularizers that constrain latent representations while fostering a similarity-preserving mapping at the encoder. Before quantization, we impose noise on the latent codes and use a Bayesian estimator to optimize the quantizer-based representation. The introduced bottleneck Bayesian estimator outputs the posterior mean of the centroids to the decoder, and thus, is performing soft quantization of the noisy latent codes. We show that our proposed regularization method results in improved latent representations for both supervised learning and clustering downstream tasks when compared to autoencoders using other bottleneck structures.
arXiv commentAAAI 2020
References in corpus (14)
- Auto-Encoding Variational Bayes
- Neural Discrete Representation Learning
- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
- End-to-end Optimized Image Compression
- NIPS 2016 Tutorial: Generative Adversarial Networks
- Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations
- Deep Variational Information Bottleneck
- Information-theoretic analysis of generalization capability of learning algorithms
- Clustering with Deep Learning: Taxonomy and New Methods
- Fast Decoding in Sequence Models using Discrete Latent Variables
- SOM-VAE: Interpretable Discrete Representation Learning on Time Series
- Denoising Criterion for Variational Auto-Encoding Framework
- Amortized Inference Regularization
- Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis