Theory and Experiments on Vector Quantized Autoencoders

Abstract

Deep neural networks with discrete latent variables offer the promise of better symbolic reasoning, and learning abstractions that are more useful to new tasks. There has been a surge in interest in discrete latent variable models, however, despite several recent improvements, the training of discrete latent variable models has remained challenging and their performance has mostly failed to match their continuous counterparts. Recent work on vector quantized autoencoders (VQ-VAE) has made substantial progress in this direction, with its perplexity almost matching that of a VAE on datasets such as CIFAR-10. In this work, we investigate an alternate training technique for VQ-VAE, inspired by its connection to the Expectation Maximization (EM) algorithm. Training the discrete bottleneck with EM helps us achieve better image generation results on CIFAR-10, and together with knowledge distillation, allows us to develop a non-autoregressive machine translation model whose accuracy almost matches a strong greedy autoregressive baseline Transformer, while being 3.3 times faster at inference.

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Distilling the Knowledge in a Neural Network
• Sequence to Sequence Learning with Neural Networks
• WaveNet: A Generative Model for Raw Audio
• Categorical Reparameterization with Gumbel-Softmax
• Conditional Image Generation with PixelCNN Decoders
• Lossy Image Compression with Compressive Autoencoders
• Fast Decoding in Sequence Models using Discrete Latent Variables
• Sequence-Level Knowledge Distillation
• Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
• Parallel Multiscale Autoregressive Density Estimation
• Discrete Autoencoders for Sequence Models

Cited by in corpus (23)

• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech
• Fast Structured Decoding for Sequence Models
• Lifelike Agility and Play in Quadrupedal Robots using Reinforcement Learning and Generative Pre-trained Models
• Minimizing the Bag-of-Ngrams Difference for Non-Autoregressive Neural Machine Translation
• Robust Training of Vector Quantized Bottleneck Models
• Simple, Distributed, and Accelerated Probabilistic Programming
• A Tutorial on Deep Latent Variable Models of Natural Language
• Preventing Posterior Collapse with delta-VAEs
• Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posterior
• VQ-NeRF: Neural Reflectance Decomposition and Editing with Vector Quantization
• Generative Models for Improved Naturalness, Intelligibility, and Voicing of Whispered Speech
• Latent Programmer: Discrete Latent Codes for Program Synthesis
• Max-Affine Spline Insights into Deep Generative Networks
• Direct Evolutionary Optimization of Variational Autoencoders With Binary Latents
• Extractive Summary as Discrete Latent Variables
• Learning Product Codebooks using Vector Quantized Autoencoders for Image Retrieval
• MaGNET: Uniform Sampling from Deep Generative Network Manifolds Without Retraining
• Supervised Vector Quantized Variational Autoencoder for Learning Interpretable Global Representations
• Discrete Variational Attention Models for Language Generation
• Positioning yourself in the maze of Neural Text Generation: A Task-Agnostic Survey
• Discrete Auto-regressive Variational Attention Models for Text Modeling
• Depthwise Discrete Representation Learning