Dueling Decoders: Regularizing Variational Autoencoder Latent Spaces

Abstract

Variational autoencoders learn unsupervised data representations, but these models frequently converge to minima that fail to preserve meaningful semantic information. For example, variational autoencoders with autoregressive decoders often collapse into autodecoders, where they learn to ignore the encoder input. In this work, we demonstrate that adding an auxiliary decoder to regularize the latent space can prevent this collapse, but successful auxiliary decoding tasks are domain dependent. Auxiliary decoders can increase the amount of semantic information encoded in the latent space and visible in the reconstructions. The semantic information in the variational autoencoder's representation is only weakly correlated with its rate, distortion, or evidence lower bound. Compared to other popular strategies that modify the training objective, our regularization of the latent space generally increased the semantic information content.

16 pages, 9 figures, supplemental

References in corpus (16)

• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Neural Discrete Representation Learning
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• Conditional Image Generation with PixelCNN Decoders
• Wide Residual Networks
• Semi-Supervised Learning with Deep Generative Models
• InfoVAE: Information Maximizing Variational Autoencoders
• Deep Variational Information Bottleneck
• Improving Variational Inference with Inverse Autoregressive Flow
• Lagging Inference Networks and Posterior Collapse in Variational Autoencoders
• Hyperspherical Variational Auto-Encoders
• Uncertainty in the Variational Information Bottleneck
• VAE with a VampPrior
• Generative Models of Visually Grounded Imagination
• Avoiding Latent Variable Collapse With Generative Skip Models
• TherML: Thermodynamics of Machine Learning