From Variational to Deterministic Autoencoders

Abstract

Variational Autoencoders (VAEs) provide a theoretically-backed and popular framework for deep generative models. However, learning a VAE from data poses still unanswered theoretical questions and considerable practical challenges. In this work, we propose an alternative framework for generative modeling that is simpler, easier to train, and deterministic, yet has many of the advantages of VAEs. We observe that sampling a stochastic encoder in a Gaussian VAE can be interpreted as simply injecting noise into the input of a deterministic decoder. We investigate how substituting this kind of stochasticity, with other explicit and implicit regularization schemes, can lead to an equally smooth and meaningful latent space without forcing it to conform to an arbitrarily chosen prior. To retrieve a generative mechanism to sample new data, we introduce an ex-post density estimation step that can be readily applied also to existing VAEs, improving their sample quality. We show, in a rigorous empirical study, that the proposed regularized deterministic autoencoders are able to generate samples that are comparable to, or better than, those of VAEs and more powerful alternatives when applied to images as well as to structured data such as molecules. \footnote{An implementation is available at: \url{https://github.com/ParthaEth/Regularized_autoencoders-RAE-}}

arXiv commentPartha Ghosh and Mehdi S. M. Sajjadi contributed equally to this work

References in corpus (23)

• Auto-Encoding Variational Bayes
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• Neural Discrete Representation Learning
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Wide Residual Networks
• Improved Training of Wasserstein GANs
• End-to-end Optimized Image Compression
• Junction Tree Variational Autoencoder for Molecular Graph Generation
• Which Training Methods for GANs do actually Converge?
• MADE: Masked Autoencoder for Distribution Estimation
• Adversarial Autoencoders
• Variational Lossy Autoencoder
• Wasserstein Auto-Encoders
• IntroVAE: Introspective Variational Autoencoders for Photographic Image Synthesis
• Amortised MAP Inference for Image Super-resolution
• REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models
• Towards Deeper Understanding of Variational Autoencoding Models
• Generating Diverse High-Fidelity Images with VQ-VAE-2
• Diagnosing and Enhancing VAE Models
• Geometry of Optimization and Implicit Regularization in Deep Learning
• Distribution Matching in Variational Inference
• A Hybrid Convolutional Variational Autoencoder for Text Generation

Cited by in corpus (24)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• ControlVAE: Controllable Variational Autoencoder
• AstroVaDEr: Astronomical Variational Deep Embedder for Unsupervised Morphological Classification of Galaxies and Synthetic Image Generation
• The Free Energy Principle for Perception and Action: A Deep Learning Perspective
• Physics-aware Reduced-order Modeling of Transonic Flow via $\beta$-Variational Autoencoder
• Simple and Effective VAE Training with Calibrated Decoders
• Temporally coherent backmapping of molecular trajectories from coarse-grained to atomistic resolution
• Forward and inverse reinforcement learning sharing network weights and hyperparameters
• Perceptual Generative Autoencoders
• Unpacking Information Bottlenecks: Unifying Information-Theoretic Objectives in Deep Learning
• Disentangling Geometric Deformation Spaces in Generative Latent Shape Models
• Exploratory State Representation Learning
• Pythae: Unifying Generative Autoencoders in Python -- A Benchmarking Use Case
• Robust Variational Autoencoder for Tabular Data with Beta Divergence
• Batch norm with entropic regularization turns deterministic autoencoders into generative models
• Multi-modal data generation with a deep metric variational autoencoder
• ControlVAE: Tuning, Analytical Properties, and Performance Analysis
• Double cycle-consistent generative adversarial network for unsupervised conditional generation
• LDC-VAE: A Latent Distribution Consistency Approach to Variational AutoEncoders
• Cauchy-Schwarz Regularized Autoencoder
• ByPE-VAE: Bayesian Pseudocoresets Exemplar VAE
• Feedback Recurrent AutoEncoder
• Momentum Contrastive Autoencoder: Using Contrastive Learning for Latent Space Distribution Matching in WAE
• Group-disentangled Representation Learning with Weakly-Supervised Regularization