InfoVAE: Information Maximizing Variational Autoencoders

Abstract

A key advance in learning generative models is the use of amortized inference distributions that are jointly trained with the models. We find that existing training objectives for variational autoencoders can lead to inaccurate amortized inference distributions and, in some cases, improving the objective provably degrades the inference quality. In addition, it has been observed that variational autoencoders tend to ignore the latent variables when combined with a decoding distribution that is too flexible. We again identify the cause in existing training criteria and propose a new class of objectives (InfoVAE) that mitigate these problems. We show that our model can significantly improve the quality of the variational posterior and can make effective use of the latent features regardless of the flexibility of the decoding distribution. Through extensive qualitative and quantitative analyses, we demonstrate that our models outperform competing approaches on multiple performance metrics.

References in corpus (3)

• Improved Training of Wasserstein GANs
• Generative Moment Matching Networks
• A Kernel Method for the Two-Sample Problem

Cited by in corpus (132)

• Recent Advances in Autoencoder-Based Representation Learning
• Deep Learning Approaches for Data Augmentation in Medical Imaging: A Review
• Learning Factorized Multimodal Representations
• Wasserstein Auto-Encoders
• Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing
• HDMI: High-order Deep Multiplex Infomax
• Entropy and mutual information in models of deep neural networks
• Isolating Sources of Disentanglement in Variational Autoencoders
• adVAE: A self-adversarial variational autoencoder with Gaussian anomaly prior knowledge for anomaly detection
• Lagging Inference Networks and Posterior Collapse in Variational Autoencoders
• Learning on Attribute-Missing Graphs
• ODE${}^2$VAE: Deep generative second order ODEs with Bayesian neural networks
• Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trends
• PI-VAE: Physics-Informed Variational Auto-Encoder for stochastic differential equations
• Conditional Flow Variational Autoencoders for Structured Sequence Prediction
• Disentangling Disentanglement in Variational Autoencoders
• Social-WaGDAT: Interaction-aware Trajectory Prediction via Wasserstein Graph Double-Attention Network
• Rethinking Softmax with Cross-Entropy: Neural Network Classifier as Mutual Information Estimator
• Simple and Effective VAE Training with Calibrated Decoders
• A Tutorial on Deep Latent Variable Models of Natural Language
• The Information Autoencoding Family: A Lagrangian Perspective on Latent Variable Generative Models
• Educating Text Autoencoders: Latent Representation Guidance via Denoising
• Auto-Encoding Total Correlation Explanation
• Deep Semi-Supervised Anomaly Detection
• Deep Generative Models for Detector Signature Simulation: A Taxonomic Review
• Fairness without the sensitive attribute via Causal Variational Autoencoder
• Information bottleneck through variational glasses
• Variational Template Machine for Data-to-Text Generation
• Bayesian Neural Networks With Maximum Mean Discrepancy Regularization
• Unsupervised Discrete Sentence Representation Learning for Interpretable Neural Dialog Generation
• Poincaré Wasserstein Autoencoder
• Finite sample properties of parametric MMD estimation: robustness to misspecification and dependence
• Towards Visually Explaining Variational Autoencoders
• Rethink Maximum Mean Discrepancy for Domain Adaptation
• Machines Learn to Infer Stellar Parameters Just by Looking at a Large Number of Spectra
• Reaching Through Latent Space: From Joint Statistics to Path Planning in Manipulation
• Degeneration in VAE: in the Light of Fisher Information Loss
• Intrinsically Motivated Discovery of Diverse Patterns in Self-Organizing Systems
• Generating Tertiary Protein Structures via an Interpretative Variational Autoencoder
• MAE: Mutual Posterior-Divergence Regularization for Variational AutoEncoders
• Effect of The Latent Structure on Clustering with GANs
• Maximizing Mutual Information for Tacotron
• DLow: Diversifying Latent Flows for Diverse Human Motion Prediction
• Learning-Aided Physical Layer Attacks Against Multicarrier Communications in IoT
• Pairwise Supervised Hashing with Bernoulli Variational Auto-Encoder and Self-Control Gradient Estimator
• Dirichlet Graph Variational Autoencoder
• Information Bottleneck and its Applications in Deep Learning
• Bounded Information Rate Variational Autoencoders
• Safe Interactive Model-Based Learning
• Counterfactual Reasoning for Fair Clinical Risk Prediction
• PCAE: A Framework of Plug-in Conditional Auto-Encoder for Controllable Text Generation
• Riemannian Normalizing Flow on Variational Wasserstein Autoencoder for Text Modeling
• AI Fairness via Domain Adaptation
• Flow Models for Arbitrary Conditional Likelihoods
• InfoCatVAE: Representation Learning with Categorical Variational Autoencoders
• A Unified Joint Maximum Mean Discrepancy for Domain Adaptation
• PointAtrousGraph: Deep Hierarchical Encoder-Decoder with Point Atrous Convolution for Unorganized 3D Points
• Assisted Sound Sample Generation with Musical Conditioning in Adversarial Auto-Encoders
• Learning Hierarchical Priors in VAEs
• Multimodal Generative Models for Compositional Representation Learning
• Pythae: Unifying Generative Autoencoders in Python -- A Benchmarking Use Case
• Semantics Preserving Adversarial Learning
• A deep generative model for gene expression profiles from single-cell RNA sequencing
• Conditional deep surrogate models for stochastic, high-dimensional, and multi-fidelity systems
• Preventing Posterior Collapse with Levenshtein Variational Autoencoder
• SHOT-VAE: Semi-supervised Deep Generative Models With Label-aware ELBO Approximations
• KinePose: A temporally optimized inverse kinematics technique for 6DOF human pose estimation with biomechanical constraints
• Implicit Deep Latent Variable Models for Text Generation
• mu-Forcing: Training Variational Recurrent Autoencoders for Text Generation
• Generative Dual Adversarial Network for Generalized Zero-shot Learning
• A Variational Prosody Model for Mapping the Context-Sensitive Variation of Functional Prosodic Prototypes
• Spatio-Temporal Graph Dual-Attention Network for Multi-Agent Prediction and Tracking
• A Simple Framework for Uncertainty in Contrastive Learning
• A Batch Normalized Inference Network Keeps the KL Vanishing Away
• APo-VAE: Text Generation in Hyperbolic Space
• LumièreNet: Lecture Video Synthesis from Audio
• Atlas Based Segmentations via Semi-Supervised Diffeomorphic Registrations
• Agent Modelling under Partial Observability for Deep Reinforcement Learning
• Learning Image Representations for Content Based Image Retrieval of Radiotherapy Treatment Plans
• Batch norm with entropic regularization turns deterministic autoencoders into generative models
• HAVANA: Hierarchical and Variation-Normalized Autoencoder for Person Re-identification
• Learning Global and Local Features of Normal Brain Anatomy for Unsupervised Abnormality Detection
• Wasserstein Autoencoders for Collaborative Filtering
• Improved Variational Neural Machine Translation by Promoting Mutual Information
• Improving the Robustness of QA Models to Challenge Sets with Variational Question-Answer Pair Generation
• Infomax Neural Joint Source-Channel Coding via Adversarial Bit Flip
• On the Transformation of Latent Space in Autoencoders
• Dueling Decoders: Regularizing Variational Autoencoder Latent Spaces
• "Best-of-Many-Samples" Distribution Matching
• Adversarial Networks and Autoencoders: The Primal-Dual Relationship and Generalization Bounds
• Learning to Find Correlated Features by Maximizing Information Flow in Convolutional Neural Networks
• Dispersed Exponential Family Mixture VAEs for Interpretable Text Generation
• Investigating and Improving Latent Density Segmentation Models for Aleatoric Uncertainty Quantification in Medical Imaging
• Learning Product Codebooks using Vector Quantized Autoencoders for Image Retrieval
• Joint haze image synthesis and dehazing with mmd-vae losses
• MIM: Mutual Information Machine
• Exploiting Chain Rule and Bayes' Theorem to Compare Probability Distributions
• Characterizing and Avoiding Problematic Global Optima of Variational Autoencoders
• Fast and Efficient Scene Categorization for Autonomous Driving using VAEs
• No Representation without Transformation
• MaskAAE: Latent space optimization for Adversarial Auto-Encoders
• KernelNet: A Data-Dependent Kernel Parameterization for Deep Generative Modeling
• Controllable Gradient Item Retrieval
• D2C: Diffusion-Denoising Models for Few-shot Conditional Generation
• Scalable Modeling of Spatiotemporal Data using the Variational Autoencoder: an Application in Glaucoma
• VMI-VAE: Variational Mutual Information Maximization Framework for VAE With Discrete and Continuous Priors
• Variational Mutual Information Maximization Framework for VAE Latent Codes with Continuous and Discrete Priors
• Understanding VAEs in Fisher-Shannon Plane
• SGVAE: Sequential Graph Variational Autoencoder
• Unsupervised Representation Adversarial Learning Network: from Reconstruction to Generation
• Exploring Autoencoder-based Error-bounded Compression for Scientific Data
• Adaptation and Generalization for Unknown Sensitive Factors of Variations
• Decoupled Variational Embedding for Signed Directed Networks
• Automatic Feature Extraction for Heartbeat Anomaly Detection
• Group Equivariant Subsampling
• A Surprisingly Effective Fix for Deep Latent Variable Modeling of Text
• $\rho$-VAE: Autoregressive parametrization of the VAE encoder
• Neural Network Classifier as Mutual Information Evaluator
• Learning from Multiple Sources for Data-to-Text and Text-to-Data
• Informative GANs via Structured Regularization of Optimal Transport
• Boosting Summarization with Normalizing Flows and Aggressive Training
• The Representation Jensen-Rényi Divergence
• Deep Sequence Learning for Video Anticipation: From Discrete and Deterministic to Continuous and Stochastic
• The Variational InfoMax Learning Objective
• Node Attribute Generation on Graphs
• Noise Contrastive Variational Autoencoders
• Causality Learning With Wasserstein Generative Adversarial Networks
• Improve variational autoEncoder with auxiliary softmax multiclassifier
• Bayesian Distributional Policy Gradients
• Longitudinal Self-Supervised Learning
• Adversarial Canonical Correlation Analysis
• In search of the weirdest galaxies in the Universe