Recent Advances in Autoencoder-Based Representation Learning

Abstract

Learning useful representations with little or no supervision is a key challenge in artificial intelligence. We provide an in-depth review of recent advances in representation learning with a focus on autoencoder-based models. To organize these results we make use of meta-priors believed useful for downstream tasks, such as disentanglement and hierarchical organization of features. In particular, we uncover three main mechanisms to enforce such properties, namely (i) regularizing the (approximate or aggregate) posterior distribution, (ii) factorizing the encoding and decoding distribution, or (iii) introducing a structured prior distribution. While there are some promising results, implicit or explicit supervision remains a key enabler and all current methods use strong inductive biases and modeling assumptions. Finally, we provide an analysis of autoencoder-based representation learning through the lens of rate-distortion theory and identify a clear tradeoff between the amount of prior knowledge available about the downstream tasks, and how useful the representation is for this task.

Presented at the third workshop on Bayesian Deep Learning (NeurIPS 2018)

References in corpus (33)

• Auto-Encoding Variational Bayes
• Categorical Reparameterization with Gumbel-Softmax
• Neural Discrete Representation Learning
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Tutorial on Variational Autoencoders
• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
• Deep Convolutional Inverse Graphics Network
• Adversarial Feature Learning
• Disentangling by Factorising
• Estimating divergence functionals and the likelihood ratio by convex risk minimization
• Adversarially Learned Inference
• Autoencoding beyond pixels using a learned similarity metric
• Decomposing Motion and Content for Natural Video Sequence Prediction
• InfoVAE: Information Maximizing Variational Autoencoders
• Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations
• Deep Variational Information Bottleneck
• Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm
• Variational Lossy Autoencoder
• Disentangling factors of variation in deep representations using adversarial training
• Variational Approaches for Auto-Encoding Generative Adversarial Networks
• Unsupervised Learning of Disentangled Representations from Video
• Composing graphical models with neural networks for structured representations and fast inference
• Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data
• Toward Controlled Generation of Text
• Variational inference for Monte Carlo objectives
• Learning Disentangled Representations with Semi-Supervised Deep Generative Models
• Information Constraints on Auto-Encoding Variational Bayes
• Disentangled Sequential Autoencoder
• Learning Disentangled Joint Continuous and Discrete Representations
• TherML: Thermodynamics of Machine Learning
• Detach and Adapt: Learning Cross-Domain Disentangled Deep Representation

Cited by in corpus (81)

• Deep Learning in Medical Image Registration: A Review
• On Mutual Information Maximization for Representation Learning
• Self-supervised learning methods and applications in medical imaging analysis: A survey
• ForensicTransfer: Weakly-supervised Domain Adaptation for Forgery Detection
• A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
• Are Disentangled Representations Helpful for Abstract Visual Reasoning?
• Combining a Convolutional Neural Network with Autoencoders to Predict the Survival Chance of COVID-19 Patients
• Review: Deep Learning in Electron Microscopy
• ODE${}^2$VAE: Deep generative second order ODEs with Bayesian neural networks
• Weakly-Supervised Disentanglement Without Compromises
• Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trends
• Disentangling Factors of Variation Using Few Labels
• ParSNIP: Generative Models of Transient Light Curves with Physics-Enabled Deep Learning
• Machine Learning with Confidential Computing: A Systematization of Knowledge
• Hierarchical Quantized Autoencoders
• Universal Rate-Distortion-Perception Representations for Lossy Compression
• Separating physically distinct mechanisms in complex infrared plasmonic nanostructures via machine learning enhanced electron energy loss spectroscopy
• Machines Learn to Infer Stellar Parameters Just by Looking at a Large Number of Spectra
• Exploiting auto-encoders and segmentation methods for middle-level explanations of image classification systems
• A Semantic-based Medical Image Fusion Approach
• Autoencoding Variational Autoencoder
• Intrinsically Motivated Discovery of Diverse Patterns in Self-Organizing Systems
• Exploring the Landscape of Ubiquitous In-home Health Monitoring: A Comprehensive Survey
• Contrastively Disentangled Sequential Variational Autoencoder
• Theory and Evaluation Metrics for Learning Disentangled Representations
• MultiCapCLIP: Auto-Encoding Prompts for Zero-Shot Multilingual Visual Captioning
• Measuring the Discrepancy between Conditional Distributions: Methods, Properties and Applications
• VAEs in the Presence of Missing Data
• On the Road with 16 Neurons: Mental Imagery with Bio-inspired Deep Neural Networks
• Disentangling Influence: Using Disentangled Representations to Audit Model Predictions
• Disentangled Representations from Non-Disentangled Models
• On the Fairness of Disentangled Representations
• Disentangling and Learning Robust Representations with Natural Clustering
• Improving ClusterGAN Using Self-Augmented Information Maximization of Disentangling Latent Spaces
• CSNNs: Unsupervised, Backpropagation-free Convolutional Neural Networks for Representation Learning
• An A* Curriculum Approach to Reinforcement Learning for RGBD Indoor Robot Navigation
• Augmenting Variational Autoencoders with Sparse Labels: A Unified Framework for Unsupervised, Semi-(un)supervised, and Supervised Learning
• Device Authentication Codes based on RF Fingerprinting using Deep Learning
• On the Compressive Power of Boolean Threshold Autoencoders
• Multi-Task Variational Information Bottleneck
• Data-efficient visuomotor policy training using reinforcement learning and generative models
• Target-Embedding Autoencoders for Supervised Representation Learning
• On Disentanglement in Gaussian Process Variational Autoencoders
• Disentangling representations of retinal images with generative models
• On the Size and Width of the Decoder of a Boolean Threshold Autoencoder
• A Causal Lens for Peeking into Black Box Predictive Models: Predictive Model Interpretation via Causal Attribution
• PRI-VAE: Principle-of-Relevant-Information Variational Autoencoders
• Adversarial Disentanglement with Grouped Observations
• HAVANA: Hierarchical and Variation-Normalized Autoencoder for Person Re-identification
• Flexible Variational Information Bottleneck: Achieving Diverse Compression with a Single Training
• White-Box Adversarial Defense via Self-Supervised Data Estimation
• The State of Lifelong Learning in Service Robots: Current Bottlenecks in Object Perception and Manipulation
• Annotation Cost Reduction of Stream-based Active Learning by Automated Weak Labeling using a Robot Arm
• Gaussian Process Bandit Optimization of the Thermodynamic Variational Objective
• Random Polytope Descriptors
• Looking for change? Roll the Dice and demand Attention
• Double cycle-consistent generative adversarial network for unsupervised conditional generation
• A Meta-learning Approach to Reservoir Computing: Time Series Prediction with Limited Data
• Constrained Deep One-Class Feature Learning For Classifying Imbalanced Medical Images
• Quantum Patch-Based Autoencoder for Anomaly Segmentation
• Entropy-based Active Learning of Graph Neural Network Surrogate Models for Materials Properties
• A manifold learning perspective on representation learning: Learning decoder and representations without an encoder
• Tourbillon: a Physically Plausible Neural Architecture
• Odd-One-Out Representation Learning
• Mixture Representation Learning with Coupled Autoencoders
• Particle Filter Bridge Interpolation
• Universal Physiological Representation Learning with Soft-Disentangled Rateless Autoencoders
• An Interactive Insight Identification and Annotation Framework for Power Grid Pixel Maps using DenseU-Hierarchical VAE
• $\rho$-VAE: Autoregressive parametrization of the VAE encoder
• Multi-Decoder RNN Autoencoder Based on Variational Bayes Method
• Time Series Source Separation with Slow Flows
• Weakly Supervised Recovery of Semantic Attributes
• Vector Embeddings with Subvector Permutation Invariance using a Triplet Enhanced Autoencoder
• Towards Learning Generalizable Driving Policies from Restricted Latent Representations
• Improving Relational Regularized Autoencoders with Spherical Sliced Fused Gromov Wasserstein
• Gradient flow encoding with distance optimization adaptive step size
• Neuronal Learning Analysis using Cycle-Consistent Adversarial Networks
• DAReN: A Collaborative Approach Towards Reasoning And Disentangling
• Polyline Generative Navigable Space Segmentation for Autonomous Visual Navigation
• Self-Supervised Encoder for Fault Prediction in Electrochemical Cells
• Towards Better Understanding of Disentangled Representations via Mutual Information