Self-Supervised Variational Auto-Encoders
arXiv:2010.02014 · doi:10.3390/e23060747
Abstract
Density estimation, compression and data generation are crucial tasks in artificial intelligence. Variational Auto-Encoders (VAEs) constitute a single framework to achieve these goals. Here, we present a novel class of generative models, called self-supervised Variational Auto-Encoder (selfVAE), that utilizes deterministic and discrete variational posteriors. This class of models allows to perform both conditional and unconditional sampling, while simplifying the objective function. First, we use a single self-supervised transformation as a latent variable, where a transformation is either downscaling or edge detection. Next, we consider a hierarchical architecture, i.e., multiple transformations, and we show its benefits compared to the VAE. The flexibility of selfVAE in data reconstruction finds a particularly interesting use case in data compression tasks, where we can trade-off memory for better data quality, and vice-versa. We present performance of our approach on three benchmark image data (Cifar10, Imagenette64, and CelebA).
arXiv comment19 pages, 14 figures, 2 tables
References in corpus (27)
- Adam: A Method for Stochastic Optimization
- Denoising Diffusion Probabilistic Models
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- Stochastic Backpropagation and Approximate Inference in Deep Generative Models
- Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
- Densely Connected Convolutional Networks
- Improved Training of Wasserstein GANs
- Self-supervised Learning: Generative or Contrastive
- Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
- Data-Efficient Image Recognition with Contrastive Predictive Coding
- Density estimation using Real NVP
- ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
- NVAE: A Deep Hierarchical Variational Autoencoder
- Image Super-Resolution Using Very Deep Residual Channel Attention Networks
- Auxiliary Deep Generative Models
- Variational Lossy Autoencoder
- Importance Weighted Autoencoders
- Video Compression With Rate-Distortion Autoencoders
- Glow: Generative Flow with Invertible 1x1 Convolutions
- BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling
- Sylvester Normalizing Flows for Variational Inference
- VAE with a VampPrior
- Distribution Matching in Variational Inference
- Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images
- Self-Supervised Variational Auto-Encoders
- Super-resolution Variational Auto-Encoders
- The Convolution Exponential and Generalized Sylvester Flows