Variational Autoencoder for Deep Learning of Images, Labels and Captions

Abstract

A novel variational autoencoder is developed to model images, as well as associated labels or captions. The Deep Generative Deconvolutional Network (DGDN) is used as a decoder of the latent image features, and a deep Convolutional Neural Network (CNN) is used as an image encoder; the CNN is used to approximate a distribution for the latent DGDN features/code. The latent code is also linked to generative models for labels (Bayesian support vector machine) or captions (recurrent neural network). When predicting a label/caption for a new image at test, averaging is performed across the distribution of latent codes; this is computationally efficient as a consequence of the learned CNN-based encoder. Since the framework is capable of modeling the image in the presence/absence of associated labels/captions, a new semi-supervised setting is manifested for CNN learning with images; the framework even allows unsupervised CNN learning, based on images alone.

arXiv commentNIPS 2016 (To appear)

References in corpus (11)

• Auto-Encoding Variational Bayes
• Sequence to Sequence Learning with Neural Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Semi-Supervised Learning with Deep Generative Models
• Theano: new features and speed improvements
• Deep Convolutional Inverse Graphics Network
• Semi-Supervised Learning with Ladder Networks
• Neural Variational Inference and Learning in Belief Networks
• Max-margin Deep Generative Models
• Learning to Generate Chairs, Tables and Cars with Convolutional Networks
• Generative Deep Deconvolutional Learning

Cited by in corpus (88)

• An Introduction to Variational Autoencoders
• MixMatch: A Holistic Approach to Semi-Supervised Learning
• Realistic Evaluation of Deep Semi-Supervised Learning Algorithms
• Multimodal Generative Models for Scalable Weakly-Supervised Learning
• Efficient Self-supervised Vision Transformers for Representation Learning
• In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning
• From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI
• PasteGAN: A Semi-Parametric Method to Generate Image from Scene Graph
• Topic-Guided Variational Autoencoders for Text Generation
• Generative Models of Visually Grounded Imagination
• An advanced hybrid deep adversarial autoencoder for parameterized nonlinear fluid flow modelling
• 6GCVAE: Gated Convolutional Variational Autoencoder for IPv6 Target Generation
• Deep Co-Training for Semi-Supervised Image Recognition
• EM-like Learning Chaotic Dynamics from Noisy and Partial Observations
• JointGAN: Multi-Domain Joint Distribution Learning with Generative Adversarial Nets
• Neural Language Generation: Formulation, Methods, and Evaluation
• Spatial-Phase Shallow Learning: Rethinking Face Forgery Detection in Frequency Domain
• Decision-Making with Auto-Encoding Variational Bayes
• Quantum Receiver Enhanced by Adaptive Learning
• Learning Layout and Style Reconfigurable GANs for Controllable Image Synthesis
• RETRIEVE: Coreset Selection for Efficient and Robust Semi-Supervised Learning
• Mixing autoencoder with classifier: conceptual data visualization
• Semi-supervised learning based on generative adversarial network: a comparison between good GAN and bad GAN approach
• Deep-learning-assisted reconfigurable metasurface antenna for real-time holographic beam steering
• Robust Variational Autoencoder
• Seq2Seq2Sentiment: Multimodal Sequence to Sequence Models for Sentiment Analysis
• ContCap: A scalable framework for continual image captioning
• Regularizing linear inverse problems with convolutional neural networks
• Leveraging the Exact Likelihood of Deep Latent Variable Models
• Towards Phytoplankton Parasite Detection Using Autoencoders
• Statistical Inference in Mean-Field Variational Bayes
• Wasserstein Variational Inference
• Blind Image Denoising and Inpainting Using Robust Hadamard Autoencoders
• Iranis: A Large-scale Dataset of Farsi License Plate Characters
• Quantum autoencoders using mixed reference states
• Domain Embedded Multi-model Generative Adversarial Networks for Image-based Face Inpainting
• Sequential Graph Convolutional Network for Active Learning
• Probabilistic Programming with Programmable Variational Inference
• Persuasive Faces: Generating Faces in Advertisements
• Multi-Task Curriculum Framework for Open-Set Semi-Supervised Learning
• Variational Autoencoders for Jet Simulation
• Factor Analysis, Probabilistic Principal Component Analysis, Variational Inference, and Variational Autoencoder: Tutorial and Survey
• On Attribution of Deepfakes
• Negative sampling in semi-supervised learning
• Modality Conversion of Handwritten Patterns by Cross Variational Autoencoders
• Repetitive Reprediction Deep Decipher for Semi-Supervised Learning
• Label Propagation Adaptive Resonance Theory for Semi-supervised Continuous Learning
• DeltaGAN: Towards Diverse Few-shot Image Generation with Sample-Specific Delta
• Gaussian Smoothen Semantic Features (GSSF) -- Exploring the Linguistic Aspects of Visual Captioning in Indian Languages (Bengali) Using MSCOCO Framework
• An Empirical Study of Language CNN for Image Captioning
• Modeling Eye Gaze Velocity Trajectories using GANs with Spectral Loss for Enhanced Fidelity
• Copula-based Sensitivity Analysis for Multi-Treatment Causal Inference with Unobserved Confounding
• Skill Transfer in Deep Reinforcement Learning under Morphological Heterogeneity
• Physics-Informed Learning for High Impedance Faults Detection
• Exploring Global Information for Session-based Recommendation
• Generating various airfoil shapes with required lift coefficient using conditional variational autoencoders
• Semi-supervised Learning using Adversarial Training with Good and Bad Samples
• General-Purpose User Embeddings based on Mobile App Usage
• F2GAN: Fusing-and-Filling GAN for Few-shot Image Generation
• Anatomy-Aware Cardiac Motion Estimation
• Addressing Feature Suppression in Unsupervised Visual Representations
• Evaluating for Diversity in Question Generation over Text
• DO-GAN: A Double Oracle Framework for Generative Adversarial Networks
• Learning a Hierarchical Latent-Variable Model of 3D Shapes
• Autoencoding Improves Pre-trained Word Embeddings
• An ETF view of Dropout regularization
• Deep generative demixing: Recovering Lipschitz signals from noisy subgaussian mixtures
• Learning a Representation Map for Robot Navigation using Deep Variational Autoencoder
• Improving latent variable descriptiveness with AutoGen
• Data Augmentation via Mixed Class Interpolation using Cycle-Consistent Generative Adversarial Networks Applied to Cross-Domain Imagery
• Regularized Sequential Latent Variable Models with Adversarial Neural Networks
• Sum-Product-Attention Networks: Leveraging Self-Attention in Probabilistic Circuits
• Reinforcement Evolutionary Learning Method for self-learning
• Learn Proportional Derivative Controllable Latent Space from Pixels
• Generative Memorize-Then-Recall framework for low bit-rate Surveillance Video Compression
• Learning to Conceal: A Deep Learning Based Method for Preserving Privacy and Avoiding Prejudice
• Variational Autoencoded Regression: High Dimensional Regression of Visual Data on Complex Manifold
• Towards Stable Imbalanced Data Classification via Virtual Big Data Projection
• Decoupled Variational Embedding for Signed Directed Networks
• Learning more expressive joint distributions in multimodal variational methods
• Generative Low-Shot Network Expansion
• That looks interesting! Personalizing Communication and Segmentation with Random Forest Node Embeddings
• Loosely Coupled Federated Learning Over Generative Models
• Tackling Long-Tailed Relations and Uncommon Entities in Knowledge Graph Completion
• Image to Language Understanding: Captioning approach
• Enhanced Variational Inference with Dyadic Transformation
• Learning a Generative Model of Cancer Metastasis
• Learning Sparsity of Representations with Discrete Latent Variables