Joint Multimodal Learning with Deep Generative Models

Abstract

We investigate deep generative models that can exchange multiple modalities bi-directionally, e.g., generating images from corresponding texts and vice versa. Recently, some studies handle multiple modalities on deep generative models, such as variational autoencoders (VAEs). However, these models typically assume that modalities are forced to have a conditioned relation, i.e., we can only generate modalities in one direction. To achieve our objective, we should extract a joint representation that captures high-level concepts among all modalities and through which we can exchange them bi-directionally. As described herein, we propose a joint multimodal variational autoencoder (JMVAE), in which all modalities are independently conditioned on joint representation. In other words, it models a joint distribution of modalities. Furthermore, to be able to generate missing modalities from the remaining modalities properly, we develop an additional method, JMVAE-kl, that is trained by reducing the divergence between JMVAE's encoder and prepared networks of respective modalities. Our experiments show that our proposed method can obtain appropriate joint representation from multiple modalities and that it can generate and reconstruct them more properly than conventional VAEs. We further demonstrate that JMVAE can generate multiple modalities bi-directionally.

Cited by in corpus (27)

• Think Locally, Act Globally: Federated Learning with Local and Global Representations
• Multi-modal Deep Analysis for Multimedia
• Patchwork Learning: A Paradigm Towards Integrative Analysis across Diverse Biomedical Data Sources
• A Whole Brain Probabilistic Generative Model: Toward Realizing Cognitive Architectures for Developmental Robots
• Imitation and Mirror Systems in Robots through Deep Modality Blending Networks
• A multimodal dynamical variational autoencoder for audiovisual speech representation learning
• Gaussian Process Prior Variational Autoencoders
• Bayesian Structural Model Updating with Multimodal Variational Autoencoder
• Disentangling by Partitioning: A Representation Learning Framework for Multimodal Sensory Data
• JADE: Joint Autoencoders for Dis-Entanglement
• SMIL: Multimodal Learning with Severely Missing Modality
• lamBERT: Language and Action Learning Using Multimodal BERT
• Private-Shared Disentangled Multimodal VAE for Learning of Hybrid Latent Representations
• CMTS: Conditional Multiple Trajectory Synthesizer for Generating Safety-critical Driving Scenarios
• Improving Normative Modeling for Multi-modal Neuroimaging Data using mixture-of-product-of-experts variational autoencoders
• Multimodal Generative Models for Compositional Representation Learning
• Disentangled VAE Representations for Multi-Aspect and Missing Data
• MHVAE: a Human-Inspired Deep Hierarchical Generative Model for Multimodal Representation Learning
• Cross-modal Variational Auto-encoder with Distributed Latent Spaces and Associators
• Variational Selective Autoencoder: Learning from Partially-Observed Heterogeneous Data
• Symbol Emergence as an Interpersonal Multimodal Categorization
• Hierarchical Bayesian Model for the Transfer of Knowledge on Spatial Concepts based on Multimodal Information
• Multi-Source Neural Variational Inference
• On the Limitations of Multimodal VAEs
• Learning more expressive joint distributions in multimodal variational methods
• Variational learning across domains with triplet information
• Recommending Burgers based on Pizza Preferences: Addressing Data Sparsity with a Product of Experts