A survey of multimodal deep generative models

Abstract

Multimodal learning is a framework for building models that make predictions based on different types of modalities. Important challenges in multimodal learning are the inference of shared representations from arbitrary modalities and cross-modal generation via these representations; however, achieving this requires taking the heterogeneous nature of multimodal data into account. In recent years, deep generative models, i.e., generative models in which distributions are parameterized by deep neural networks, have attracted much attention, especially variational autoencoders, which are suitable for accomplishing the above challenges because they can consider heterogeneity and infer good representations of data. Therefore, various multimodal generative models based on variational autoencoders, called multimodal deep generative models, have been proposed in recent years. In this paper, we provide a categorized survey of studies on multimodal deep generative models.

arXiv commentPublished in Advanced Robotics

References in corpus (12)

• Pyro: Deep Universal Probabilistic Programming
• A Survey of Multi-View Representation Learning
• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Learning Factorized Multimodal Representations
• Self-Supervised MultiModal Versatile Networks
• Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models
• Audio-visual Speech Enhancement Using Conditional Variational Auto-Encoders
• Generative Models of Visually Grounded Imagination
• Neuro-SERKET: Development of Integrative Cognitive System through the Composition of Deep Probabilistic Generative Models
• A Whole Brain Probabilistic Generative Model: Toward Realizing Cognitive Architectures for Developmental Robots
• Coordination Among Neural Modules Through a Shared Global Workspace
• Variational Inference via $\chi$-Upper Bound Minimization

Cited by in corpus (7)

• Interpretable and intervenable ultrasonography-based machine learning models for pediatric appendicitis
• Creativity and Machine Learning: A Survey
• A multimodal dynamical variational autoencoder for audiovisual speech representation learning
• Bridging Language, Vision and Action: Multimodal VAEs in Robotic Manipulation Tasks
• On Parallelism in Music and Language: A Perspective from Symbol Emergence Systems based on Probabilistic Generative Models
• An Overview on Generative AI at Scale with Edge-Cloud Computing
• Instruction-Driven 3D Facial Expression Generation and Transition