Learning Factorized Multimodal Representations

Abstract

Learning multimodal representations is a fundamentally complex research problem due to the presence of multiple heterogeneous sources of information. Although the presence of multiple modalities provides additional valuable information, there are two key challenges to address when learning from multimodal data: 1) models must learn the complex intra-modal and cross-modal interactions for prediction and 2) models must be robust to unexpected missing or noisy modalities during testing. In this paper, we propose to optimize for a joint generative-discriminative objective across multimodal data and labels. We introduce a model that factorizes representations into two sets of independent factors: multimodal discriminative and modality-specific generative factors. Multimodal discriminative factors are shared across all modalities and contain joint multimodal features required for discriminative tasks such as sentiment prediction. Modality-specific generative factors are unique for each modality and contain the information required for generating data. Experimental results show that our model is able to learn meaningful multimodal representations that achieve state-of-the-art or competitive performance on six multimodal datasets. Our model demonstrates flexible generative capabilities by conditioning on independent factors and can reconstruct missing modalities without significantly impacting performance. Lastly, we interpret our factorized representations to understand the interactions that influence multimodal learning.

arXiv commentICLR 2019

References in corpus (16)

• Zero-Shot Learning Through Cross-Modal Transfer
• Deep Convolutional Inverse Graphics Network
• InfoVAE: Information Maximizing Variational Autoencoders
• Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis
• A kernel for time series based on global alignments
• Wasserstein Auto-Encoders
• Multimodal Machine Learning: A Survey and Taxonomy
• Discovering Hidden Factors of Variation in Deep Networks
• Gated-Attention Architectures for Task-Oriented Language Grounding
• Efficient Low-rank Multimodal Fusion with Modality-Specific Factors
• Bayesian representation learning with oracle constraints
• Improving One-Shot Learning through Fusing Side Information
• Learning Robust Visual-Semantic Embeddings
• On the Latent Space of Wasserstein Auto-Encoders
• Disentangling by Partitioning: A Representation Learning Framework for Multimodal Sensory Data
• "Dependency Bottleneck" in Auto-encoding Architectures: an Empirical Study

Cited by in corpus (42)

• Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
• Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directions
• Learning Language-guided Adaptive Hyper-modality Representation for Multimodal Sentiment Analysis
• UR-FUNNY: A Multimodal Language Dataset for Understanding Humor
• A survey of multimodal deep generative models
• Unlocking the Emotional World of Visual Media: An Overview of the Science, Research, and Impact of Understanding Emotion
• MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis
• Patchwork Learning: A Paradigm Towards Integrative Analysis across Diverse Biomedical Data Sources
• Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors
• MSAF: Multimodal Split Attention Fusion
• Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities
• Graph Capsule Aggregation for Unaligned Multimodal Sequences
• Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysis
• Multimodal Contrastive Learning via Uni-Modal Coding and Cross-Modal Prediction for Multimodal Sentiment Analysis
• MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
• Modality to Modality Translation: An Adversarial Representation Learning and Graph Fusion Network for Multimodal Fusion
• DialogueTRM: Exploring the Intra- and Inter-Modal Emotional Behaviors in the Conversation
• Improving Multimodal Accuracy Through Modality Pre-training and Attention
• Two-Faced Humans on Twitter and Facebook: Harvesting Social Multimedia for Human Personality Profiling
• Towards Open-World Feature Extrapolation: An Inductive Graph Learning Approach
• Relating by Contrasting: A Data-efficient Framework for Multimodal Generative Models
• Maximum Likelihood Estimation for Multimodal Learning with Missing Modality
• Integrating Multimodal Information in Large Pretrained Transformers
• Neural Methods for Point-wise Dependency Estimation
• Dizygotic Conditional Variational AutoEncoder for Multi-Modal and Partial Modality Absent Few-Shot Learning
• A Plug-in Method for Representation Factorization in Connectionist Models
• Multimodal Local-Global Ranking Fusion for Emotion Recognition
• Modality-Transferable Emotion Embeddings for Low-Resource Multimodal Emotion Recognition
• Learning Multimodal VAEs through Mutual Supervision
• Video Sentiment Analysis with Bimodal Information-augmented Multi-Head Attention
• MTAG: Modal-Temporal Attention Graph for Unaligned Human Multimodal Language Sequences
• Multimodal End-to-End Sparse Model for Emotion Recognition
• Unsupervised Multimodal Language Representations using Convolutional Autoencoders
• TEASEL: A Transformer-Based Speech-Prefixed Language Model
• Multimodal Embeddings from Language Models
• Instance Explainable Temporal Network For Multivariate Timeseries
• A Dataset and Benchmarks for Multimedia Social Analysis
• Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion
• Multimodal Language Analysis with Recurrent Multistage Fusion
• An Empirical Evaluation of Sketched SVD and its Application to Leverage Score Ordering
• Which is Making the Contribution: Modulating Unimodal and Cross-modal Dynamics for Multimodal Sentiment Analysis
• On the Limitations of Multimodal VAEs