Semi-supervised Deep Generative Modelling of Incomplete Multi-Modality Emotional Data

Abstract

There are threefold challenges in emotion recognition. First, it is difficult to recognize human's emotional states only considering a single modality. Second, it is expensive to manually annotate the emotional data. Third, emotional data often suffers from missing modalities due to unforeseeable sensor malfunction or configuration issues. In this paper, we address all these problems under a novel multi-view deep generative framework. Specifically, we propose to model the statistical relationships of multi-modality emotional data using multiple modality-specific generative networks with a shared latent space. By imposing a Gaussian mixture assumption on the posterior approximation of the shared latent variables, our framework can learn the joint deep representation from multiple modalities and evaluate the importance of each modality simultaneously. To solve the labeled-data-scarcity problem, we extend our multi-view model to semi-supervised learning scenario by casting the semi-supervised classification problem as a specialized missing data imputation task. To address the missing-modality problem, we further extend our semi-supervised multi-view model to deal with incomplete data, where a missing view is treated as a latent variable and integrated out during inference. This way, the proposed overall framework can utilize all available (both labeled and unlabeled, as well as both complete and incomplete) data to improve its generalization ability. The experiments conducted on two real multi-modal emotion datasets demonstrated the superiority of our framework.

arXiv commentarXiv admin note: text overlap with arXiv:1704.07548, 2018 ACM Multimedia Conference (MM'18)

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Auto-Encoding Variational Bayes
• Stochastic Backpropagation and Approximate Inference in Deep Generative Models
• Semi-Supervised Learning with Deep Generative Models
• End-to-End Multimodal Emotion Recognition using Deep Neural Networks
• Matrix Completion and Low-Rank SVD via Fast Alternating Least Squares
• Auxiliary Deep Generative Models
• Ladder Variational Autoencoders
• Importance Weighted Autoencoders
• Improving Variational Inference with Inverse Autoregressive Flow
• Deep Variational Canonical Correlation Analysis
• Classification with Low Rank and Missing Data

Cited by in corpus (7)

• A Concise yet Effective model for Non-Aligned Incomplete Multi-view and Missing Multi-label Learning
• Inconsistent Matters: A Knowledge-guided Dual-consistency Network for Multi-modal Rumor Detection
• Semi-supervised Multi-modal Emotion Recognition with Cross-Modal Distribution Matching
• From the Lab to the Wild: Affect Modeling via Privileged Information
• Maximum Likelihood Estimation for Multimodal Learning with Missing Modality
• Learning Noise-Robust Joint Representation for Multimodal Emotion Recognition under Incomplete Data Scenarios
• HetMAML: Task-Heterogeneous Model-Agnostic Meta-Learning for Few-Shot Learning Across Modalities