Modality-Transferable Emotion Embeddings for Low-Resource Multimodal Emotion Recognition

Abstract

Despite the recent achievements made in the multi-modal emotion recognition task, two problems still exist and have not been well investigated: 1) the relationship between different emotion categories are not utilized, which leads to sub-optimal performance; and 2) current models fail to cope well with low-resource emotions, especially for unseen emotions. In this paper, we propose a modality-transferable model with emotion embeddings to tackle the aforementioned issues. We use pre-trained word embeddings to represent emotion categories for textual data. Then, two mapping functions are learned to transfer these embeddings into visual and acoustic spaces. For each modality, the model calculates the representation distance between the input sequence and target emotions and makes predictions based on the distances. By doing so, our model can directly adapt to the unseen emotions in any modality since we have their pre-trained embeddings and modality mapping functions. Experiments show that our model achieves state-of-the-art performance on most of the emotion categories. In addition, our model also outperforms existing baselines in the zero-shot and few-shot scenarios for unseen emotions.

arXiv comment12 pages, 5 figures

References in corpus (6)

• Overcoming catastrophic forgetting in neural networks
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Overcoming Catastrophic Forgetting by Incremental Moment Matching
• Learning Factorized Multimodal Representations
• Weakly Supervised Cross-Lingual Named Entity Recognition via Effective Annotation and Representation Projection
• EmoGraph: Capturing Emotion Correlations using Graph Networks

Cited by in corpus (2)

• Weakly-supervised Multi-task Learning for Multimodal Affect Recognition
• Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization