A Transformer-Based Model With Self-Distillation for Multimodal Emotion Recognition in Conversations

Abstract

Emotion recognition in conversations (ERC), the task of recognizing the emotion of each utterance in a conversation, is crucial for building empathetic machines. Existing studies focus mainly on capturing context- and speaker-sensitive dependencies on the textual modality but ignore the significance of multimodal information. Different from emotion recognition in textual conversations, capturing intra- and inter-modal interactions between utterances, learning weights between different modalities, and enhancing modal representations play important roles in multimodal ERC. In this paper, we propose a transformer-based model with self-distillation (SDT) for the task. The transformer-based model captures intra- and inter-modal interactions by utilizing intra- and inter-modal transformers, and learns weights between modalities dynamically by designing a hierarchical gated fusion strategy. Furthermore, to learn more expressive modal representations, we treat soft labels of the proposed model as extra training supervision. Specifically, we introduce self-distillation to transfer knowledge of hard and soft labels from the proposed model to each modality. Experiments on IEMOCAP and MELD datasets demonstrate that SDT outperforms previous state-of-the-art baselines.

13 pages, 10 figures. Accepted by IEEE Transactions on Multimedia (TMM)

References in corpus (7)

• Adam: A Method for Stochastic Optimization
• Distilling the Knowledge in a Neural Network
• FitNets: Hints for Thin Deep Nets
• Leveraging Recent Advances in Deep Learning for Audio-Visual Emotion Recognition
• Feature-map-level Online Adversarial Knowledge Distillation
• HiGRU: Hierarchical Gated Recurrent Units for Utterance-level Emotion Recognition
• Sparse Fusion for Multimodal Transformers

Cited by in corpus (1)

• Relational graph-driven differential denoising and diffusion attention fusion for multimodal conversation emotion recognition