Multilogue-Net: A Context Aware RNN for Multi-modal Emotion Detection and Sentiment Analysis in Conversation

Abstract

Sentiment Analysis and Emotion Detection in conversation is key in several real-world applications, with an increase in modalities available aiding a better understanding of the underlying emotions. Multi-modal Emotion Detection and Sentiment Analysis can be particularly useful, as applications will be able to use specific subsets of available modalities, as per the available data. Current systems dealing with Multi-modal functionality fail to leverage and capture - the context of the conversation through all modalities, the dependency between the listener(s) and speaker emotional states, and the relevance and relationship between the available modalities. In this paper, we propose an end to end RNN architecture that attempts to take into account all the mentioned drawbacks. Our proposed model, at the time of writing, out-performs the state of the art on a benchmark dataset on a variety of accuracy and regression metrics.

arXiv comment10 pages, 3 figures, 5 tables; Published in Proceedings of the Second Grand Challenge and Workshop on Multimodal Language (Challenge-HML) in the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020)

References in corpus (5)

• Adam: A Method for Stochastic Optimization
• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos
• Memory Fusion Network for Multi-view Sequential Learning

Cited by in corpus (10)

• Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representations
• GraphMFT: A Graph Network based Multimodal Fusion Technique for Emotion Recognition in Conversation
• Trustworthy Multimodal Fusion for Sentiment Analysis in Ordinal Sentiment Space
• A Review of Text Style Transfer using Deep Learning
• Detecting Hate Speech in Multi-modal Memes
• Conversation Understanding using Relational Temporal Graph Neural Networks with Auxiliary Cross-Modality Interaction
• Analyzing Unaligned Multimodal Sequence via Graph Convolution and Graph Pooling Fusion
• Improving Humanness of Virtual Agents and Users' Cooperation through Emotions
• Which is Making the Contribution: Modulating Unimodal and Cross-modal Dynamics for Multimodal Sentiment Analysis
• Modulated Fusion using Transformer for Linguistic-Acoustic Emotion Recognition