Multi-Modal Emotion recognition on IEMOCAP Dataset using Deep Learning

Abstract

Emotion recognition has become an important field of research in Human Computer Interactions as we improve upon the techniques for modelling the various aspects of behaviour. With the advancement of technology our understanding of emotions are advancing, there is a growing need for automatic emotion recognition systems. One of the directions the research is heading is the use of Neural Networks which are adept at estimating complex functions that depend on a large number and diverse source of input data. In this paper we attempt to exploit this effectiveness of Neural networks to enable us to perform multimodal Emotion recognition on IEMOCAP dataset using data from Speech, Text, and Motion capture data from face expressions, rotation and hand movements. Prior research has concentrated on Emotion detection from Speech on the IEMOCAP dataset, but our approach is the first that uses the multiple modes of data offered by IEMOCAP for a more robust and accurate emotion detection.

References in corpus (3)

• Convolutional Neural Networks for Sentence Classification
• Learning to Generate Reviews and Discovering Sentiment
• Reusing Neural Speech Representations for Auditory Emotion Recognition

Cited by in corpus (14)

• Speech Emotion Recognition Using Multi-hop Attention Mechanism
• Emotion recognition by fusing time synchronous and time asynchronous representations
• Deep Learning based Emotion Recognition System Using Speech Features and Transcriptions
• Two-stage dimensional emotion recognition by fusing predictions of acoustic and text networks using SVM
• A cross-modal fusion network based on self-attention and residual structure for multimodal emotion recognition
• Recent Advances and Trends in Multimodal Deep Learning: A Review
• A Transfer Learning Method for Speech Emotion Recognition from Automatic Speech Recognition
• Cross Lingual Cross Corpus Speech Emotion Recognition
• Language Models as Emotional Classifiers for Textual Conversations
• Bimodal Speech Emotion Recognition Using Pre-Trained Language Models
• Multi-modal Attention for Speech Emotion Recognition
• Multi-Window Data Augmentation Approach for Speech Emotion Recognition
• Multi-Modal Emotion Detection with Transfer Learning
• Detecting Interlocutor Confusion in Situated Human-Avatar Dialogue: A Pilot Study