Bimodal Speech Emotion Recognition Using Pre-Trained Language Models

Abstract

Speech emotion recognition is a challenging task and an important step towards more natural human-machine interaction. We show that pre-trained language models can be fine-tuned for text emotion recognition, achieving an accuracy of 69.5% on Task 4A of SemEval 2017, improving upon the previous state of the art by over 3% absolute. We combine these language models with speech emotion recognition, achieving results of 73.5% accuracy when using provided transcriptions and speech data on a subset of four classes of the IEMOCAP dataset. The use of noise-induced transcriptions and speech data results in an accuracy of 71.4%. For our experiments, we created IEmoNet, a modular and adaptable bimodal framework for speech emotion recognition based on pre-trained language models. Lastly, we discuss the idea of using an emotional classifier as a reward for reinforcement learning as a step towards more successful and convenient human-machine interaction.

Life-Long Learning for Spoken Language Systems ASRU 2019

References in corpus (7)

• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Multi-Modal Emotion recognition on IEMOCAP Dataset using Deep Learning
• On the Choice of Modeling Unit for Sequence-to-Sequence Speech Recognition
• Improving speech emotion recognition via Transformer-based Predictive Coding through transfer learning
• A novel pyramidal-FSMN architecture with lattice-free MMI for speech recognition
• Incremental processing of noisy user utterances in the spoken language understanding task

Cited by in corpus (3)

• Language Models as Emotional Classifiers for Textual Conversations
• Multi-Modal Emotion Detection with Transfer Learning
• Hierachical Delta-Attention Method for Multimodal Fusion