Emotion Recognition in Speech using Cross-Modal Transfer in the Wild

Abstract

Obtaining large, human labelled speech datasets to train models for emotion recognition is a notoriously challenging task, hindered by annotation cost and label ambiguity. In this work, we consider the task of learning embeddings for speech classification without access to any form of labelled audio. We base our approach on a simple hypothesis: that the emotional content of speech correlates with the facial expression of the speaker. By exploiting this relationship, we show that annotations of expression can be transferred from the visual domain (faces) to the speech domain (voices) through cross-modal distillation. We make the following contributions: (i) we develop a strong teacher network for facial emotion recognition that achieves the state of the art on a standard benchmark; (ii) we use the teacher to train a student, tabula rasa, to learn representations (embeddings) for speech emotion recognition without access to labelled audio data; and (iii) we show that the speech emotion embedding can be used for speech emotion recognition on external benchmark datasets. Code, models and data are available.

arXiv commentConference paper at ACM Multimedia 2018

References in corpus (12)

• Distilling the Knowledge in a Neural Network
• VoxCeleb: a large-scale speaker identification dataset
• Do Deep Nets Really Need to be Deep?
• Return of the Devil in the Details: Delving Deep into Convolutional Nets
• Unsupervised Visual Representation Learning by Context Prediction
• SoundNet: Learning Sound Representations from Unlabeled Video
• Self-supervised learning of a facial attribute embedding from video
• Challenges in Representation Learning: A report on three machine learning contests
• Moonshine: Distilling with Cheap Convolutions
• See, Hear, and Read: Deep Aligned Representations
• Look, Listen and Learn
• On Learning Associations of Faces and Voices

Cited by in corpus (6)

• Improving Multi-Modal Learning with Uni-Modal Teachers
• MIMAMO Net: Integrating Micro- and Macro-motion for Video Emotion Recognition
• WiFE: WiFi and Vision based Intelligent Facial-Gesture Emotion Recognition
• Speech Fusion to Face: Bridging the Gap Between Human's Vocal Characteristics and Facial Imaging
• Indian EmoSpeech Command Dataset: A dataset for emotion based speech recognition in the wild
• XFlow: Cross-modal Deep Neural Networks for Audiovisual Classification