Does Visual Self-Supervision Improve Learning of Speech Representations for Emotion Recognition?

Abstract

Self-supervised learning has attracted plenty of recent research interest. However, most works for self-supervision in speech are typically unimodal and there has been limited work that studies the interaction between audio and visual modalities for cross-modal self-supervision. This work (1) investigates visual self-supervision via face reconstruction to guide the learning of audio representations; (2) proposes an audio-only self-supervision approach for speech representation learning; (3) shows that a multi-task combination of the proposed visual and audio self-supervision is beneficial for learning richer features that are more robust in noisy conditions; (4) shows that self-supervised pretraining can outperform fully supervised training and is especially useful to prevent overfitting on smaller sized datasets. We evaluate our learned audio representations for discrete emotion recognition, continuous affect recognition and automatic speech recognition. We outperform existing self-supervised methods for all tested downstream tasks. Our results demonstrate the potential of visual self-supervision for audio feature learning and suggest that joint visual and audio self-supervision leads to more informative audio representations for speech and emotion recognition.

arXiv commentAccepted for publication in IEEE Transactions on Affective Computing; v3: Publication-ready version including additional experiments and discussion

References in corpus (10)

• Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation
• Light Gated Recurrent Units for Speech Recognition
• Deep Affect Prediction in-the-wild: Aff-Wild Database and Challenge, Deep Architectures, and Beyond
• SEWA DB: A Rich Database for Audio-Visual Emotion and Sentiment Research in the Wild
• Unsupervised speech representation learning using WaveNet autoencoders
• Affect Recognition in Ads with Application to Computational Advertising
• Evaluating Content-centric vs User-centric Ad Affect Recognition
• Looking Beyond a Clever Narrative: Visual Context and Attention are Primary Drivers of Affect in Video Advertisements
• Learning audio representations via phase prediction
• Secost: Sequential co-supervision for large scale weakly labeled audio event detection

Cited by in corpus (1)

• Self-supervised Graphs for Audio Representation Learning with Limited Labeled Data