Speech SIMCLR: Combining Contrastive and Reconstruction Objective for Self-supervised Speech Representation Learning

Abstract

Self-supervised visual pretraining has shown significant progress recently. Among those methods, SimCLR greatly advanced the state of the art in self-supervised and semi-supervised learning on ImageNet. The input feature representations for speech and visual tasks are both continuous, so it is natural to consider applying similar objective on speech representation learning. In this paper, we propose Speech SimCLR, a new self-supervised objective for speech representation learning. During training, Speech SimCLR applies augmentation on raw speech and its spectrogram. Its objective is the combination of contrastive loss that maximizes agreement between differently augmented samples in the latent space and reconstruction loss of input representation. The proposed method achieved competitive results on speech emotion recognition and speech recognition.

References in corpus (14)

• A Simple Framework for Contrastive Learning of Visual Representations
• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• A Comparative Study on Transformer vs RNN in Speech Applications
• Libri-Light: A Benchmark for ASR with Limited or No Supervision
• Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
• TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
• vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
• Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognition
• Improving Transformer-based Speech Recognition Using Unsupervised Pre-training
• Effectiveness of self-supervised pre-training for speech recognition
• Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation
• A Further Study of Unsupervised Pre-training for Transformer Based Speech Recognition
• Transformer based unsupervised pre-training for acoustic representation learning

Cited by in corpus (8)

• Exploring Self-Supervised Representation Ensembles for COVID-19 Cough Classification
• PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition
• Conditional independence for pretext task selection in Self-supervised speech representation learning
• Injecting Text in Self-Supervised Speech Pretraining
• Comparison of Self-Supervised Speech Pre-Training Methods on Flemish Dutch
• TENET: A Time-reversal Enhancement Network for Noise-robust ASR
• Addressing Feature Suppression in Unsupervised Visual Representations
• Pretext Tasks selection for multitask self-supervised speech representation learning