Label-efficient audio classification through multitask learning and self-supervision

Abstract

While deep learning has been incredibly successful in modeling tasks with large, carefully curated labeled datasets, its application to problems with limited labeled data remains a challenge. The aim of the present work is to improve the label efficiency of large neural networks operating on audio data through a combination of multitask learning and self-supervised learning on unlabeled data. We trained an end-to-end audio feature extractor based on WaveNet that feeds into simple, yet versatile task-specific neural networks. We describe several easily implemented self-supervised learning tasks that can operate on any large, unlabeled audio corpus. We demonstrate that, in scenarios with limited labeled training data, one can significantly improve the performance of three different supervised classification tasks individually by up to 6% through simultaneous training with these additional self-supervised tasks. We also show that incorporating data augmentation into our multitask setting leads to even further gains in performance.

Presented at ICLR 2019 Limited Labeled Data (LLD) Workshop

References in corpus (9)

• WaveNet: A Generative Model for Raw Audio
• An Overview of Multi-Task Learning in Deep Neural Networks
• Deep Speech 2: End-to-End Speech Recognition in English and Mandarin
• VoxCeleb: a large-scale speaker identification dataset
• Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition
• Deep Speaker: an End-to-End Neural Speaker Embedding System
• Universal representations:The missing link between faces, text, planktons, and cat breeds
• Colorization as a Proxy Task for Visual Understanding
• Combining High-Level Features of Raw Audio Waves and Mel-Spectrograms for Audio Tagging

Cited by in corpus (1)

• SP-SEDT: Self-supervised Pre-training for Sound Event Detection Transformer