Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers

Abstract

In this paper, we focus on Whisper, a recent automatic speech recognition model trained with a massive 680k hour labeled speech corpus recorded in diverse conditions. We first show an interesting finding that while Whisper is very robust against real-world background sounds (e.g., music), its audio representation is actually not noise-invariant, but is instead highly correlated to non-speech sounds, indicating that Whisper recognizes speech conditioned on the noise type. With this finding, we build a unified audio tagging and speech recognition model Whisper-AT by freezing the backbone of Whisper, and training a lightweight audio tagging model on top of it. With <1% extra computational cost, Whisper-AT can recognize audio events, in addition to spoken text, in a single forward pass.

arXiv commentAccepted at Interspeech 2023. Code at https://github.com/yuangongnd/whisper-at

References in corpus (8)

• Adam: A Method for Stochastic Optimization
• Robust Speech Recognition via Large-Scale Weak Supervision
• Libri-Light: A Benchmark for ASR with Limited or No Supervision
• Attention Bottlenecks for Multimodal Fusion
• Masked Autoencoders that Listen
• Invariant Representations for Noisy Speech Recognition
• HEAR: Holistic Evaluation of Audio Representations
• A Noise-Robust Self-supervised Pre-training Model Based Speech Representation Learning for Automatic Speech Recognition

Cited by in corpus (5)

• Who Said What? An Automated Approach to Analyzing Speech in Preschool Classrooms
• Speech foundation models on intelligibility prediction for hearing-impaired listeners
• From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Data
• Loudspeaker Beamforming to Enhance Speech Recognition Performance of Voice Driven Applications
• ASR Personalization in Pathological Voice: Accuracy, Efficiency, and Duration Effects