Robust Speech Recognition via Large-Scale Weak Supervision

Abstract

We study the capabilities of speech processing systems trained simply to predict large amounts of transcripts of audio on the internet. When scaled to 680,000 hours of multilingual and multitask supervision, the resulting models generalize well to standard benchmarks and are often competitive with prior fully supervised results but in a zero-shot transfer setting without the need for any fine-tuning. When compared to humans, the models approach their accuracy and robustness. We are releasing models and inference code to serve as a foundation for further work on robust speech processing.

Cited by in corpus (70)

• Intelligent Clinical Documentation: Harnessing Generative AI for Patient-Centric Clinical Note Generation
• Reproducible scaling laws for contrastive language-image learning
• ChatGPT for Shaping the Future of Dentistry: The Potential of Multi-Modal Large Language Model
• 14 Examples of How LLMs Can Transform Materials Science and Chemistry: A Reflection on a Large Language Model Hackathon
• GPT Models in Construction Industry: Opportunities, Limitations, and a Use Case Validation
• Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers
• BOMs Away! Inside the Minds of Stakeholders: A Comprehensive Study of Bills of Materials for Software Systems
• A Survey on State-of-the-art Deep Learning Applications and Challenges
• Mobile Foundation Model as Firmware
• Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognition
• The Future of Intelligent Healthcare: A Systematic Analysis and Discussion on the Integration and Impact of Robots Using Large Language Models for Healthcare
• PEFT-SER: On the Use of Parameter Efficient Transfer Learning Approaches For Speech Emotion Recognition Using Pre-trained Speech Models
• N-best T5: Robust ASR Error Correction using Multiple Input Hypotheses and Constrained Decoding Space
• Augmented Datasheets for Speech Datasets and Ethical Decision-Making
• Drastic Circuit Depth Reductions with Preserved Adversarial Robustness by Approximate Encoding for Quantum Machine Learning
• Context-aware attention layers coupled with optimal transport domain adaptation and multimodal fusion methods for recognizing dementia from spontaneous speech
• UniCATS: A Unified Context-Aware Text-to-Speech Framework with Contextual VQ-Diffusion and Vocoding
• Design and Evaluation of a Socially Assistive Robot Schoolwork Companion for College Students with ADHD
• DRAGON: A Dialogue-Based Robot for Assistive Navigation with Visual Language Grounding
• Cross-lingual Alzheimer's Disease detection based on paralinguistic and pre-trained features
• Using Think-Aloud Data to Understand Relations between Self-Regulation Cycle Characteristics and Student Performance in Intelligent Tutoring Systems
• A Statistical Framework of Watermarks for Large Language Models: Pivot, Detection Efficiency and Optimal Rules
• A Few-Shot Approach to Dysarthric Speech Intelligibility Level Classification Using Transformers
• PixIT: Joint Training of Speaker Diarization and Speech Separation from Real-world Multi-speaker Recordings
• Alzheimer Disease Classification through ASR-based Transcriptions: Exploring the Impact of Punctuation and Pauses
• A Parameter-Efficient Learning Approach to Arabic Dialect Identification with Pre-Trained General-Purpose Speech Model
• LLM-Mediated Domain-Specific Voice Agents: The Case of TextileBot
• SpeechCraft: A Fine-grained Expressive Speech Dataset with Natural Language Description
• Relax: Composable Abstractions for End-to-End Dynamic Machine Learning
• Automated speech audiometry: Can it work using open-source pre-trained Kaldi-NL automatic speech recognition?
• LanSER: Language-Model Supported Speech Emotion Recognition
• VISION: A Modular AI Assistant for Natural Human-Instrument Interaction at Scientific User Facilities
• Group-level Brain Decoding with Deep Learning
• LLM-based speaker diarization correction: A generalizable approach
• Considerations for Ethical Speech Recognition Datasets
• A Multimodal Approach to Device-Directed Speech Detection with Large Language Models
• Enhancing Supermarket Robot Interaction: A Multi-Level LLM Conversational Interface for Handling Diverse Customer Intents
• Adapting an ASR Foundation Model for Spoken Language Assessment
• ITALIC: An Italian Intent Classification Dataset
• OverFlow: Putting flows on top of neural transducers for better TTS
• How to Estimate Model Transferability of Pre-Trained Speech Models?
• Leveraging Semantic Information for Efficient Self-Supervised Emotion Recognition with Audio-Textual Distilled Models
• Steganography Beyond Space-Time with Chain of Multimodal AI
• Advancing Audio Emotion and Intent Recognition with Large Pre-Trained Models and Bayesian Inference
• Comparing Apples to Oranges: LLM-powered Multimodal Intention Prediction in an Object Categorization Task
• BERSting at the Screams: A Benchmark for Distanced, Emotional and Shouted Speech Recognition
• Adapting an Unadaptable ASR System
• Single-Channel Robot Ego-Speech Filtering during Human-Robot Interaction
• The Face of Populism: Examining Differences in Facial Emotional Expressions of Political Leaders Using Machine Learning
• GenAI Voice Mode in Programming Education
• Parameter Selection for Analyzing Conversations with Autism Spectrum Disorder
• Toxicity Begets Toxicity: Unraveling Conversational Chains in Political Podcasts
• Open-Ended Multi-Modal Relational Reasoning for Video Question Answering
• Transfer Learning of Transformer-based Speech Recognition Models from Czech to Slovak
• Student Engagement with GenAI's Tutoring Feedback: A Mixed Methods Study
• An Adapter-Based Unified Model for Multiple Spoken Language Processing Tasks
• Remote Inference of Cognitive Scores in ALS Patients Using a Picture Description
• PSST! Prosodic Speech Segmentation with Transformers
• Computational analyses of linguistic features with schizophrenic and autistic traits along with formal thought disorders
• On minimal variations for unsupervised representation learning
• A Multistakeholder Approach to Value-Driven Co-Design of Recommender System Evaluation Metrics in Digital Archives
• torchdistill Meets Hugging Face Libraries for Reproducible, Coding-Free Deep Learning Studies: A Case Study on NLP
• Learning Co-Speech Gesture for Multimodal Aphasia Type Detection
• Automatic Speech Recognition for Sanskrit with Transfer Learning
• Segment-Level Vectorized Beam Search Based on Partially Autoregressive Inference
• AVIN-Chat: An Audio-Visual Interactive Chatbot System with Emotional State Tuning
• NUTS, NARS, and Speech
• Understanding Personalization Gains in Whisper ASR
• Modular Speech-to-Text Translation for Zero-Shot Cross-Modal Transfer
• Investigating self-supervised features for expressive, multilingual voice conversion