Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages

Abstract

We introduce the Universal Speech Model (USM), a single large model that performs automatic speech recognition (ASR) across 100+ languages. This is achieved by pre-training the encoder of the model on a large unlabeled multilingual dataset of 12 million (M) hours spanning over 300 languages, and fine-tuning on a smaller labeled dataset. We use multilingual pre-training with random-projection quantization and speech-text modality matching to achieve state-of-the-art performance on downstream multilingual ASR and speech-to-text translation tasks. We also demonstrate that despite using a labeled training set 1/7-th the size of that used for the Whisper model, our model exhibits comparable or better performance on both in-domain and out-of-domain speech recognition tasks across many languages.

arXiv comment20 pages, 7 figures, 8 tables

Cited by in corpus (9)

• Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Prompting
• Multilingual DistilWhisper: Efficient Distillation of Multi-task Speech Models via Language-Specific Experts
• Large Language Model Based Generative Error Correction: A Challenge and Baselines for Speech Recognition, Speaker Tagging, and Emotion Recognition
• LLM-based speaker diarization correction: A generalizable approach
• Parameter-Efficient Learning for Text-to-Speech Accent Adaptation
• Adapting an ASR Foundation Model for Spoken Language Assessment
• Improving vision-inspired keyword spotting using dynamic module skipping in streaming conformer encoder
• Adapting an Unadaptable ASR System
• SpeechColab Leaderboard: An Open-Source Platform for Automatic Speech Recognition Evaluation