Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling

Abstract

Lingvo is a Tensorflow framework offering a complete solution for collaborative deep learning research, with a particular focus towards sequence-to-sequence models. Lingvo models are composed of modular building blocks that are flexible and easily extensible, and experiment configurations are centralized and highly customizable. Distributed training and quantized inference are supported directly within the framework, and it contains existing implementations of a large number of utilities, helper functions, and the newest research ideas. Lingvo has been used in collaboration by dozens of researchers in more than 20 papers over the last two years. This document outlines the underlying design of Lingvo and serves as an introduction to the various pieces of the framework, while also offering examples of advanced features that showcase the capabilities of the framework.

References in corpus (2)

• Hierarchical Generative Modeling for Controllable Speech Synthesis
• Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translation

Cited by in corpus (68)

• SpeechBrain: A General-Purpose Speech Toolkit
• Conformer: Convolution-augmented Transformer for Speech Recognition
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
• Personalizing ASR for Dysarthric and Accented Speech with Limited Data
• The Missing Ingredient in Zero-Shot Neural Machine Translation
• fairseq S2T: Fast Speech-to-Text Modeling with fairseq
• Julia Language in Machine Learning: Algorithms, Applications, and Open Issues
• ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context
• Optimizing Speech Recognition For The Edge
• Gmail Smart Compose: Real-Time Assisted Writing
• Massively Multilingual Neural Machine Translation
• On the Choice of Modeling Unit for Sequence-to-Sequence Speech Recognition
• Leveraging Monolingual Data with Self-Supervision for Multilingual Neural Machine Translation
• Scalable Second Order Optimization for Deep Learning
• Localising In-Domain Adaptation of Transformer-Based Biomedical Language Models
• Tied & Reduced RNN-T Decoder
• From English to More Languages: Parameter-Efficient Model Reprogramming for Cross-Lingual Speech Recognition
• Dual-mode ASR: Unify and Improve Streaming ASR with Full-context Modeling
• Direct speech-to-speech translation with a sequence-to-sequence model
• Espresso: A Fast End-to-end Neural Speech Recognition Toolkit
• Simple, Scalable Adaptation for Neural Machine Translation
• Multi-head Monotonic Chunkwise Attention For Online Speech Recognition
• Scaling End-to-End Models for Large-Scale Multilingual ASR
• A comparison of end-to-end models for long-form speech recognition
• Faster Transformer Decoding: N-gram Masked Self-Attention
• RNN-T Models Fail to Generalize to Out-of-Domain Audio: Causes and Solutions
• Controlling Computation versus Quality for Neural Sequence Models
• Identifying Audio Adversarial Examples via Anomalous Pattern Detection
• Primer: Searching for Efficient Transformers for Language Modeling
• Human-Paraphrased References Improve Neural Machine Translation
• Revealing and Protecting Labels in Distributed Training
• ESPnet-ST: All-in-One Speech Translation Toolkit
• Unified End-to-End Speech Recognition and Endpointing for Fast and Efficient Speech Systems
• WaveGuard: Understanding and Mitigating Audio Adversarial Examples
• Disentangling Adaptive Gradient Methods from Learning Rates
• Towards Resistant Audio Adversarial Examples
• Textual Echo Cancellation
• FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization
• Streaming Object Detection for 3-D Point Clouds
• The 2020 ESPnet update: new features, broadened applications, performance improvements, and future plans
• DELTA: A DEep learning based Language Technology plAtform
• Fill in the Blanks: Imputing Missing Sentences for Larger-Context Neural Machine Translation
• Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Model
• Practical Perspectives on Quality Estimation for Machine Translation
• Speech Sentiment Analysis via Pre-trained Features from End-to-end ASR Models
• Improving Longer-range Dialogue State Tracking
• A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latency
• Towards Fast and Accurate Streaming End-to-End ASR
• Confidence Estimation for Attention-based Sequence-to-sequence Models for Speech Recognition
• Learning Word-Level Confidence For Subword End-to-End ASR
• Language Dependencies in Adversarial Attacks on Speech Recognition Systems
• RSD-GAN: Regularized Sobolev Defense GAN Against Speech-to-Text Adversarial Attacks
• Data Troubles in Sentence Level Confidence Estimation for Machine Translation
• A Better and Faster End-to-End Model for Streaming ASR
• Multi-Task Learning for End-to-End ASR Word and Utterance Confidence with Deletion Prediction
• Rapid Domain Adaptation for Machine Translation with Monolingual Data
• CAT: A CTC-CRF based ASR Toolkit Bridging the Hybrid and the End-to-end Approaches towards Data Efficiency and Low Latency
• Deliberation Model Based Two-Pass End-to-End Speech Recognition
• Parallel Rescoring with Transformer for Streaming On-Device Speech Recognition
• Exploring Targeted Universal Adversarial Perturbations to End-to-end ASR Models
• Reinforcement Learning based Curriculum Optimization for Neural Machine Translation
• GE2E-KWS: Generalized End-to-End Training and Evaluation for Zero-shot Keyword Spotting
• Cross-attention conformer for context modeling in speech enhancement for ASR
• Transformer Based Deliberation for Two-Pass Speech Recognition
• HintedBT: Augmenting Back-Translation with Quality and Transliteration Hints