Publications (37)
Seq2Edits: Sequence Transduction Using Span-level Edit Operations
Felix Stahlberg, Shankar Kumar
We propose Seq2Edits, an open-vocabulary approach to sequence editing for natural language processing (NLP) tasks with a high degree of overlap between input and output texts. In t…
Gemini Embedding 2: A Native Multimodal Embedding Model from Gemini
Madhuri Shanbhogue, Zhe Li, Shanfeng Zhang +86
We introduce Gemini Embedding 2, a native multimodal embedding model that allows embedding video, audio, image, and text modalities in a unified representation space. We leverage t…
Large Scale Language Modeling in Automatic Speech Recognition
Ciprian Chelba, Dan Bikel, Maria Shugrina +2
Large language models have been proven quite beneficial for a variety of automatic speech recognition tasks in Google. We summarize results on Voice Search and a few YouTube speech…
Corpora Generation for Grammatical Error Correction
Jared Lichtarge, Chris Alberti, Shankar Kumar +3
Grammatical Error Correction (GEC) has been recently modeled using the sequence-to-sequence framework. However, unlike sequence transduction problems such as machine translation, G…
Long-Form Speech Translation through Segmentation with Finite-State Decoding Constraints on Large Language Models
Arya D. McCarthy, Hao Zhang, Shankar Kumar +2
One challenge in speech translation is that plenty of spoken content is long-form, but short units are necessary for obtaining high-quality translations. To address this mismatch,…
Text Generation with Text-Editing Models
Eric Malmi, Yue Dong, Jonathan Mallinson +7
Text-editing models have recently become a prominent alternative to seq2seq models for monolingual text-generation tasks such as grammatical error correction, simplification, and s…
Uncertainty Determines the Adequacy of the Mode and the Tractability of Decoding in Sequence-to-Sequence Models
Felix Stahlberg, Ilia Kulikov, Shankar Kumar
In many natural language processing (NLP) tasks the same input (e.g. source sentence) can have multiple possible outputs (e.g. translations). To analyze how this ambiguity (also kn…
Measuring Re-identification Risk
CJ Carey, Travis Dick, Alessandro Epasto +8
Compact user representations (such as embeddings) form the backbone of personalization services. In this work, we present a new theoretical framework to measure re-identification r…
Heterogeneous Federated Learning Using Knowledge Codistillation
Jared Lichtarge, Ehsan Amid, Shankar Kumar +3
Federated Averaging, and many federated learning algorithm variants which build upon it, have a limitation: all clients must share the same model architecture. This results in unus…
Semantic Segmentation with Bidirectional Language Models Improves Long-form ASR
W. Ronny Huang, Hao Zhang, Shankar Kumar +2
We propose a method of segmenting long-form speech by separating semantically complete sentences within the utterance. This prevents the ASR decoder from needlessly processing fara…
Conciseness: An Overlooked Language Task
Felix Stahlberg, Aashish Kumar, Chris Alberti +1
We report on novel investigations into training models that make sentences concise. We define the task and show that it is different from related tasks such as summarization and si…
Towards an On-device Agent for Text Rewriting
Yun Zhu, Yinxiao Liu, Felix Stahlberg +7
Large Language Models (LLMs) have demonstrated impressive capabilities for text rewriting. Nonetheless, the large sizes of these models make them impractical for on-device inferenc…
Jam or Cream First? Modeling Ambiguity in Neural Machine Translation with SCONES
Felix Stahlberg, Shankar Kumar
The softmax layer in neural machine translation is designed to model the distribution over mutually exclusive tokens. Machine translation, however, is intrinsically uncertain: the…
Improved Long-Form Spoken Language Translation with Large Language Models
Arya D. McCarthy, Hao Zhang, Shankar Kumar +2
A challenge in spoken language translation is that plenty of spoken content is long-form, but short units are necessary for obtaining high-quality translations. To address this mis…
Data Weighted Training Strategies for Grammatical Error Correction
Jared Lichtarge, Chris Alberti, Shankar Kumar
Recent progress in the task of Grammatical Error Correction (GEC) has been driven by addressing data sparsity, both through new methods for generating large and noisy pretraining d…
Lookup-Table Recurrent Language Models for Long Tail Speech Recognition
W. Ronny Huang, Tara N. Sainath, Cal Peyser +3
We introduce Lookup-Table Language Models (LookupLM), a method for scaling up the size of RNN language models with only a constant increase in the floating point operations, by inc…
Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling
Jonathan Shen, Patrick Nguyen, Yonghui Wu +88
Lingvo is a Tensorflow framework offering a complete solution for collaborative deep learning research, with a particular focus towards sequence-to-sequence models. Lingvo models a…
NN-grams: Unifying neural network and n-gram language models for Speech Recognition
Babak Damavandi, Shankar Kumar, Noam Shazeer +1
We present NN-grams, a novel, hybrid language model integrating n-grams and neural networks (NN) for speech recognition. The model takes as input both word histories as well as n-g…
Lattice Rescoring Strategies for Long Short Term Memory Language Models in Speech Recognition
Shankar Kumar, Michael Nirschl, Daniel Holtmann-Rice +3
Recurrent neural network (RNN) language models (LMs) and Long Short Term Memory (LSTM) LMs, a variant of RNN LMs, have been shown to outperform traditional N-gram LMs on speech rec…
No Need for a Lexicon? Evaluating the Value of the Pronunciation Lexica in End-to-End Models
Tara N. Sainath, Rohit Prabhavalkar, Shankar Kumar +9
For decades, context-dependent phonemes have been the dominant sub-word unit for conventional acoustic modeling systems. This status quo has begun to be challenged recently by end-…
Scaling Language Model Size in Cross-Device Federated Learning
Jae Hun Ro, Theresa Breiner, Lara McConnaughey +4
Most studies in cross-device federated learning focus on small models, due to the server-client communication and on-device computation bottlenecks. In this work, we leverage vario…
Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss
Qian Zhang, Han Lu, Hasim Sak +4
In this paper we present an end-to-end speech recognition model with Transformer encoders that can be used in a streaming speech recognition system. Transformer computation blocks…
Retrieval Augmented Question Answering: When Should LLMs Admit Ignorance?
Dingmin Wang, Ji Ma, Shankar Kumar
The success of expanded context windows in Large Language Models (LLMs) has driven increased use of broader context in retrieval-augmented generation. We investigate the use of LLM…
Spelling Correction through Rewriting of Non-Autoregressive ASR Lattices
Leonid Velikovich, Christopher Li, Diamantino Caseiro +4
For end-to-end Automatic Speech Recognition (ASR) models, recognizing personal or rare phrases can be hard. A promising way to improve accuracy is through spelling correction (or r…
Capitalization Normalization for Language Modeling with an Accurate and Efficient Hierarchical RNN Model
Hao Zhang, You-Chi Cheng, Shankar Kumar +3
Capitalization normalization (truecasing) is the task of restoring the correct case (uppercase or lowercase) of noisy text. We propose a fast, accurate and compact two-level hierar…
Massive Sound Embedding Benchmark (MSEB)
Georg Heigold, Ehsan Variani, Tom Bagby +4
Audio is a critical component of multimodal perception, and any truly intelligent system must demonstrate a wide range of auditory capabilities. These capabilities include transcri…
Neural Language Modeling with Visual Features
Antonios Anastasopoulos, Shankar Kumar, Hank Liao
Multimodal language models attempt to incorporate non-linguistic features for the language modeling task. In this work, we extend a standard recurrent neural network (RNN) language…
Transformer-based Models of Text Normalization for Speech Applications
Jae Hun Ro, Felix Stahlberg, Ke Wu +1
Text normalization, or the process of transforming text into a consistent, canonical form, is crucial for speech applications such as text-to-speech synthesis (TTS). In TTS, the sy…
Position-Invariant Truecasing with a Word-and-Character Hierarchical Recurrent Neural Network
Hao Zhang, You-Chi Cheng, Shankar Kumar +2
Truecasing is the task of restoring the correct case (uppercase or lowercase) of noisy text generated either by an automatic system for speech recognition or machine translation or…
Synthetic Data Generation for Grammatical Error Correction with Tagged Corruption Models
Felix Stahlberg, Shankar Kumar
Synthetic data generation is widely known to boost the accuracy of neural grammatical error correction (GEC) systems, but existing methods often lack diversity or are too simplisti…
Predicting Compact Phrasal Rewrites with Large Language Models for ASR Post Editing
Hao Zhang, Felix Stahlberg, Shankar Kumar
Large Language Models (LLMs) excel at rewriting tasks such as text style transfer and grammatical error correction. While there is considerable overlap between the inputs and outpu…
Simple and Effective Gradient-Based Tuning of Sequence-to-Sequence Models
Jared Lichtarge, Chris Alberti, Shankar Kumar
Recent trends towards training ever-larger language models have substantially improved machine learning performance across linguistic tasks. However, the huge cost of training larg…
Dynamic Subset Tuning: Expanding the Operational Range of Parameter-Efficient Training for Large Language Models
Felix Stahlberg, Jared Lichtarge, Shankar Kumar
We propose a novel parameter-efficient training (PET) method for large language models that adapts models to downstream tasks by optimizing a small subset of the existing model par…
Sentence-Select: Large-Scale Language Model Data Selection for Rare-Word Speech Recognition
W. Ronny Huang, Cal Peyser, Tara N. Sainath +3
Language model fusion helps smart assistants recognize words which are rare in acoustic data but abundant in text-only corpora (typed search logs). However, such corpora have prope…
Improving Tail Performance of a Deliberation E2E ASR Model Using a Large Text Corpus
Cal Peyser, Sepand Mavandadi, Tara N. Sainath +3
End-to-end (E2E) automatic speech recognition (ASR) systems lack the distinct language model (LM) component that characterizes traditional speech systems. While this simplifies the…
Weakly Supervised Grammatical Error Correction using Iterative Decoding
Jared Lichtarge, Christopher Alberti, Shankar Kumar +2
We describe an approach to Grammatical Error Correction (GEC) that is effective at making use of models trained on large amounts of weakly supervised bitext. We train the Transform…
Multilingual Open Relation Extraction Using Cross-lingual Projection
Manaal Faruqui, Shankar Kumar
Open domain relation extraction systems identify relation and argument phrases in a sentence without relying on any underlying schema. However, current state-of-the-art relation ex…