155 citations · 277 across the 7 of their papers we have counts for
8 papers
SNOO: Step-K Nesterov Outer Optimizer - The Surprising Effectiveness of Nesterov Momentum Applied to Pseudo-Gradients
Dominik Kallusky, Vinay Rao, Vishal Nandavanam +1
The rapid development of large language models (LLMs) has driven the demand for more efficient optimization techniques. Among these, the Lookahead family of optimizers employs a tw…
WebRED: Effective Pretraining And Finetuning For Relation Extraction On The Web
Robert Ormandi, Mohammad Saleh, Erin Winter +1
Relation extraction is used to populate knowledge bases that are important to many applications. Prior datasets used to train relation extraction models either suffer from noisy la…
Is Batch Norm unique? An empirical investigation and prescription to emulate the best properties of common normalizers without batch dependence
Vinay Rao, Jascha Sohl-Dickstein
We perform an extensive empirical study of the statistical properties of Batch Norm and other common normalizers. This includes an examination of the correlation between representa…
Assessing The Factual Accuracy of Generated Text
Ben Goodrich, Vinay Rao, Mohammad Saleh +1
We propose a model-based metric to estimate the factual accuracy of generated text that is complementary to typical scoring schemes like ROUGE (Recall-Oriented Understudy for Gisti…
A Mean Field Theory of Batch Normalization
Greg Yang, Jeffrey Pennington, Vinay Rao +2
We develop a mean field theory for batch normalization in fully-connected feedforward neural networks. In so doing, we provide a precise characterization of signal propagation and…
Reducing Bias in Production Speech Models
Eric Battenberg, Rewon Child, Adam Coates +13
Replacing hand-engineered pipelines with end-to-end deep learning systems has enabled strong results in applications like speech and object recognition. However, the causality and…
Active Learning for Speech Recognition: the Power of Gradients
Jiaji Huang, Rewon Child, Vinay Rao +3
In training speech recognition systems, labeling audio clips can be expensive, and not all data is equally valuable. Active learning aims to label only the most informative samples…
Deep Speech: Scaling up end-to-end speech recognition
Awni Hannun, Carl Case, Jared Casper +8
We present a state-of-the-art speech recognition system developed using end-to-end deep learning. Our architecture is significantly simpler than traditional speech systems, which r…