2.2k citations · 3.7k across the 33 of their papers we have counts for
37 papers
Compute-Optimal Quantization-Aware Training
Aleksandr Dremov, David Grangier, Angelos Katharopoulos +1
Quantization-aware training (QAT) is a leading technique for improving the accuracy of quantized neural networks. Previous work has shown that decomposing training into a full-prec…
Need a Small Specialized Language Model? Plan Early!
David Grangier, Angelos Katharopoulos, Pierre Ablin +1
Large language models are versatile tools but are not suitable for small inference budgets. Small models have more efficient inference, but their lower capacity means that their pe…
Adaptive Training Distributions with Scalable Online Bilevel Optimization
David Grangier, Pierre Ablin, Awni Hannun
Large neural networks pretrained on web-scale corpora are central to modern machine learning. In this paradigm, the distribution of the large, heterogeneous pretraining data rarely…
Transfer Learning for Structured Pruning under Limited Task Data
Lucio Dery, David Grangier, Awni Hannun
Large, pre-trained models are problematic to use in resource constrained applications. Fortunately, task-aware structured pruning methods offer a solution. These approaches reduce…
Star Temporal Classification: Sequence Classification with Partially Labeled Data
Vineel Pratap, Awni Hannun, Gabriel Synnaeve +1
We develop an algorithm which can learn from partially labeled and unsegmented sequential data. Most sequential loss functions, such as Connectionist Temporal Classification (CTC),…
Flashlight: Enabling Innovation in Tools for Machine Learning
Jacob Kahn, Vineel Pratap, Tatiana Likhomanenko +11
As the computational requirements for machine learning systems and the size and complexity of machine learning frameworks increases, essential framework innovation has become chall…
Parallel Composition of Weighted Finite-State Transducers
Shubho Sengupta, Vineel Pratap, Awni Hannun
Finite-state transducers (FSTs) are frequently used in speech recognition. Transducer composition is an essential operation for combining different sources of information at differ…
CrypTen: Secure Multi-Party Computation Meets Machine Learning
Brian Knott, Shobha Venkataraman, Awni Hannun +3
Secure multi-party computation (MPC) allows parties to perform computations on data while keeping that data private. This capability has great potential for machine-learning applic…
The History of Speech Recognition to the Year 2030
Awni Hannun
The decade from 2010 to 2020 saw remarkable improvements in automatic speech recognition. Many people now use speech recognition on a daily basis, for example to perform voice sear…
The Role of Evolution in Machine Intelligence
Awni Hannun
Machine intelligence can develop either directly from experience or by inheriting experience through evolution. The bulk of current research efforts focus on algorithms which learn…
Gradient Matching for Domain Generalization
Yuge Shi, Jeffrey Seely, Philip H. S. Torr +4
Machine learning systems typically assume that the distributions of training and test sets match closely. However, a critical requirement of such systems in the real world is their…
Fixes That Fail: Self-Defeating Improvements in Machine-Learning Systems
Ruihan Wu, Chuan Guo, Awni Hannun +1
Machine-learning systems such as self-driving cars or virtual assistants are composed of a large number of machine-learning models that recognize image content, transcribe speech,…
Measuring Data Leakage in Machine-Learning Models with Fisher Information
Awni Hannun, Chuan Guo, Laurens van der Maaten
Machine-learning models contain information about the data they were trained on. This information leaks either through the model itself or through predictions made by the model. Co…
Data Appraisal Without Data Sharing
Mimee Xu, Laurens van der Maaten, Awni Hannun
One of the most effective approaches to improving the performance of a machine learning model is to procure additional training data. A model owner seeking relevant training data f…
Differentiable Weighted Finite-State Transducers
Awni Hannun, Vineel Pratap, Jacob Kahn +1
We introduce a framework for automatic differentiation with weighted finite-state transducers (WFSTs) allowing them to be used dynamically at training time. Through the separation…
The Trade-Offs of Private Prediction
Laurens van der Maaten, Awni Hannun
Machine learning models leak information about their training data every time they reveal a prediction. This is problematic when the training data needs to remain private. Private…
Massively Multilingual ASR: 50 Languages, 1 Model, 1 Billion Parameters
Vineel Pratap, Anuroop Sriram, Paden Tomasello +4
We study training a single acoustic model for multiple languages with the aim of improving automatic speech recognition (ASR) performance on low-resource languages, and over-all si…
Iterative Pseudo-Labeling for Speech Recognition
Qiantong Xu, Tatiana Likhomanenko, Jacob Kahn +3
Pseudo-labeling has recently shown promise in end-to-end automatic speech recognition (ASR). We study Iterative Pseudo-Labeling (IPL), a semi-supervised algorithm which efficiently…
Semi-Supervised Speech Recognition via Local Prior Matching
Wei-Ning Hsu, Ann Lee, Gabriel Synnaeve +1
For sequence transduction tasks like speech recognition, a strong structured prior model encodes rich information about the target space, implicitly ruling out invalid sequences by…
Scaling Up Online Speech Recognition Using ConvNets
Vineel Pratap, Qiantong Xu, Jacob Kahn +6
We design an online end-to-end speech recognition system based on Time-Depth Separable (TDS) convolutions and Connectionist Temporal Classification (CTC). We improve the core TDS a…
Secure multiparty computations in floating-point arithmetic
Chuan Guo, Awni Hannun, Brian Knott +3
Secure multiparty computations enable the distribution of so-called shares of sensitive data to multiple parties such that the multiple parties can effectively process the data whi…
Certified Data Removal from Machine Learning Models
Chuan Guo, Tom Goldstein, Awni Hannun +1
Good data stewardship requires removal of data at the request of the data's owner. This raises the question if and how a trained machine-learning model, which implicitly stores inf…
Lead2Gold: Towards exploiting the full potential of noisy transcriptions for speech recognition
Adrien Dufraux, Emmanuel Vincent, Awni Hannun +2
The transcriptions used to train an Automatic Speech Recognition (ASR) system may contain errors. Usually, either a quality control stage discards transcriptions with too many erro…
Privacy-Preserving Multi-Party Contextual Bandits
Awni Hannun, Brian Knott, Shubho Sengupta +1
Contextual bandits are online learners that, given an input, select an arm and receive a reward for that arm. They use the reward as a learning signal and aim to maximize the total…