2.2k citations · 3.7k across the 33 of their papers we have counts for
14 papers · 1 filter
Compute-Optimal Quantization-Aware Training
Aleksandr Dremov, David Grangier, Angelos Katharopoulos +1
Quantization-aware training (QAT) is a leading technique for improving the accuracy of quantized neural networks. Previous work has shown that decomposing training into a full-prec…
Need a Small Specialized Language Model? Plan Early!
David Grangier, Angelos Katharopoulos, Pierre Ablin +1
Large language models are versatile tools but are not suitable for small inference budgets. Small models have more efficient inference, but their lower capacity means that their pe…
Adaptive Training Distributions with Scalable Online Bilevel Optimization
David Grangier, Pierre Ablin, Awni Hannun
Large neural networks pretrained on web-scale corpora are central to modern machine learning. In this paradigm, the distribution of the large, heterogeneous pretraining data rarely…
Star Temporal Classification: Sequence Classification with Partially Labeled Data
Vineel Pratap, Awni Hannun, Gabriel Synnaeve +1
We develop an algorithm which can learn from partially labeled and unsegmented sequential data. Most sequential loss functions, such as Connectionist Temporal Classification (CTC),…
Flashlight: Enabling Innovation in Tools for Machine Learning
Jacob Kahn, Vineel Pratap, Tatiana Likhomanenko +11
As the computational requirements for machine learning systems and the size and complexity of machine learning frameworks increases, essential framework innovation has become chall…
CrypTen: Secure Multi-Party Computation Meets Machine Learning
Brian Knott, Shobha Venkataraman, Awni Hannun +3
Secure multi-party computation (MPC) allows parties to perform computations on data while keeping that data private. This capability has great potential for machine-learning applic…
Gradient Matching for Domain Generalization
Yuge Shi, Jeffrey Seely, Philip H. S. Torr +4
Machine learning systems typically assume that the distributions of training and test sets match closely. However, a critical requirement of such systems in the real world is their…
Fixes That Fail: Self-Defeating Improvements in Machine-Learning Systems
Ruihan Wu, Chuan Guo, Awni Hannun +1
Machine-learning systems such as self-driving cars or virtual assistants are composed of a large number of machine-learning models that recognize image content, transcribe speech,…
Measuring Data Leakage in Machine-Learning Models with Fisher Information
Awni Hannun, Chuan Guo, Laurens van der Maaten
Machine-learning models contain information about the data they were trained on. This information leaks either through the model itself or through predictions made by the model. Co…
Data Appraisal Without Data Sharing
Mimee Xu, Laurens van der Maaten, Awni Hannun
One of the most effective approaches to improving the performance of a machine learning model is to procure additional training data. A model owner seeking relevant training data f…
Differentiable Weighted Finite-State Transducers
Awni Hannun, Vineel Pratap, Jacob Kahn +1
We introduce a framework for automatic differentiation with weighted finite-state transducers (WFSTs) allowing them to be used dynamically at training time. Through the separation…
The Trade-Offs of Private Prediction
Laurens van der Maaten, Awni Hannun
Machine learning models leak information about their training data every time they reveal a prediction. This is problematic when the training data needs to remain private. Private…
Certified Data Removal from Machine Learning Models
Chuan Guo, Tom Goldstein, Awni Hannun +1
Good data stewardship requires removal of data at the request of the data's owner. This raises the question if and how a trained machine-learning model, which implicitly stores inf…
Privacy-Preserving Multi-Party Contextual Bandits
Awni Hannun, Brian Knott, Shubho Sengupta +1
Contextual bandits are online learners that, given an input, select an arm and receive a reward for that arm. They use the reward as a learning signal and aim to maximize the total…