3k citations · 5.8k across the 70 of their papers we have counts for
80 papers
Multi-Mixer Models: Flexible Sequence Modeling with Shared Representations
Kevin Y. Li, Asher Trockman, Ananda Theertha Suresh +1
Softmax attention is the cornerstone of modern large language models, but its memory scales linearly and compute quadratically with sequence length. Linear recurrent models, such a…
CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimization
Soo Min Kwon, Ziteng Sun, Ananda Theertha Suresh +2
Group Relative Policy Optimization (GRPO) has emerged as a powerful algorithm for improving the reasoning capabilities of language models, but often fails to improve small models d…
CafeQ: Calibration-free Quantization via Learned Transformations and Adaptive Rounding
Ziteng Sun, Adrian Benton, Samuel Kushnir +4
Post-training quantization is an effective method for reducing the serving cost of large language models, where the standard approach is to use a round-to-nearest quantization leve…
On Robust Hypothesis Testing with respect to the Hellinger Distance
Eeshan Modak, Sivaraman Balakrishnan, Ananda Theertha Suresh
We study a variant of the simple hypothesis testing problem where observed samples do not necessarily come from either of the specified distributions, but rather from a close varia…
Hierarchical Retrieval: The Geometry and a Pretrain-Finetune Recipe
Chong You, Rajesh Jayaram, Ananda Theertha Suresh +3
Dual encoder (DE) models, where a pair of matching query and document are embedded into similar vector representations, are widely used in information retrieval due to their simpli…
Efficient and Asymptotically Unbiased Constrained Decoding for Large Language Models
Haotian Ye, Himanshu Jain, Chong You +4
In real-world applications of large language models, outputs are often required to be confined: selecting items from predefined product or document sets, generating phrases that co…
Rate of Model Collapse in Recursive Training
Ananda Theertha Suresh, Andrew Thangaraj, Aditya Nanda Kishore Khandavally
Given the ease of creating synthetic data from machine learning models, new models can be potentially trained on synthetic data generated by previous models. This recursive trainin…
Coupling without Communication and Drafter-Invariant Speculative Decoding
Majid Daliri, Christopher Musco, Ananda Theertha Suresh
Suppose Alice has a distribution and Bob has a distribution . Alice wants to draw a sample and Bob a sample such that with as high of probabilit…
Private federated discovery of out-of-vocabulary words for Gboard
Ziteng Sun, Peter Kairouz, Haicheng Sun +2
The vocabulary of language models in Gboard, Google's keyboard application, plays a crucial role for improving user experience. One way to improve the vocabulary is to discover fre…
Asymptotics of Language Model Alignment
Joy Qiping Yang, Salman Salamatian, Ziteng Sun +2
Let denote a generative language model. Let denote a reward model that returns a scalar that captures the degree at which a draw from is preferred. The goal of language…
Exploring and Improving Drafts in Blockwise Parallel Decoding
Taehyeon Kim, Ananda Theertha Suresh, Kishore Papineni +3
Despite the remarkable strides made by autoregressive language models, their potential is often hampered by the slow inference speeds inherent in sequential token generation. Block…
Efficient Language Model Architectures for Differentially Private Federated Learning
Jae Hun Ro, Srinadh Bhojanapalli, Zheng Xu +2
Cross-device federated learning (FL) is a technique that trains a model on data distributed across typically millions of edge devices without data leaving the devices. SGD is the s…
Block Verification Accelerates Speculative Decoding
Ziteng Sun, Uri Mendlovic, Yaniv Leviathan +4
Speculative decoding is an effective method for lossless acceleration of large language models during inference. It uses a fast model to draft a block of tokens which are then veri…
Theoretical guarantees on the best-of-n alignment policy
Ahmad Beirami, Alekh Agarwal, Jonathan Berant +4
A simple and effective method for the inference-time alignment and scaling test-time compute of generative models is best-of- sampling, where samples are drawn from a refere…
Mean estimation in the add-remove model of differential privacy
Alex Kulesza, Ananda Theertha Suresh, Yuyan Wang
Differential privacy is often studied under two different models of neighboring datasets: the add-remove model and the swap model. While the swap model is frequently used in the ac…
Multi-Group Fairness Evaluation via Conditional Value-at-Risk Testing
Lucas Monteiro Paes, Ananda Theertha Suresh, Alex Beutel +2
Machine learning (ML) models used in prediction and classification tasks may display performance disparities across population groups determined by sensitive attributes (e.g., race…
SpecTr: Fast Speculative Decoding via Optimal Transport
Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro +3
Autoregressive sampling from large language models has led to state-of-the-art results in several natural language tasks. However, autoregressive sampling generates tokens one at a…
Federated Heavy Hitter Recovery under Linear Sketching
Adria Gascon, Peter Kairouz, Ziteng Sun +1
Motivated by real-life deployments of multi-round federated analytics with secure aggregation, we investigate the fundamental communication-accuracy tradeoffs of the heavy hitter d…
FedYolo: Augmenting Federated Learning with Pretrained Transformers
Xuechen Zhang, Mingchen Li, Xiangyu Chang +4
The growth and diversity of machine learning applications motivate a rethinking of learning with mobile and edge devices. How can we address diverse client goals and learn with sca…
The importance of feature preprocessing for differentially private linear optimization
Ziteng Sun, Ananda Theertha Suresh, Aditya Krishna Menon
Training machine learning models with differential privacy (DP) has received increasing interest in recent years. One of the most popular algorithms for training differentially pri…
Subset-Based Instance Optimality in Private Estimation
Travis Dick, Alex Kulesza, Ziteng Sun +1
We propose a new definition of instance optimality for differentially private estimation algorithms. Our definition requires an optimal algorithm to compete, simultaneously for eve…
Concentration Bounds for Discrete Distribution Estimation in KL Divergence
Clément L. Canonne, Ziteng Sun, Ananda Theertha Suresh
We study the problem of discrete distribution estimation in KL divergence and provide concentration bounds for the Laplace estimator. We show that the deviation from mean scales as…
Private Domain Adaptation from a Public Source
Raef Bassily, Mehryar Mohri, Ananda Theertha Suresh
A key problem in a variety of applications is that of domain adaptation from a public source domain, for which a relatively large amount of labeled data with no privacy constraints…
Algorithms for bounding contribution for histogram estimation under user-level privacy
Yuhan Liu, Ananda Theertha Suresh, Wennan Zhu +2
We study the problem of histogram estimation under user-level differential privacy, where the goal is to preserve the privacy of all entries of any single user. We consider the het…