368 citations · 921 across the 28 of their papers we have counts for
35 papers · 1 filter
Multi-Mixer Models: Flexible Sequence Modeling with Shared Representations
Kevin Y. Li, Asher Trockman, Ananda Theertha Suresh +1
Softmax attention is the cornerstone of modern large language models, but its memory scales linearly and compute quadratically with sequence length. Linear recurrent models, such a…
CoDistill-GRPO: A Co-Distillation Recipe for Efficient Group Relative Policy Optimization
Soo Min Kwon, Ziteng Sun, Ananda Theertha Suresh +2
Group Relative Policy Optimization (GRPO) has emerged as a powerful algorithm for improving the reasoning capabilities of language models, but often fails to improve small models d…
CafeQ: Calibration-free Quantization via Learned Transformations and Adaptive Rounding
Ziteng Sun, Adrian Benton, Samuel Kushnir +4
Post-training quantization is an effective method for reducing the serving cost of large language models, where the standard approach is to use a round-to-nearest quantization leve…
Rate of Model Collapse in Recursive Training
Ananda Theertha Suresh, Andrew Thangaraj, Aditya Nanda Kishore Khandavally
Given the ease of creating synthetic data from machine learning models, new models can be potentially trained on synthetic data generated by previous models. This recursive trainin…
Asymptotics of Language Model Alignment
Joy Qiping Yang, Salman Salamatian, Ziteng Sun +2
Let denote a generative language model. Let denote a reward model that returns a scalar that captures the degree at which a draw from is preferred. The goal of language…
Efficient Language Model Architectures for Differentially Private Federated Learning
Jae Hun Ro, Srinadh Bhojanapalli, Zheng Xu +2
Cross-device federated learning (FL) is a technique that trains a model on data distributed across typically millions of edge devices without data leaving the devices. SGD is the s…