11 papers
ACTG-ARL: Differentially Private Conditional Text Generation with RL-Boosted Control
Yuzheng Hu, Ryan McKenna, Da Yu +4
Generating high-quality synthetic text under differential privacy (DP) is critical for training and evaluating language models without compromising user privacy. Prior work on synt…
MAPLE: Metadata Augmented Private Language Evolution
Eli Chien, Yuzheng Hu, Ryan McKenna +3
Differentially private (DP) fine-tuning of large language models (LLMs) requires massive compute and full model access, which rules out state-of-the-art proprietary APIs for genera…
Humanity's Last Exam
Long Phan, Alice Gatti, Ziwen Han +1144
Benchmarks are important tools for tracking the rapid advancements in large language model (LLM) capabilities. However, benchmarks are not keeping pace in difficulty: LLMs now achi…
A Unified Theory of Random Projection for Influence Functions
Pingbang Hu, Yuzheng Hu, Jiaqi W. Ma +1
Influence functions and related data attribution scores take the form of , where is a curvature operator. In modern overparametrized models,…
OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration
Shaobo Wang, Xuan Ouyang, Tianyi Xu +9
As high-quality public text approaches exhaustion, a phenomenon known as the Data Wall, pre-training is shifting from more tokens to better tokens. However, existing methods either…
Taming Hyperparameter Sensitivity in Data Attribution: Practical Selection Without Costly Retraining
Weiyi Wang, Junwei Deng, Yuzheng Hu +5
Data attribution methods, which quantify the influence of individual training data points on a machine learning model, have gained increasing popularity in data-centric application…