84 citations · 655 across the 44 of their papers we have counts for
5 papers · 1 filter
Scaling Repeated-Data Pretraining: Batch Size, Regularization, and GPU-Time Tradeoffs
Sewon Min
We study pretraining when compute is abundant but unique training data are finite, requiring repeated training on the same data. We first revisit infinite-compute pretraining (Kim…
Base Models Can Reason By Taking a Cue From Training Data
Sophie L. Wang, Amil Dravid, Rulin Shao +3
In this paper, we study how training data creates associations between the tokens at the start of a base model's response and the reasoning behavior that follows. First, we demonst…
Reference-Based Distillation Detection in LLMs
Rajat Rawat, Sizhe Chen, Akshay Anand +3
Model distillation -- training on outputs from stronger third-party models -- is widely used to boost performance, but raises concerns about unfair advantages and policy violations…
Learning to Detect Language Model Training Data via Active Reconstruction
Junjie Oscar Yin, John X. Morris, Vitaly Shmatikov +2
Detecting LLM training data is generally framed as a membership inference attack (MIA) problem. However, conventional MIAs operate passively on fixed model weights, using log-likel…
RLAC: Reinforcement Learning with Adversarial Critic for Free-Form Generation Tasks
Mian Wu, Gavin Zhang, Sewon Min +2
Open-ended generation tasks require outputs to satisfy diverse and often implicit task-specific evaluation rubrics. The sheer number of relevant rubrics leads to prohibitively high…