84 citations · 655 across the 44 of their papers we have counts for
52 papers
Scaling Repeated-Data Pretraining: Batch Size, Regularization, and GPU-Time Tradeoffs
Sewon Min
We study pretraining when compute is abundant but unique training data are finite, requiring repeated training on the same data. We first revisit infinite-compute pretraining (Kim…
Base Models Can Reason By Taking a Cue From Training Data
Sophie L. Wang, Amil Dravid, Rulin Shao +3
In this paper, we study how training data creates associations between the tokens at the start of a base model's response and the reasoning behavior that follows. First, we demonst…
No More Free Lunch: Corpus Task Complexity Matters as Corpora Grow
Prasann Singhal, Amanda Bertsch, Jacob Steinhardt +1
Given a large corpus, the questions one might ask can vary -- from "When was the first human heart transplant?" to "What are all the contradictory claims in this literature?" -- bu…
Can Language Models Actually Retrieve In-Context? Drowning in Documents at Million Token Scale
Siddharth Gollapudi, Nilesh Gupta, Prasann Singhal +1
Language models (LMs) raise an intriguing alternative to vector-based retrieval: conditioning on an in-context corpus and directly generating a relevant answer. However, prior work…
Reference-Based Distillation Detection in LLMs
Rajat Rawat, Sizhe Chen, Akshay Anand +3
Model distillation -- training on outputs from stronger third-party models -- is widely used to boost performance, but raises concerns about unfair advantages and policy violations…
Learning to Detect Language Model Training Data via Active Reconstruction
Junjie Oscar Yin, John X. Morris, Vitaly Shmatikov +2
Detecting LLM training data is generally framed as a membership inference attack (MIA) problem. However, conventional MIAs operate passively on fixed model weights, using log-likel…