5 papers
Data-efficient pre-training by scaling synthetic megadocs
Konwoo Kim, Suhas Kotha, Yejin Choi +3
Synthetic data augmentation has emerged as a promising solution when pre-training is constrained by data rather than compute. We study how to design synthetic data algorithms that…
Pre-training under infinite compute
Konwoo Kim, Suhas Kotha, Percy Liang +1
Since compute grows much faster than web text available for language model pre-training, we ask how one should approach pre-training under fixed data and no compute constraints. We…
Auditing Prompt Caching in Language Model APIs
Chenchen Gu, Xiang Lisa Li, Rohith Kuditipudi +2
Prompt caching in large language models (LLMs) results in data-dependent timing variations: cached prompts are processed faster than non-cached prompts. These timing differences in…
s1: Simple test-time scaling
Niklas Muennighoff, Zitong Yang, Weijia Shi +7
Test-time scaling is a promising new approach to language modeling that uses extra test-time compute to improve performance. Recently, OpenAI's o1 model showed this capability but…
Eliciting Language Model Behaviors with Investigator Agents
Xiang Lisa Li, Neil Chowdhury, Daniel D. Johnson +4
Language models exhibit complex, diverse behaviors when prompted with free-form text, making it difficult to characterize the space of possible outputs. We study the problem of beh…