1 citations · 1 across the 10 of their papers we have counts for
6 papers · 1 filter
Scaling Self-Play with Self-Guidance
Luke Bailey, Kaiyue Wen, Kefan Dong +2
LLM self-play algorithms are notable in that, in principle, nothing bounds their learning: a Conjecturer model creates problems for a Solver, and both improve together. However, in…
Data-efficient pre-training by scaling synthetic megadocs
Konwoo Kim, Suhas Kotha, Yejin Choi +3
Synthetic data augmentation has emerged as a promising solution when pre-training is constrained by data rather than compute. We study how to design synthetic data algorithms that…
Reasoning to Learn from Latent Thoughts
Yangjun Ruan, Neil Band, Chris J. Maddison +1
Compute scaling for language model (LM) pretraining has outpaced the growth of human-written texts, leading to concerns that data will become the bottleneck to LM scaling. To conti…
Synthetic continued pretraining
Zitong Yang, Neil Band, Shuangping Li +2
Pretraining on large-scale, unstructured internet text enables language models to acquire a significant amount of world knowledge. However, this knowledge acquisition is data-ineff…
Linguistic Calibration of Long-Form Generations
Neil Band, Xuechen Li, Tengyu Ma +1
Language models (LMs) may lead their users to make suboptimal downstream decisions when they confidently hallucinate. This issue can be mitigated by having the LM verbally convey t…
Language Models with Conformal Factuality Guarantees
Christopher Mohri, Tatsunori Hashimoto
Guaranteeing the correctness and factuality of language model (LM) outputs is a major open problem. In this work, we propose conformal factuality, a framework that can ensure high…