4 papers
Test-Time Scaling Makes Overtraining Compute-Optimal
Nicholas Roberts, Sungjun Cho, Zhiqi Gao +7
Modern LLMs scale at test-time, e.g. via repeated sampling, where inference cost grows with model size and the number of samples. This creates a trade-off that pretraining scaling…
Pareto Optimal Code Generation
Gabriel Orlanski, Nicholas Roberts, Aws Albarghouthi +1
Generate-then-rank is the dominant test-time scaling (TTS) paradigm for code generation, but scaling accuracy by sampling and executing more candidates makes comprehensive verifica…
R&B: Domain Regrouping and Data Mixture Balancing for Efficient Foundation Model Training
Albert Ge, Tzu-Heng Huang, John Cooper +7
Data mixing strategies have successfully reduced the costs involved in training language models. While promising, such methods suffer from two flaws. First, they rely on predetermi…
Tabby: A Language Model Architecture for Tabular and Structured Data Synthesis
Sonia Cromp, Satya Sai Srinath Namburi GNVV, Mohammed Alkhudhayri +4
While advances in large language models (LLMs) have greatly improved the quality of synthetic text data in recent years, synthesizing tabular data has received relatively less atte…