19 papers
Domain-Aware Scaling Laws Uncover Data Synergy
Kimia Hamidieh, Lester Mackey, David Alvarez-Melis
The paper defines and measures how mixing data from different domains during language model pretraining can produce synergistic or interfering effects, and shows that accounting fo…
Understanding Layer Patching in Model Size Interpolation
Sara Kangaslahti, Jonathan Geuter, Nihal V. Nayak +3
Zero-shot model size interpolation aims to create new models of intermediate target sizes by combining existing models without additional training. Recent work on boomerang distill…
A Critical Look at Targeted Instruction Selection: Disentangling What Matters (and What Doesn't)
Nihal V. Nayak, Paula Rodriguez-Diaz, Neha Hulkund +2
Instruction fine-tuning of large language models (LLMs) often involves selecting a subset of instruction training data from a large candidate pool, using a small query set from the…
Stop Training for the Worst: Progressive Unmasking Accelerates Masked Diffusion Training
Jaeyeon Kim, Jonathan Geuter, David Alvarez-Melis +2
Masked Diffusion Models (MDMs) have emerged as a promising approach for generative modeling in discrete spaces. By generating sequences in any order and allowing for parallel decod…
Integrating chemical structures as treatments improves representations of microscopy images for morphological profiling
Yemin Yu, Emre Hayir, Neil Tenenholtz +5
Recent advances in self-supervised deep learning have improved our ability to quantify cellular morphological changes in high-throughput microscopy screens, a process known as morp…
Guided Speculative Inference for Efficient Test-Time Alignment of LLMs
Jonathan Geuter, Youssef Mroueh, David Alvarez-Melis
We propose Guided Speculative Inference (GSI), a novel algorithm for efficient reward-guided decoding in large language models. GSI combines soft best-of- test-time scaling with…