10 papers
On-Policy Self-Distillation with Sampled Demonstrations Reduces Output Diversity
Andrei Liviu Nicolicioiu, Mohammad Pezeshki, Aaron Courville
On-policy self-distillation achieves strong pass@1 accuracy by using a single model as both teacher and student, with the teacher conditioned on a correct demonstration to provide…
Beyond Multi-Token Prediction: Pretraining LLMs with Future Summaries
Divyat Mahajan, Sachin Goyal, Badr Youbi Idrissi +4
Next-token prediction (NTP) has driven the success of large language models (LLMs), but it struggles with long-horizon reasoning, planning, and creative writing, with these limitat…
ReasonCACHE: Teaching LLMs To Reason Without Weight Updates
Sharut Gupta, Phillip Isola, Stefanie Jegelka +4
Can Large language models (LLMs) learn to reason without any weight update and only through in-context learning (ICL)? ICL is strikingly sample-efficient, often learning from only…
Why Less is More (Sometimes): A Theory of Data Curation
Elvis Dohmatob, Mohammad Pezeshki, Reyhane Askari-Hemmat
This paper introduces a theoretical framework to resolve a central paradox in modern machine learning: When is it better to use less data? This question has become critical as clas…
Iterative Amortized Inference: Unifying In-Context Learning and Learned Optimizers
Sarthak Mittal, Divyat Mahajan, Guillaume Lajoie +1
Modern learning systems increasingly rely on amortized learning - the idea of reusing computation or inductive biases shared across tasks to enable rapid generalization to novel pr…
Compositional Risk Minimization
Divyat Mahajan, Mohammad Pezeshki, Charles Arnal +3
Compositional generalization is a crucial step towards developing data-efficient intelligent machines that generalize in human-like ways. In this work, we tackle a challenging form…