9 papers · 1 filter
ReasonCACHE: Teaching LLMs To Reason Without Weight Updates
Sharut Gupta, Phillip Isola, Stefanie Jegelka +4
Can Large language models (LLMs) learn to reason without any weight update and only through in-context learning (ICL)? ICL is strikingly sample-efficient, often learning from only…
Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability
Shobhita Sundaram, John Quan, Ariel Kwiatkowski +3
RL methods for scaling large reasoning models stall on datasets with low initial success rates, and thus little training signal. We investigate a fundamental question: Can a pretra…
Operationalizing Quantized Disentanglement
Vitoria Barin-Pacela, Kartik Ahuja, Simon Lacoste-Julien +1
Recent theoretical work established the unsupervised identifiability of quantized factors under any diffeomorphism. The theory assumes that quantization thresholds correspond to ax…
Beyond Multi-Token Prediction: Pretraining LLMs with Future Summaries
Divyat Mahajan, Sachin Goyal, Badr Youbi Idrissi +4
Next-token prediction (NTP) has driven the success of large language models (LLMs), but it struggles with long-horizon reasoning, planning, and creative writing, with these limitat…
Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling
Sachin Goyal, David Lopez-Paz, Kartik Ahuja
In the past year, distillation has seen a renewed prominence in large language model (LLM) pretraining, exemplified by the Llama-3.2 and Gemma model families. While distillation ha…
Compositional Risk Minimization
Divyat Mahajan, Mohammad Pezeshki, Charles Arnal +3
Compositional generalization is a crucial step towards developing data-efficient intelligent machines that generalize in human-like ways. In this work, we tackle a challenging form…
DRoP: Distributionally Robust Data Pruning
Artem Vysogorets, Kartik Ahuja, Julia Kempe
In the era of exceptionally data-hungry models, careful selection of the training data is essential to mitigate the extensive costs of deep learning. Data pruning offers a solution…
On Provable Length and Compositional Generalization
Kartik Ahuja, Amin Mansouri
Out-of-distribution generalization capabilities of sequence-to-sequence models can be studied from the lens of two crucial forms of generalization: length generalization -- the abi…
Context is Environment
Sharut Gupta, Stefanie Jegelka, David Lopez-Paz +1
Two lines of work are taking the central stage in AI research. On the one hand, the community is making increasing efforts to build models that discard spurious correlations and ge…