7 papers · 1 filter
From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models
Seunggeun Kim, Jaeyeon Kim, Taekyun Lee +4
Many discrete reasoning tasks, such as code generation, are inherently non-causal: programmers move between high-level structure and local details, a process we call any-order infe…
A Defense of the Quadratic Model
Alexandru Meterez, Pranav Ajit Nair, Depen Morwani +3
Due to the complexity of neural network loss landscapes, optimization theory is forced to rely on idealized models, and there is generally a tradeoff between how theoretically trac…
How Post-Training Shapes Biological Reasoning Models
Lukas Fesser, Hanlin Zhang, Michelle M. Li +5
Scientific reasoning models for biology combine language models with foundation models trained on multimodal biological data, including DNA, RNA, and proteins. These models are bui…
Compute Efficiency and Serial Runtime Tradeoffs for Stochastic Momentum Methods
Depen Morwani, Alexandru Meterez, Pranav Nair +1
Stochastic momentum methods such as heavy ball (HB), Nesterov momentum, and variants of Accelerated SGD (ASGD) [Kidambi et al., 2018] are widely used in modern training, but their…
A Unifying View of Attention Sinks: From Mechanisms to Architectural Interventions
Lukas Fesser, Lukas Fesser*, Mozes Jacobs* +5
When attention concentrates on a single token, a sink, what is the model actually computing? Attention sinks are ubiquitous in softmax transformers, yet this shared visual signatur…
RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training
Rachit Bansal, Clara Mohri, Tian Qin +2
The standard LLM training pipeline applies reinforcement learning (RL) only after pre-training and supervised fine-tuning (SFT). We question this status quo by training a LLM from…
The Recurrent Transformer: Greater Effective Depth and Efficient Decoding
Costin-Andrei Oncescu, Depen Morwani, Samy Jelassi +3
Transformers process tokens in parallel but are temporally shallow: at position , each layer attends to key-value pairs computed based on the previous layer, yielding a depth ca…