11 papers
From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models
Seunggeun Kim, Jaeyeon Kim, Taekyun Lee +4
The paper investigates how to give language models a native ability to reason and generate text in any order, introducing insertion‑based and latent‑space masked diffusion methods…
A Defense of the Quadratic Model
Alexandru Meterez, Pranav Ajit Nair, Depen Morwani +3
Due to the complexity of neural network loss landscapes, optimization theory is forced to rely on idealized models, and there is generally a tradeoff between how theoretically trac…
Peer-Predictive Self-Training for Language Model Reasoning
Shi Feng, Hanlin Zhang, Fan Nie +2
Mechanisms for continued self-improvement of language models without external supervision remain an open challenge. We propose Peer-Predictive Self-Training (PST), a label-free fin…
How Post-Training Shapes Biological Reasoning Models
Lukas Fesser, Hanlin Zhang, Michelle M. Li +5
Scientific reasoning models for biology combine language models with foundation models trained on multimodal biological data, including DNA, RNA, and proteins. These models are bui…
Compute Efficiency and Serial Runtime Tradeoffs for Stochastic Momentum Methods
Depen Morwani, Alexandru Meterez, Pranav Nair +1
Stochastic momentum methods such as heavy ball (HB), Nesterov momentum, and variants of Accelerated SGD (ASGD) [Kidambi et al., 2018] are widely used in modern training, but their…
A Unifying View of Attention Sinks: Two Algorithms, Two Solutions
Lukas Fesser, Mozes Jacobs, Thomas Fel +2
When attention concentrates on a single token, a sink, what is the model actually computing? Attention sinks are ubiquitous in softmax transformers, yet this shared visual signatur…