1 citations · 1 across the 15 of their papers we have counts for
15 papers
Almost Free State Prediction Separation
John Langford, Nathan Godey, Giovanni Monea +5
State--prediction separation (SPS) relieves a language model's hidden state of two competing burdens---summarizing the context and predicting the next token---by splitting the forw…
Dion3: Full-Stack Orthogonal Updates
Noah Amsel, Jack Zhang, Kwangjun Ahn +5
The Muon optimizer incurs a significant overhead cost due to its cubic-time Newton-Schulz orthogonalization step. When weights are sharded, communication overhead compounds this co…
Full-bandwidth transformer
Xi Wang, Ziyang Cai, Zheng Zhan +5
Autoregressive transformers compute along two axes: horizontally across generated tokens, and vertically through model depth. Dense attention gives each token broad horizontal acce…
Hierarchical Latent Prediction for Language Models
Chang Shi, Tim Pearce, Manan Tomar +2
While standard Next-Token Prediction (NTP) lays the foundation of language model pre- training, its teacher-forced training paradigm may not be optimal for long-horizon reasoning a…
MEMENTO: Teaching LLMs to Manage Their Own Context
Vasilis Kontonis, Yuchen Zeng, Shivam Garg +7
Reasoning models think in long, unstructured streams with no mechanism for compressing or organizing their own intermediate state. We introduce MEMENTO: a method that teaches model…
Phi-4-reasoning-vision-15B Technical Report
Jyoti Aneja, Michael Harrison, Neel Joshi +3
We present Phi-4-reasoning-vision-15B, a compact open-weight multimodal reasoning model, and share the motivations, design choices, experiments, and learnings that informed its dev…
When Does Predictive Inverse Dynamics Outperform Behavior Cloning?
Lukas Schäfer, Pallavi Choudhury, Abdelhak Lemkhenter +10
Behavior cloning (BC) is a practical offline imitation learning method, but it often fails when expert demonstrations are limited. Recent works have introduced a class of architect…
Dion2: A Simple Method to Shrink Matrix in Muon
Kwangjun Ahn, Noah Amsel, John Langford
The Muon optimizer enjoys strong empirical performance and theoretical grounding. However, the super-linear cost of its orthonormalization step introduces increasing overhead with…
Next-Latent Prediction Transformers Learn Compact World Models
Jayden Teoh, Manan Tomar, Kwangjun Ahn +7
Transformers replace recurrence with a memory that grows with sequence length and self-attention that enables ad-hoc lookups over past tokens. Consequently, they lack an inherent i…
Dion: Distributed Orthonormalized Updates
Kwangjun Ahn, Byron Xu, Natalie Abreu +5
Orthonormalized updates accelerate training, improve stability, and enable robust hyperparameter transfer, but existing methods like Muon rely on dense matrix operations that clash…
Efficient Joint Prediction of Multiple Future Tokens
Kwangjun Ahn, Alex Lamb, John Langford
In this short report, we introduce joint multi-token prediction (JTP), a lightweight modification of standard next-token prediction designed to enrich hidden state representations…
The Belief State Transformer
Edward S. Hu, Kwangjun Ahn, Qinghua Liu +9
We introduce the "Belief State Transformer", a next-token predictor that takes both a prefix and suffix as inputs, with a novel objective of predicting both the next token for the…
Video Occupancy Models
Manan Tomar, Philippe Hansen-Estruch, Philip Bachman +4
We introduce a new family of video prediction models designed to support downstream control tasks. We call these models Video Occupancy models (VOCs). VOCs operate in a compact lat…
Towards Principled Representation Learning from Videos for Reinforcement Learning
Dipendra Misra, Akanksha Saran, Tengyang Xie +2
We study pre-training representations for decision-making using video data, which is abundantly available for tasks such as game agents and software testing. Even though significan…
PcLast: Discovering Plannable Continuous Latent States
Anurag Koul, Shivakanth Sujit, Shaoru Chen +11
Goal-conditioned planning benefits from learned low-dimensional representations of rich observations. While compact latent representations typically learned from variational autoen…