5 papers
The Potential of Second-Order Optimization for LLMs: A Study with Full Gauss-Newton
Natalie Abreu, Nikhil Vyas, Sham Kakade +1
Recent efforts to accelerate LLM pretraining have focused on computationally-efficient approximations that exploit second-order structure. This raises a key question for large-scal…
A Taxonomy of Transcendence
Natalie Abreu, Edwin Zhang, Eran Malach +1
Although language models are trained to mimic humans, the resulting systems display capabilities beyond the scope of any one person. To understand this phenomenon, we use a control…
Dion: Distributed Orthonormalized Updates
Kwangjun Ahn, Byron Xu, Natalie Abreu +5
Orthonormalized updates accelerate training, improve stability, and enable robust hyperparameter transfer, but existing methods like Muon rely on dense matrix operations that clash…
Addressing Discrepancies in Semantic and Visual Alignment in Neural Networks
Natalie Abreu, Nathan Vaska, Victoria Helus
For the task of image classification, neural networks primarily rely on visual patterns. In robust networks, we would expect for visually similar classes to be represented similarl…
Addressing Mistake Severity in Neural Networks with Semantic Knowledge
Natalie Abreu, Nathan Vaska, Victoria Helus
Robustness in deep neural networks and machine learning algorithms in general is an open research challenge. In particular, it is difficult to ensure algorithmic performance is mai…