174 citations · 991 across the 46 of their papers we have counts for
46 papers
Understanding Reasoning from Pretraining to Post-Training
Jingyan Shen, Ang Li, Salman Rahman +4
Reinforcement learning (RL) has become central to improving large language models (LLMs) on complex reasoning tasks, yet RL post-training is largely studied in isolation from the p…
Basic Inequalities for First-Order Optimization with Applications to Statistical Risk Analysis
Seunghoon Paik, Kangjie Zhou, Matus Telgarsky +1
In this work, we introduce for first-order iterative optimization algorithms, forming a simple yet versatile framework which connects implicit and exp…
Benefits of Early Stopping in Gradient Descent for Overparameterized Logistic Regression
Jingfeng Wu, Peter Bartlett, Matus Telgarsky +1
In overparameterized logistic regression, gradient descent (GD) iterates diverge in norm while converging in direction to the maximum -margin solution -- a phenomenon known…
One-layer transformers fail to solve the induction heads task
Clayton Sanford, Daniel Hsu, Matus Telgarsky
A simple communication complexity argument proves that no one-layer transformer can solve the induction heads task unless its size is exponentially larger than the size sufficient…
Transformers, parallel computation, and logarithmic depth
Clayton Sanford, Daniel Hsu, Matus Telgarsky
We show that a constant number of self-attention layers can efficiently simulate, and be simulated by, a constant number of communication rounds of Massively Parallel Computation.…
Spectrum Extraction and Clipping for Implicitly Linear Layers
Ali Ebrahimpour Boroojeny, Matus Telgarsky, Hari Sundaram
We show the effectiveness of automatic differentiation in efficiently and correctly computing and controlling the spectrum of implicitly linear operators, a rich family of layer ty…