40 citations · 97 across the 16 of their papers we have counts for
10 papers · 1 filter
Training Dynamics of Softmax Self-Attention: Fast Global Convergence via Preconditioning
Gautam Goel, Mahdi Soltanolkotabi, Peter Bartlett
We study the training dynamics of gradient descent in a softmax self-attention layer trained to perform linear regression and show that a simple first-order optimization algorithm…
Can a Transformer Represent a Kalman Filter?
Gautam Goel, Peter Bartlett
Transformers are a class of autoregressive deep learning architectures which have recently achieved state-of-the-art performance in various vision, language, and robotics tasks. We…
Best of Both Worlds in Online Control: Competitive Ratio and Policy Regret
Gautam Goel, Naman Agarwal, Karan Singh +1
We consider the fundamental problem of online control of a linear dynamical system from two different viewpoints: regret minimization and competitive analysis. We prove that the op…
Online estimation and control with optimal pathlength regret
Gautam Goel, Babak Hassibi
A natural goal when designing online learning algorithms for non-stationary environments is to bound the regret of the algorithm in terms of the temporal variation of the input seq…
Regret-optimal Estimation and Control
Gautam Goel, Babak Hassibi
We consider estimation and control in linear time-varying dynamical systems from the perspective of regret minimization. Unlike most prior work in this area, we focus on the proble…
Regret-optimal control in dynamic environments
Gautam Goel, Babak Hassibi
We consider control in linear time-varying dynamical systems from the perspective of regret minimization. Unlike most prior work in this area, we focus on the problem of designing…
Online Optimization with Predictions and Non-convex Losses
Yiheng Lin, Gautam Goel, Adam Wierman
We study online optimization in a setting where an online learner seeks to optimize a per-round hitting cost, which may be non-convex, while incurring a movement cost when changing…
Beyond Online Balanced Descent: An Optimal Algorithm for Smoothed Online Optimization
Gautam Goel, Yiheng Lin, Haoyuan Sun +1
We study online convex optimization in a setting where the learner seeks to minimize the sum of a per-round hitting cost and a movement cost which is incurred when changing decisio…
Smoothed Online Optimization for Regression and Control
Gautam Goel, Adam Wierman
We consider Online Convex Optimization (OCO) in the setting where the costs are -strongly convex and the online learner pays a switching cost for changing decisions between roun…
Smoothed Online Convex Optimization in High Dimensions via Online Balanced Descent
Niangjun Chen, Gautam Goel, Adam Wierman
We study Smoothed Online Convex Optimization, a version of online convex optimization where the learner incurs a penalty for changing her actions between rounds. Given a $Ω(\sqrt{d…