24 citations · 55 across the 9 of their papers we have counts for
10 papers
Glocal Smoothness: Line search and adaptive step sizes can help in theory too!
Curtis Fox, Aaron Mishkin, Sharan Vaswani +1
Iteration complexities for optimizing smooth functions with first-order algorithms are typically stated in terms of a global Lipschitz constant of the gradient, and near-optimal re…
Exploring the loss landscape of regularized neural networks via convex duality
Sungyoon Kim, Aaron Mishkin, Mert Pilanci
We discuss several aspects of the loss landscape of regularized neural networks: the structure of stationary points, connectivity of optimal solutions, path with nonincreasing loss…
Faster Convergence of Stochastic Accelerated Gradient Descent under Interpolation
Aaron Mishkin, Mert Pilanci, Mark Schmidt
We prove new convergence rates for a generalized version of stochastic Nesterov acceleration under interpolation conditions. Unlike previous analyses, our approach accelerates any…
A Library of Mirrors: Deep Neural Nets in Low Dimensions are Convex Lasso Models with Reflection Features
Emi Zeger, Yifei Wang, Aaron Mishkin +3
We prove that training neural networks on 1-D data is equivalent to solving convex Lasso problems with discrete, explicitly defined dictionary matrices. We consider neural networks…
Analyzing and Improving Greedy 2-Coordinate Updates for Equality-Constrained Optimization via Steepest Descent in the 1-Norm
Amrutha Varshini Ramesh, Aaron Mishkin, Mark Schmidt +3
We consider minimizing a smooth function subject to a summation constraint over its variables. By exploiting a connection between the greedy 2-coordinate update for this problem an…
Optimal Sets and Solution Paths of ReLU Networks
Aaron Mishkin, Mert Pilanci
We develop an analytical framework to characterize the set of optimal ReLU neural networks by reformulating the non-convex training problem as a convex program. We show that the gl…
Fast Convex Optimization for Two-Layer ReLU Networks: Equivalent Model Classes and Cone Decompositions
Aaron Mishkin, Arda Sahiner, Mert Pilanci
We develop fast algorithms and robust software for convex optimization of two-layer neural networks with ReLU activation functions. Our work leverages a convex reformulation of the…
To Each Optimizer a Norm, To Each Norm its Generalization
Sharan Vaswani, Reza Babanezhad, Jose Gallego-Posada +3
We study the implicit regularization of optimization methods for linear models interpolating the training data in the under-parametrized and over-parametrized regimes. Since it is…
Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates
Sharan Vaswani, Aaron Mishkin, Issam Laradji +3
Recent works have shown that stochastic gradient descent (SGD) achieves the fast convergence rates of full-batch gradient descent for over-parameterized models satisfying certain i…
SLANG: Fast Structured Covariance Approximations for Bayesian Deep Learning with Natural Gradient
Aaron Mishkin, Frederik Kunstner, Didrik Nielsen +2
Uncertainty estimation in large deep-learning models is a computationally challenging task, where it is difficult to form even a Gaussian approximation to the posterior distributio…