63.6k citations · 104.5k across the 42 of their papers we have counts for
43 papers
Stability-Aware Training of Machine Learning Force Fields with Differentiable Boltzmann Estimators
Sanjeev Raja, Ishan Amin, Fabian Pedregosa +1
Machine learning force fields (MLFFs) are an attractive alternative to ab-initio methods for molecular dynamics (MD) simulations. However, they can produce unstable simulations, li…
On the Interplay Between Stepsize Tuning and Progressive Sharpening
Vincent Roulet, Atish Agarwala, Fabian Pedregosa
Recent empirical work has revealed an intriguing property of deep learning models by which the sharpness (largest eigenvalue of the Hessian) increases throughout optimization until…
On Implicit Bias in Overparameterized Bilevel Optimization
Paul Vicol, Jonathan Lorraine, Fabian Pedregosa +2
Many problems in machine learning involve bilevel optimization (BLO), including hyperparameter optimization, meta-learning, and dataset distillation. Bilevel problems consist of tw…
A Novel Stochastic Gradient Descent Algorithm for Learning Principal Subspaces
Charline Le Lan, Joshua Greaves, Jesse Farebrother +4
Many machine learning problems encode their data as a matrix with a possibly very large number of rows and columns. In several applications like neuroscience, image compression or…
When is Momentum Extragradient Optimal? A Polynomial-Based Analysis
Junhyung Lyle Kim, Gauthier Gidel, Anastasios Kyrillidis +1
The extragradient method has gained popularity due to its robust convergence properties for differentiable games. Unlike single-objective optimization, game dynamics involve comple…
Second-order regression models exhibit progressive sharpening to the edge of stability
Atish Agarwala, Fabian Pedregosa, Jeffrey Pennington
Recent studies of gradient descent with large step sizes have shown that there is often a regime with an initial increase in the largest eigenvalue of the loss Hessian (progressive…
The Curse of Unrolling: Rate of Differentiating Through Optimization
Damien Scieur, Quentin Bertrand, Gauthier Gidel +1
Computing the Jacobian of the solution of an optimization problem is a central problem in machine learning, with applications in hyperparameter optimization, meta-learning, optimiz…
Only Tails Matter: Average-Case Universality and Robustness in the Convex Regime
Leonardo Cunha, Gauthier Gidel, Fabian Pedregosa +2
The recently developed average-case analysis of optimization methods allows a more fine-grained and representative convergence analysis than usual worst-case results. In exchange,…
Cutting Some Slack for SGD with Adaptive Polyak Stepsizes
Robert M. Gower, Mathieu Blondel, Nidham Gazagnadou +1
Tuning the step size of stochastic gradient descent is tedious and error prone. This has motivated the development of methods that automatically adapt the step size using readily a…
GradMax: Growing Neural Networks using Gradient Information
Utku Evci, Bart van Merriënboer, Thomas Unterthiner +2
The architecture and the parameters of neural networks are often optimized independently, which requires costly retraining of the parameters whenever the architecture is modified.…
Super-Acceleration with Cyclical Step-sizes
Baptiste Goujaud, Damien Scieur, Aymeric Dieuleveut +2
We develop a convergence-rate analysis of momentum with cyclical step-sizes. We show that under some assumption on the spectral gap of Hessians in machine learning, cyclical step-s…
Boosting Variational Inference With Locally Adaptive Step-Sizes
Gideon Dresdner, Saurav Shekhar, Fabian Pedregosa +2
Variational Inference makes a trade-off between the capacity of the variational family and the tractability of finding an approximate posterior distribution. Instead, Boosting Vari…
Efficient and Modular Implicit Differentiation
Mathieu Blondel, Quentin Berthet, Marco Cuturi +5
Automatic differentiation (autodiff) has revolutionized machine learning. It allows to express complex computations by composing elementary ones in creative ways and removes the bu…
SGD in the Large: Average-case Analysis, Asymptotics, and Stepsize Criticality
Courtney Paquette, Kiwon Lee, Fabian Pedregosa +1
We propose a new framework, inspired by random matrix theory, for analyzing the dynamics of stochastic gradient descent (SGD) when both number of samples and dimensions are large.…
Bridging the Gap Between Adversarial Robustness and Optimization Bias
Fartash Faghri, Sven Gowal, Cristina Vasconcelos +3
We demonstrate that the choice of optimizer, neural network architecture, and regularizer significantly affect the adversarial robustness of linear neural networks, providing guara…
Average-case Acceleration for Bilinear Games and Normal Matrices
Carles Domingo-Enrich, Fabian Pedregosa, Damien Scieur
Advances in generative modeling and adversarial learning have given rise to renewed interest in smooth games. However, the absence of symmetry in the matrix of second derivatives p…
Halting Time is Predictable for Large Models: A Universality Property and Average-case Analysis
Courtney Paquette, Bart van Merriënboer, Elliot Paquette +1
Average-case analysis computes the complexity of an algorithm averaged over all possible inputs. Compared to worst-case analysis, it is more representative of the typical behavior…
The Geometry of Sign Gradient Descent
Lukas Balles, Fabian Pedregosa, Nicolas Le Roux
Sign-based optimization methods have become popular in machine learning due to their favorable communication cost in distributed optimization and their surprisingly good performanc…
Stochastic Frank-Wolfe for Constrained Finite-Sum Minimization
Geoffrey Négiar, Gideon Dresdner, Alicia Tsai +4
We propose a novel Stochastic Frank-Wolfe (a.k.a. conditional gradient) algorithm for constrained smooth finite-sum minimization with a generalized linear prediction/structure. Thi…
Average-case Acceleration Through Spectral Density Estimation
Fabian Pedregosa, Damien Scieur
We develop a framework for the average-case analysis of random quadratic problems and derive algorithms that are optimal under this analysis. This yields a new class of methods tha…
Universal Average-Case Optimality of Polyak Momentum
Damien Scieur, Fabian Pedregosa
Polyak momentum (PM), also known as the heavy-ball method, is a widely used optimization method that enjoys an asymptotic optimal worst-case complexity on quadratic objectives. How…
A Test for Shared Patterns in Cross-modal Brain Activation Analysis
Elena Kalinina, Fabian Pedregosa, Vittorio Iacovella +2
Determining the extent to which different cognitive modalities (understood here as the set of cognitive processes underlying the elaboration of a stimulus by the brain) rely on ove…
SciPy 1.0--Fundamental Algorithms for Scientific Computing in Python
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant +32
SciPy is an open source scientific computing library for the Python programming language. SciPy 1.0 was released in late 2017, about 16 years after the original version 0.1 release…
The Difficulty of Training Sparse Neural Networks
Utku Evci, Fabian Pedregosa, Aidan Gomez +1
We investigate the difficulties of training sparse neural networks and make new observations about optimization dynamics and the energy landscape within the sparse regime. Recent w…