157 citations · 529 across the 78 of their papers we have counts for
8 papers · 1 filter
Finite Regret and Cycles with Fixed Step-Size via Alternating Gradient Descent-Ascent
James P. Bailey, Gauthier Gidel, Georgios Piliouras
Gradient descent is arguably one of the most popular online optimization methods with a wide array of applications. However, the standard implementation where agents simultaneously…
Linear Lower Bounds and Conditioning of Differentiable Games
Adam Ibrahim, Waïss Azizian, Gauthier Gidel +1
Recent successes of game-theoretic formulations in ML have caused a resurgence of research interest in differentiable games. Overwhelmingly, that research focuses on methods and up…
A Tight and Unified Analysis of Gradient-Based Methods for a Whole Spectrum of Games
Waïss Azizian, Ioannis Mitliagkas, Simon Lacoste-Julien +1
We consider differentiable games where the goal is to find a Nash equilibrium. The machine learning community has recently started using variants of the gradient method (GD). Prime…
A Closer Look at the Optimization Landscapes of Generative Adversarial Networks
Hugo Berard, Gauthier Gidel, Amjad Almahairi +2
Generative adversarial networks have been very successful in generative modeling, however they remain relatively challenging to train compared to standard deep neural networks. In…
Non-normal Recurrent Neural Network (nnRNN): learning long time dependencies while improving expressivity with transient dynamics
Giancarlo Kerg, Kyle Goyette, Maximilian Puelma Touzel +4
A recent strategy to circumvent the exploding and vanishing gradient problem in RNNs, and to allow the stable propagation of signals over long time scales, is to constrain recurren…
Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates
Sharan Vaswani, Aaron Mishkin, Issam Laradji +3
Recent works have shown that stochastic gradient descent (SGD) achieves the fast convergence rates of full-batch gradient descent for over-parameterized models satisfying certain i…
Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networks
Gauthier Gidel, Francis Bach, Simon Lacoste-Julien
When optimizing over-parameterized models, such as deep neural networks, a large set of parameters can achieve zero training error. In such cases, the choice of the optimization al…
Reducing Noise in GAN Training with Variance Reduced Extragradient
Tatjana Chavdarova, Gauthier Gidel, François Fleuret +1
We study the effect of the stochastic gradient noise on the training of generative adversarial networks (GANs) and show that it can prevent the convergence of standard game optimiz…