205 citations · 777 across the 38 of their papers we have counts for
5 papers · 1 filter
True Asymptotic Natural Gradient Optimization
Yann Ollivier
We introduce a simple algorithm, True Asymptotic Natural Gradient Optimization (TANGO), that converges to a true natural gradient descent in the limit of small learning rates, with…
Natural Langevin Dynamics for Neural Networks
Gaétan Marceau-Caron, Yann Ollivier
One way to avoid overfitting in machine learning is to use model parameters distributed according to a Bayesian posterior given the data, rather than the maximum likelihood estimat…
Unbiasing Truncated Backpropagation Through Time
Corentin Tallec, Yann Ollivier
Truncated Backpropagation Through Time (truncated BPTT) is a widespread method for learning recurrent computational graphs. Truncated BPTT keeps the computational benefits of Backp…
Online Natural Gradient as a Kalman Filter
Yann Ollivier
We cast Amari's natural gradient in statistical learning as a specific case of Kalman filtering. Namely, applying an extended Kalman filter to estimate a fixed unknown parameter of…
Unbiased Online Recurrent Optimization
Corentin Tallec, Yann Ollivier
The novel Unbiased Online Recurrent Optimization (UORO) algorithm allows for online learning of general recurrent computational graphs such as recurrent network models. It works in…