538 citations · 2.3k across the 54 of their papers we have counts for
10 papers · 1 filter
Simplifying Momentum-based Positive-definite Submanifold Optimization with Applications to Deep Learning
Wu Lin, Valentin Duruisseaux, Melvin Leok +3
Riemannian submanifold optimization with momentum is computationally challenging because, to ensure that the iterates remain on the submanifold, we often need to solve difficult di…
Structured second-order methods via natural gradient descent
Wu Lin, Frank Nielsen, Mohammad Emtiyaz Khan +1
In this paper, we propose new structured second-order methods and structured adaptive-gradient methods obtained by performing natural-gradient descent on structured parameter space…
Tractable structured natural gradient descent using local parameterizations
Wu Lin, Frank Nielsen, Mohammad Emtiyaz Khan +1
Natural-gradient descent (NGD) on structured parameter spaces (e.g., low-rank covariances) is computationally challenging due to difficult Fisher-matrix computations. We address th…
Handling the Positive-Definite Constraint in the Bayesian Learning Rule
Wu Lin, Mark Schmidt, Mohammad Emtiyaz Khan
The Bayesian learning rule is a natural-gradient variational inference method, which not only contains many existing learning algorithms as special cases but also enables the desig…
Stein's Lemma for the Reparameterization Trick with Exponential Family Mixtures
Wu Lin, Mohammad Emtiyaz Khan, Mark Schmidt
Stein's method (Stein, 1973; 1981) is a powerful tool for statistical applications and has significantly impacted machine learning. Stein's lemma plays an essential role in Stein's…
Fast and Simple Natural-Gradient Variational Inference with Mixture of Exponential-family Approximations
Wu Lin, Mohammad Emtiyaz Khan, Mark Schmidt
Natural-gradient methods enable fast and simple algorithms for variational inference, but due to computational difficulties, their use is mostly limited to \emph{minimal} exponenti…
Faster Stochastic Variational Inference using Proximal-Gradient Methods with General Divergence Functions
Mohammad Emtiyaz Khan, Reza Babanezhad, Wu Lin +2
Several recent works have explored stochastic gradient methods for variational inference that exploit the geometry of the variational-parameter space. However, the theoretical prop…
Non-Uniform Stochastic Average Gradient Method for Training Conditional Random Fields
Mark Schmidt, Reza Babanezhad, Mohamed Osama Ahmed +3
We apply stochastic average gradient (SAG) algorithms for training conditional random fields (CRFs). We describe a practical implementation that uses structure in the CRF gradient…
Group Sparse Priors for Covariance Estimation
Benjamin Marlin, Mark Schmidt, Kevin Murphy
Recently it has become popular to learn sparse Gaussian graphical models (GGMs) by imposing l1 or group l1,2 penalties on the elements of the precision matrix. Thispenalized likeli…
Modeling Discrete Interventional Data using Directed Cyclic Graphical Models
Mark Schmidt, Kevin Murphy
We outline a representation for discrete multivariate distributions in terms of interventional potential functions that are globally normalized. This representation can be used to…