112 citations · 119 across the 9 of their papers we have counts for
9 papers
Curvature-Informed SGD via General Purpose Lie-Group Preconditioners
Omead Pooladzandi, Xi-Lin Li
We present a novel approach to accelerate stochastic gradient descent (SGD) by utilizing curvature information obtained from Hessian-vector products or finite differences of parame…
Stochastic Hessian Fittings with Lie Groups
Xi-Lin Li
This report investigates the fitting of the Hessian or its inverse for stochastic optimizations using a Hessian fitting criterion derived from the preconditioned stochastic gradien…
Independent Vector Analysis with Deep Neural Network Source Priors
Xi-Lin Li
This paper studies the density priors for independent vector analysis (IVA) with convolutive speech mixture separation as the exemplary application. Most existing source priors for…
A Triangular Network For Density Estimation
Xi-Lin Li
We report a triangular neural network implementation of neural autoregressive flow (NAF). Unlike many universal autoregressive density models, our design is highly modular, paramet…
A Multiclass Multiple Instance Learning Method with Exact Likelihood
Xi-Lin Li
We study a multiclass multiple instance learning (MIL) problem where the labels only suggest whether any instance of a class exists or does not exist in a training sample or exampl…
Preconditioner on Matrix Lie Group for SGD
Xi-Lin Li
We study two types of preconditioners and preconditioned stochastic gradient descent (SGD) methods in a unified framework. We call the first one the Newton type due to its close re…
Online Second Order Methods for Non-Convex Stochastic Optimizations
Xi-Lin Li
This paper proposes a family of online second order methods for possibly non-convex stochastic optimizations based on the theory of preconditioned stochastic gradient descent (PSGD…
Recurrent neural network training with preconditioned stochastic gradient descent
Xi-Lin Li
This paper studies the performance of a recently proposed preconditioned stochastic gradient descent (PSGD) algorithm on recurrent neural network (RNN) training. PSGD adaptively es…
Preconditioned Stochastic Gradient Descent
Xi-Lin Li
Stochastic gradient descent (SGD) still is the workhorse for many practical problems. However, it converges slow, and can be difficult to tune. It is possible to precondition SGD t…