6 citations · 20 across the 9 of their papers we have counts for
9 papers
Effective Bilevel Optimization via Minimax Reformulation
Xiaoyu Wang, Rui Pan, Renjie Pi +1
Bilevel optimization has found successful applications in various machine learning problems, including hyper-parameter optimization, data cleaning, and meta-learning. However, its…
Generalized Polyak Step Size for First Order Optimization with Momentum
Xiaoyu Wang, Mikael Johansson, Tong Zhang
In machine learning applications, it is well known that carefully designed learning rate (step size) schedules can significantly improve the convergence of commonly used first-orde…
A Lifted Bregman Formulation for the Inversion of Deep Neural Networks
Xiaoyu Wang, Martin Benning
We propose a novel framework for the regularised inversion of deep neural networks. The framework is based on the authors' recent work on training feed-forward neural networks with…
Zeroth-Order Randomized Subspace Newton Methods
Erik Berglund, Sarit Khirirat, Xiaoyu Wang
Zeroth-order methods have become important tools for solving problems where we have access only to function evaluations. However, the zeroth-order methods only using gradient appro…
On Uniform Boundedness Properties of SGD and its Momentum Variants
Xiaoyu Wang, Mikael Johansson
A theoretical, and potentially also practical, problem with stochastic gradient descent is that trajectories may escape to infinity. In this note, we investigate uniform boundednes…
Bandwidth-based Step-Sizes for Non-Convex Stochastic Optimization
Xiaoyu Wang, Mikael Johansson
Many popular learning-rate schedules for deep neural networks combine a decaying trend with local perturbations that attempt to escape saddle points and bad local minima. We derive…
On the Convergence of Step Decay Step-Size for Stochastic Optimization
Xiaoyu Wang, Sindri Magnússon, Mikael Johansson
The convergence of stochastic gradient descent is highly dependent on the step-size, especially on non-convex problems such as neural network training. Step decay step-size schedul…
On the Convergence of Stochastic Gradient Descent with Bandwidth-based Step Size
Xiaoyu Wang, Ya-xiang Yuan
We investigate the stochastic gradient descent (SGD) method where the step size lies within a banded region instead of being given by a fixed formula. The optimal convergence rate…
Stochastic Trust Region Methods with Trust Region Radius Depending on Probabilistic Models
Xiaoyu Wang, Ya-xiang Yuan
We present a stochastic trust-region model-based framework in which its radius is related to the probabilistic models. Especially, we propose a specific algorithm, termed STRME, in…