1 citations · 2 across the 20 of their papers we have counts for
6 papers · 1 filter
Convergence of cyclic steepest descent for quadratics with two distinct eigenvalues
Yu Li, Qihang Wang
The cyclic steepest descent method computes the exact steepest descent stepsize at the start of each cycle and reuses it for a fixed number of iterations. We study this met…
Sharper Analysis of Single-Loop Methods for Bilevel Optimization
Yubo Zhou, Jun Shu, Luo Luo +4
Bilevel optimization underpins many machine learning applications, including hyperparameter optimization, meta-learning, neural architecture search, and reinforcement learning. Whi…
Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning
Yao Fu, Chunxia Zhang, Junmin Liu +3
Generalization remains a pivotal challenge in deep learning, where traditional optimizers like Stochastic Gradient Descent (SGD) often converge to sharp minima, leading to overfitt…
Riemannian Momentum Tracking: Distributed Optimization with Momentum on Compact Submanifolds
Jun Chen, Tianyi Zhu, Haishan Ye +5
Gradient descent with momentum has been widely applied in various signal processing and machine learning tasks, demonstrating a notable empirical advantage over standard gradient d…
On the Convergence of Single-Loop Stochastic Bilevel Optimization with Approximate Implicit Differentiation
Yubo Zhou, Luo Luo, Guang Dai +1
Stochastic Bilevel Optimization has emerged as a fundamental framework for meta-learning and hyperparameter optimization. Despite the practical prevalence of single-loop algorithms…
ESSAM: A Novel Competitive Evolution Strategies Approach to Reinforcement Learning for Memory Efficient LLMs Fine-Tuning
Zhishen Sun, Sizhe Dang, Guang Dai +1
Reinforcement learning (RL) has become a key training step for improving mathematical reasoning in large language models (LLMs), but it often has high GPU memory usage, which makes…