5 papers
AGZO: Activation-Guided Zeroth-Order Optimization for LLM Fine-Tuning
Wei Lin, Yining Jiang, Qingyu Song +2
Zeroth-Order (ZO) optimization has emerged as a promising solution for fine-tuning LLMs under strict memory constraints, as it avoids the prohibitive memory cost of storing activat…
The Multi-Query Paradox in Zeroth-Order Optimization
Wei Lin, Qingyu Song, Hong Xu
Zeroth-order (ZO) optimization provides a powerful framework for problems where explicit gradients are unavailable and have to be approximated using only queries to function value.…
Learning Provably Improves the Convergence of Gradient Descent
Qingyu Song, Wei Lin, Hong Xu
Learn to Optimize (L2O) trains deep neural network-based solvers for optimization, achieving success in accelerating convex problems and improving non-convex solutions. However, L2…
Towards Robust Learning to Optimize with Theoretical Guarantees
Qingyu Song, Wei Lin, Juncheng Wang +1
Learning to optimize (L2O) is an emerging technique to solve mathematical optimization problems with learning-based methods. Although with great success in many real-world scenario…
A Learn-to-Optimize Approach for Coordinate-Wise Step Sizes for Quasi-Newton Methods
Wei Lin, Qingyu Song, Hong Xu
Tuning step sizes is crucial for the stability and efficiency of optimization algorithms. While adaptive coordinate-wise step sizes have been shown to outperform scalar step size i…