1 citations · 2 across the 20 of their papers we have counts for
21 papers
Convergence of cyclic steepest descent for quadratics with two distinct eigenvalues
Yu Li, Qihang Wang
The cyclic steepest descent method computes the exact steepest descent stepsize at the start of each cycle and reuses it for a fixed number of iterations. We study this met…
Sharper Analysis of Single-Loop Methods for Bilevel Optimization
Yubo Zhou, Jun Shu, Luo Luo +4
Bilevel optimization underpins many machine learning applications, including hyperparameter optimization, meta-learning, neural architecture search, and reinforcement learning. Whi…
Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning
Yao Fu, Chunxia Zhang, Junmin Liu +3
Generalization remains a pivotal challenge in deep learning, where traditional optimizers like Stochastic Gradient Descent (SGD) often converge to sharp minima, leading to overfitt…
Riemannian Momentum Tracking: Distributed Optimization with Momentum on Compact Submanifolds
Jun Chen, Tianyi Zhu, Haishan Ye +5
Gradient descent with momentum has been widely applied in various signal processing and machine learning tasks, demonstrating a notable empirical advantage over standard gradient d…
On the Convergence of Single-Loop Stochastic Bilevel Optimization with Approximate Implicit Differentiation
Yubo Zhou, Luo Luo, Guang Dai +1
Stochastic Bilevel Optimization has emerged as a fundamental framework for meta-learning and hyperparameter optimization. Despite the practical prevalence of single-loop algorithms…
ESSAM: A Novel Competitive Evolution Strategies Approach to Reinforcement Learning for Memory Efficient LLMs Fine-Tuning
Zhishen Sun, Sizhe Dang, Guang Dai +1
Reinforcement learning (RL) has become a key training step for improving mathematical reasoning in large language models (LLMs), but it often has high GPU memory usage, which makes…
Zero-Order Sharpness-Aware Minimization
Yao Fu, Yihang Jin, Chunxia Zhang +3
Prompt learning has become a key method for adapting large language models to specific tasks with limited data. However, traditional gradient-based optimization methods for tuning…
MSCR: Exploring the Vulnerability of LLMs' Mathematical Reasoning Abilities Using Multi-Source Candidate Replacement
Zhishen Sun, Guang Dai, Haishan Ye
LLMs demonstrate performance comparable to human abilities in complex tasks such as mathematical reasoning, but their robustness in mathematical reasoning under minor input perturb…
Numerical Sensitivity and Robustness: Exploring the Flaws of Mathematical Reasoning in Large Language Models
Zhishen Sun, Guang Dai, Ivor Tsang +1
LLMs have made significant progress in the field of mathematical reasoning, but whether they have true the mathematical understanding ability is still controversial. To explore thi…
A Unified Zeroth-Order Optimization Framework via Oblivious Randomized Sketching
Haishan Ye, Xiangyu Chang, Xi Chen
We propose a new framework for analyzing zeroth-order optimization (ZOO) from the perspective of \emph{oblivious randomized sketching}.In this framework, commonly used gradient est…
Can a One-Point Feedback Zeroth-order Algorithm Achieve Linear Dimension Dependent Sample Complexity?
Haishan Ye, Xiangyu Chang
We revisit the one-point feedback zeroth-order (ZO) optimization problem, a classical setting in derivative-free optimization where only a single noisy function evaluation is avail…
FZOO: Fast Zeroth-Order Optimizer for Fine-Tuning Large Language Models towards Adam-Scale Speed
Sizhe Dang, Yangyang Guo, Yanjun Zhao +4
Fine-tuning large language models (LLMs) often faces GPU memory bottlenecks: the backward pass of first-order optimizers like Adam increases memory usage to more than 10 times the…
Stochastic Diagonal Estimation Based on Matrix Quadratic Form Oracles
Haishan Ye, Xiangyu Chang
We study the problem of estimating the diagonal of an implicitly given matrix $\Ab$. For such a matrix we have access to an oracle that allows us to evaluate the matrix quadratic f…
Privacy Leaks by Adversaries: Adversarial Iterations for Membership Inference Attack
Jing Xue, Zhishen Sun, Haishan Ye +4
Membership inference attack (MIA) has become one of the most widely used and effective methods for evaluating the privacy risks of machine learning models. These attacks aim to det…
Towards Understanding The Calibration Benefits of Sharpness-Aware Minimization
Chengli Tan, Yubo Zhou, Haishan Ye +7
Deep neural networks have been increasingly used in safety-critical applications such as medical diagnosis and autonomous driving. However, many studies suggest that they are prone…
Breaking the Prompt Wall (I): A Real-World Case Study of Attacking ChatGPT via Lightweight Prompt Injection
Xiangyu Chang, Guang Dai, Hao Di +1
This report presents a real-world case study demonstrating how prompt injection can attack large language model platforms such as ChatGPT according to a proposed injection framewor…
An Enhanced Zeroth-Order Stochastic Frank-Wolfe Framework for Constrained Finite-Sum Optimization
Haishan Ye, Yinghui Huang, Hao Di +1
We propose an enhanced zeroth-order stochastic Frank-Wolfe framework to address constrained finite-sum optimization problems, a structure prevalent in large-scale machine-learning…
Double Variance Reduction: A Smoothing Trick for Composite Optimization Problems without First-Order Gradient
Hao Di, Haishan Ye, Yueling Zhang +3
Variance reduction techniques are designed to decrease the sampling variance, thereby accelerating convergence rates of first-order (FO) and zeroth-order (ZO) optimization methods.…
Anderson Acceleration Without Restart: A Novel Method with -Step Super Quadratic Convergence Rate
Haishan Ye, Dachao Lin, Xiangyu Chang +1
In this paper, we propose a novel Anderson's acceleration method to solve nonlinear equations, which does \emph{not} require a restart strategy to achieve numerical stability. We p…
Second-Order Fine-Tuning without Pain for LLMs:A Hessian Informed Zeroth-Order Optimizer
Yanjun Zhao, Sizhe Dang, Haishan Ye +3
Fine-tuning large language models (LLMs) with classic first-order optimizers entails prohibitive GPU memory due to the backpropagation process. Recent works have turned to zeroth-o…
PPFL: A Personalized Federated Learning Framework for Heterogeneous Population
Hao Di, Yi Yang, Haishan Ye +1
Personalization aims to characterize individual preferences and is widely applied across many fields. However, conventional personalized methods operate in a centralized manner, po…