22 citations · 36 across the 5 of their papers we have counts for
7 papers · 1 filter
Aurora: A Leverage-Aware Spectral Optimizer
Alec Dewulf, Dhruv Pai, Li Yang +2
We show that for tall matrix parameters, like projection matrices in the MLP layers, the Muon update can have row norms that are arbitrarily non-uniform. This can lead to a self-re…
Dominant-Layer ZO: A Single Layer Dominates Zeroth-Order Fine-Tuning of LLMs
Wanhao Yu, Ziyan Wang, Zheng Wang +7
Zeroth-order (ZO) optimization enables memory-efficient fine-tuning of large language models (LLMs) using only forward passes, but it remains unclear how useful adaptation is distr…
Beyond Not-Forgetting: Continual Learning with Backward Knowledge Transfer
Sen Lin, Li Yang, Deliang Fan +1
By learning a sequence of tasks continually, an agent in continual learning (CL) can improve the learning performance of both a new task and `old' tasks by leveraging the forward k…
TRGP: Trust Region Gradient Projection for Continual Learning
Sen Lin, Li Yang, Deliang Fan +1
Catastrophic forgetting is one of the major challenges in continual learning. To address this issue, some existing methods put restrictive constraints on the optimization space of…
GROWN: GRow Only When Necessary for Continual Learning
Li Yang, Sen Lin, Junshan Zhang +1
Catastrophic forgetting is a notorious issue in deep learning, referring to the fact that Deep Neural Networks (DNN) could forget the knowledge about earlier tasks when learning ne…
RA-BNN: Constructing Robust & Accurate Binary Neural Network to Simultaneously Defend Adversarial Bit-Flip Attack and Improve Accuracy
Adnan Siraj Rakin, Li Yang, Jingtao Li +5
Recently developed adversarial weight attack, a.k.a. bit-flip attack (BFA), has shown enormous success in compromising Deep Neural Network (DNN) performance with an extremely small…