6 citations · 14 across the 16 of their papers we have counts for
7 papers · 1 filter
ZOAF: Towards Efficient Zeroth-Order Optimization for Analog/RF Circuit Design
Liyan Tan, Yequan Zhao, Jinming Lu +3
Circuit optimization is an indispensable step in analog/RF IC design. Classical fast gradient-based optimization methods are typically infeasible due to lack of access to simulator…
GRZO: Group-Relative Zeroth-Order Optimization for Large Language Model Fine-Tuning
Liyan Tan, Yequan Zhao, Yifan Yang +3
Zeroth-order (ZO) optimization is a memory-efficient alternative to backpropagation for fine-tuning large language models, but its deployment is limited by the high variance of gra…
FuRA: Full-Rank Parameter-Efficient Fine-Tuning with Spectral Preconditioning
Yequan Zhao, Ruijie Zhang, Liyan Tan +3
Both full fine-tuning (Full FT) and parameter-efficient fine-tuning methods such as LoRA introduce weight updates without accounting for the spectral structure established during p…
MuonQ: Enhancing Low-Bit Muon Quantization via Directional Fidelity Optimization
Yupeng Su, Ruijie Zhang, Ziyue Liu +2
The Muon optimizer has emerged as a compelling alternative to Adam for training large language models, achieving remarkable computational savings through gradient orthogonalization…
Muon: Boosting Muon via Adaptive Second-Moment Preconditioning
Ziyue Liu, Ruijie Zhang, Zhengyang Wang +4
Muon has emerged as a promising optimizer for large-scale foundation model pre-training by exploiting the matrix structure of neural network updates through iterative orthogonaliza…
TEON: Tensorized Orthonormalization Beyond Layer-Wise Muon for Large Language Model Pre-Training
Ruijie Zhang, Yequan Zhao, Ziyue Liu +5
The Muon optimizer has demonstrated strong empirical performance in pre-training large language models by performing matrix-level gradient (or momentum) orthogonalization in each l…
MUON+: Towards More Effective Muon via One Additional Normalization Step for LLM Pre-training
Ruijie Zhang, Yequan Zhao, Ziyue Liu +4
Muon has recently emerged as a strong optimizer for large language model pre-training, orthogonalizing the momentum matrix via Newton--Schulz polar iterations. A natural intuition…