6 citations · 14 across the 16 of their papers we have counts for
14 papers · 1 filter
GRZO: Group-Relative Zeroth-Order Optimization for Large Language Model Fine-Tuning
Liyan Tan, Yequan Zhao, Yifan Yang +3
Zeroth-order (ZO) optimization is a memory-efficient alternative to backpropagation for fine-tuning large language models, but its deployment is limited by the high variance of gra…
FuRA: Full-Rank Parameter-Efficient Fine-Tuning with Spectral Preconditioning
Yequan Zhao, Ruijie Zhang, Liyan Tan +3
Both full fine-tuning (Full FT) and parameter-efficient fine-tuning methods such as LoRA introduce weight updates without accounting for the spectral structure established during p…
MuonQ: Enhancing Low-Bit Muon Quantization via Directional Fidelity Optimization
Yupeng Su, Ruijie Zhang, Ziyue Liu +2
The Muon optimizer has emerged as a compelling alternative to Adam for training large language models, achieving remarkable computational savings through gradient orthogonalization…
Muon: Boosting Muon via Adaptive Second-Moment Preconditioning
Ziyue Liu, Ruijie Zhang, Zhengyang Wang +4
Muon has emerged as a promising optimizer for large-scale foundation model pre-training by exploiting the matrix structure of neural network updates through iterative orthogonaliza…
TEON: Tensorized Orthonormalization Beyond Layer-Wise Muon for Large Language Model Pre-Training
Ruijie Zhang, Yequan Zhao, Ziyue Liu +5
The Muon optimizer has demonstrated strong empirical performance in pre-training large language models by performing matrix-level gradient (or momentum) orthogonalization in each l…
MUON+: Towards More Effective Muon via One Additional Normalization Step for LLM Pre-training
Ruijie Zhang, Yequan Zhao, Ziyue Liu +4
Muon has recently emerged as a strong optimizer for large language model pre-training, orthogonalizing the momentum matrix via Newton--Schulz polar iterations. A natural intuition…
Tensor-Compressed and Fully-Quantized Training of Neural PDE Solvers
Jinming Lu, Jiayi Tian, Yequan Zhao +2
Physics-Informed Neural Networks (PINNs) have emerged as a promising paradigm for solving partial differential equations (PDEs) by embedding physical laws into neural network train…
QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Models
Jiajun Zhou, Yifan Yang, Kai Zhen +6
Language Models (LLMs) are often quantized to lower precision to reduce the memory cost and latency in inference. However, quantization often degrades model performance, thus fine-…
Scalable Back-Propagation-Free Training of Optical Physics-Informed Neural Networks
Yequan Zhao, Xinling Yu, Xian Xiao +6
Physics intelligence and digital twins often require rapid and repeated performance evaluation of various engineering systems (e.g. robots, autonomous vehicles, semiconductor chips…
Experimental Demonstration of an Optical Neural PDE Solver via On-Chip PINN Training
Yequan Zhao, Xian Xiao, Antoine Descos +6
Partial differential equation (PDE) is an important math tool in science and engineering. This paper experimentally demonstrates an optical neural PDE solver by leveraging the back…
Poor Man's Training on MCUs: A Memory-Efficient Quantized Back-Propagation-Free Approach
Yequan Zhao, Hai Li, Ian Young +1
Back propagation (BP) is the default solution for gradient computation in neural network training. However, implementing BP-based training on various edge devices such as FPGA, mic…
Separable Operator Networks
Xinling Yu, Sean Hooten, Ziyue Liu +4
Operator learning has become a powerful tool in machine learning for modeling complex physical systems governed by partial differential equations (PDEs). Although Deep Operator Net…
Real-Time FJ/MAC PDE Solvers via Tensorized, Back-Propagation-Free Optical PINN Training
Yequan Zhao, Xian Xiao, Xinling Yu +5
Solving partial differential equations (PDEs) numerically often requires huge computing time, energy cost, and hardware resources in practical applications. This has limited their…
Tensor-Compressed Back-Propagation-Free Training for (Physics-Informed) Neural Networks
Yequan Zhao, Xinling Yu, Zhixiong Chen +3
Backward propagation (BP) is widely used to compute the gradients in neural network training. However, it is hard to implement BP on edge devices due to the lack of hardware and so…