1 citations · 1 across the 4 of their papers we have counts for
4 papers · 1 filter
MONA: Muon Optimizer with Nesterov Acceleration for Scalable Language Model Training
Jiacheng Li, Jianchao Tan, Hongtao Xu +5
The Muon optimizer has recently offered a promising alternative to AdamW for large language model training, leveraging matrix orthogonalization to produce geometry-aware updates. H…
AFA-LoRA: Enabling Non-Linear Adaptations in LoRA with Activation Function Annealing
Jiacheng Li, Jianchao Tan, Zhidong Yang +4
Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient fine-tuning (PEFT) method. However, its linear adaptation process limits its expressive power. This means there i…
Exploring Landscapes for Better Minima along Valleys
Tong Zhao, Jiacheng Li, Yuanchang Zhou +2
Finding lower and better-generalizing minima is crucial for deep learning. However, most existing optimizers stop searching the parameter space once they reach a local minimum. Giv…
WISCA: A Lightweight Model Transition Method to Improve LLM Training via Weight Scaling
Jiacheng Li, Jianchao Tan, Zhidong Yang +11
Transformer architecture gradually dominates the LLM field. Recent advances in training optimization for Transformer-based large language models (LLMs) primarily focus on architect…