5 papers
MONA: Muon Optimizer with Nesterov Acceleration for Scalable Language Model Training
Jiacheng Li, Jianchao Tan, Hongtao Xu +5
The Muon optimizer has recently offered a promising alternative to AdamW for large language model training, leveraging matrix orthogonalization to produce geometry-aware updates. H…
WISCA: A Lightweight Model Transition Method to Improve LLM Training via Weight Scaling
Jiacheng Li, Jianchao Tan, Zhidong Yang +11
Transformer architecture gradually dominates the LLM field. Recent advances in training optimization for Transformer-based large language models (LLMs) primarily focus on architect…
AFA-LoRA: Enabling Non-Linear Adaptations in LoRA with Activation Function Annealing
Jiacheng Li, Jianchao Tan, Zhidong Yang +4
Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient fine-tuning (PEFT) method. However, its linear adaptation process limits its expressive power. This means there i…
Exploring Landscapes for Better Minima along Valleys
Tong Zhao, Jiacheng Li, Yuanchang Zhou +2
Finding lower and better-generalizing minima is crucial for deep learning. However, most existing optimizers stop searching the parameter space once they reach a local minimum. Giv…
LongCat-Flash Technical Report
Meituan LongCat Team, Bayan, Bei Li +179
We introduce LongCat-Flash, a 560-billion-parameter Mixture-of-Experts (MoE) language model designed for both computational efficiency and advanced agentic capabilities. Stemming f…