1 citations · 1 across the 12 of their papers we have counts for
13 papers
How Does Local Landscape Geometry Evolve in Language Model Pre-Training?
Zhanpeng Zhou, Yuhan Sun, Bingrui Li +4
The scale and expense of pre-training language models make efficient hyperparameter tuning essential, yet a principled guidance is still missing. In this work, we analyze language…
The MiniMax-M2 Series: Mini Activations Unleashing Max Real-World Intelligence
Aili Chen, Aonian Li, Baichuan Zhou +215
We introduce the MiniMax-M2 series, a family of Mixture-of-Experts language models built around the principle that mini activations can unleash maximum real-world intelligence. The…
How Transformers Learn to Plan via Multi-Token Prediction
Jianhao Huang, Zhanpeng Zhou, Renqiu Xia +3
While next-token prediction (NTP) has been the standard objective for training language models, it often struggles to capture global structure in reasoning tasks. Multi-token predi…
IGU-LoRA: Adaptive Rank Allocation via Integrated Gradients and Uncertainty-Aware Scoring
Xuan Cui, Huiyue Li, Run Zeng +5
As large language models (LLMs) scale to billions of parameters, full-parameter fine-tuning becomes compute- and memory-prohibitive. Parameter-efficient fine-tuning (PEFT) mitigate…
On the Learning Dynamics of Two-layer Linear Networks with Label Noise SGD
Tongcheng Zhang, Zhanpeng Zhou, Mingze Wang +4
One crucial factor behind the success of deep learning lies in the implicit bias induced by noise inherent in gradient-based training algorithms. Motivated by empirical observation…
Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Laws
Jinbo Wang, Binghui Li, Zhanpeng Zhou +5
Batch size scheduling (BSS) plays a critical role in large-scale deep learning training, influencing both optimization dynamics and computational efficiency. Yet, its theoretical f…
Efficient Hyperparameter Tuning via Trajectory Invariance Principle
Bingrui Li, Jiaxin Wen, Zhanpeng Zhou +2
As hyperparameter tuning becomes increasingly costly at scale, efficient tuning methods are essential. Yet principles for guiding hyperparameter tuning remain limited. In this work…
On the Surprising Effectiveness of a Single Global Merging in Decentralized Learning
Tongtian Zhu, Tianyu Zhang, Mingze Wang +2
Decentralized learning provides a scalable alternative to parameter-server-based training, yet its performance is often hindered by limited peer-to-peer communication. In this pape…
On Path to Multimodal Historical Reasoning: HistBench and HistAgent
Jiahao Qiu, Fulian Xiao, Yimin Wang +96
Recent advances in large language models (LLMs) have led to remarkable progress across domains, yet their capabilities in the humanities, particularly history, remain underexplored…
On the Role of Label Noise in the Feature Learning Process
Andi Han, Wei Huang, Zhanpeng Zhou +5
Deep learning with noisy labels presents significant challenges. In this work, we theoretically characterize the role of label noise from a feature learning perspective. Specifical…
The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training
Jinbo Wang, Mingze Wang, Zhanpeng Zhou +3
Transformers consist of diverse building blocks, such as embedding layers, normalization layers, self-attention mechanisms, and point-wise feedforward networks. Thus, understanding…
Sharpness-Aware Minimization Efficiently Selects Flatter Minima Late in Training
Zhanpeng Zhou, Mingze Wang, Yuchen Mao +2
Sharpness-Aware Minimization (SAM) has substantially improved the generalization of neural networks under various settings. Despite the success, its effectiveness remains poorly un…
On the Optimization and Generalization of Two-layer Transformers with Sign Gradient Descent
Bingrui Li, Wei Huang, Andi Han +4
The Adam optimizer is widely used for transformer optimization in practice, which makes understanding the underlying optimization mechanisms an important problem. However, due to t…