15 citations · 20 across the 11 of their papers we have counts for
6 papers · 1 filter
When Do Larger Batches Help Scale LLM Reinforcement Learning?
Ziniu Li, Jinbo Wang, Guanhua Huang +3
Larger batches reduce the variance of stochastic gradients per update and are therefore often expected to accelerate training. Yet whether this statistical benefit translates into…
More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations
Mingze Wang, Jinbo Wang, Yikuan Xia +2
Feedforward network (FFN) layers account for a large fraction of parameters and nonlinear expressivity in Transformer-based large language models (LLMs). Despite the evolution from…
Fast Catch-Up, Late Switching: Optimal Batch Size Scheduling via Functional Scaling Laws
Jinbo Wang, Binghui Li, Zhanpeng Zhou +5
Batch size scheduling (BSS) plays a critical role in large-scale deep learning training, influencing both optimization dynamics and computational efficiency. Yet, its theoretical f…
GradPower: Powering Gradients for Faster Language Model Pre-Training
Jinbo Wang, Mingze Wang, Jiaqi Zhang +5
We propose GradPower, a lightweight gradient-transformation technique for accelerating language model pre-training. Given a gradient vector , GradPower first applies the…
The Sharpness Disparity Principle in Transformers for Accelerating Language Model Pre-Training
Jinbo Wang, Mingze Wang, Zhanpeng Zhou +3
Transformers consist of diverse building blocks, such as embedding layers, normalization layers, self-attention mechanisms, and point-wise feedforward networks. Thus, understanding…
Improving Generalization and Convergence by Enhancing Implicit Regularization
Mingze Wang, Jinbo Wang, Haotian He +6
In this work, we propose an Implicit Regularization Enhancement (IRE) framework to accelerate the discovery of flat solutions in deep learning, thereby improving generalization and…