1 citations · 1 across the 4 of their papers we have counts for
5 papers
SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learning
Qifan Yu, Xinyu Ma, Zhijian Zhuo +7
Progressive Learning (PL) reduces pre-training computational overhead by gradually increasing model scale. While prior work has extensively explored depth expansion, width expansio…
HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization
Zhijian Zhuo, Yutao Zeng, Ya Wang +5
Transformers have become the de facto architecture for a wide range of machine learning tasks, particularly in large language models (LLMs). Despite their remarkable performance, m…
Scale-Distribution Decoupling: Enabling Stable and Effective Training of Large Language Models
Ya Wang, Zhijian Zhuo, Yutao Zeng +3
Training stability is a persistent challenge in the pre-training of large language models (LLMs), particularly for architectures such as Post-Norm Transformers, which are prone to…
Polynomial Composition Activations: Unleashing the Dynamics of Large Language Models
Zhijian Zhuo, Ya Wang, Yutao Zeng +3
Transformers have found extensive applications across various domains due to the powerful fitting capabilities. This success can be partially attributed to their inherent nonlinear…
Towards a Unified Theoretical Understanding of Non-contrastive Learning via Rank Differential Mechanism
Zhijian Zhuo, Yifei Wang, Jinwen Ma +1
Recently, a variety of methods under the name of non-contrastive learning (like BYOL, SimSiam, SwAV, DINO) show that when equipped with some asymmetric architectural designs, align…