3 papers
cs.CL2025
HybridNorm: Towards Stable and Efficient Transformer Training via Hybrid Normalization
Zhijian Zhuo, Yutao Zeng, Ya Wang +5
Transformers have become the de facto architecture for a wide range of machine learning tasks, particularly in large language models (LLMs). Despite their remarkable performance, m…
cs.CV2025
FwNet-ECA: A Classification Model Enhancing Window Attention with Global Receptive Fields via Fourier Filtering Operations
Shengtian Mian, Ya Wang, Nannan Gu +2
Windowed attention mechanisms were introduced to mitigate the issue of excessive computation inherent in global attention mechanisms. In this paper, we present FwNet-ECA, a novel m…
cs.CL2025
Scale-Distribution Decoupling: Enabling Stable and Effective Training of Large Language Models
Ya Wang, Zhijian Zhuo, Yutao Zeng +3
Training stability is a persistent challenge in the pre-training of large language models (LLMs), particularly for architectures such as Post-Norm Transformers, which are prone to…