4 papers
PC Layer: Polynomial Weight Preconditioning for Improving LLM Pre-Training
Senmiao Wang, Tiantian Fang, Haoran Zhang +4
We propose a preconditioning (PC) layer, a weight parameterization via polynomial preconditioner that ensures stable weight conditioning throughout LLM training. The PC module resh…
YouZhi: Towards High-Concurrency Financial LLMs via Adaptive GQA-to-MLA Transition
PSBC LLM Team, Huawei LLM Team, Ruihan Long +56
Large language models (LLMs) drive significant financial innovations, yet their high-concurrency deployment is severely bottlenecked by KV cache memory overhead, which inflates inf…
VCORE: Variance-Controlled Optimization-based Reweighting for Chain-of-Thought Supervision
Xuan Gong, Senmiao Wang, Hanbo Huang +2
Supervised fine-tuning (SFT) on long chain-of-thought (CoT) trajectories has emerged as a crucial technique for enhancing the reasoning abilities of large language models (LLMs). H…
Exploring and Improving Initialization for Deep Graph Neural Networks: A Signal Propagation Perspective
Senmiao Wang, Yupeng Chen, Yushun Zhang +2
Graph Neural Networks (GNNs) often suffer from performance degradation as the network depth increases. This paper addresses this issue by introducing initialization methods that en…