28 citations · 43 across the 22 of their papers we have counts for
4 papers · 1 filter
On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-training
Xueyan Niu, Bo Bai, Wei Han +1
Post-training of large language models routinely interleaves supervised fine-tuning (SFT) with reinforcement learning (RL). These two methods have different objectives: SFT minimiz…
Towards Faster Graph Partitioning via Pre-training and Inductive Inference
Meng Qin, Chaorui Zhang, Yu Gao +5
Graph partitioning (GP) is a classic problem that divides the node set of a graph into densely-connected blocks. Following the IEEE HPEC Graph Challenge and recent advances in pre-…
A Mean Field Ansatz for Zero-Shot Weight Transfer
Xingyuan Chen, Wenwei Kuang, Lei Deng +3
The pre-training cost of large language models (LLMs) is prohibitive. One cutting-edge approach to reduce the cost is zero-shot weight transfer, also known as model growth for some…
Beyond Scaling Laws: Understanding Transformer Performance with Associative Memory
Xueyan Niu, Bo Bai, Lei Deng +1
Increasing the size of a Transformer does not always lead to enhanced performance. This phenomenon cannot be explained by the empirical scaling laws. Furthermore, the model's enhan…