18 citations · 98 across the 76 of their papers we have counts for
4 papers · 1 filter
Low-rank Optimization Trajectories Modeling for LLM RLVR Acceleration
Zhipeng Chen, Tao Qian, Wayne Xin Zhao +1
Recently, scaling reinforcement learning with verifiable rewards (RLVR) for large language models (LLMs) has emerged as an effective training paradigm for significantly improving m…
Irrational Complex Rotations Empower Low-bit Optimizers
Zhen Tian, Wayne Xin Zhao, Ji-Rong Wen
In this paper, we propose a novel optimizer state compression algorithm, namely -Quant, which leverages the properties of irrational numbers (e.g., ) for memory-efficient tra…
Investigating the Pre-Training Dynamics of In-Context Learning: Task Recognition vs. Task Learning
Xiaolei Wang, Xinyu Tang, Wayne Xin Zhao +1
The emergence of in-context learning (ICL) is potentially attributed to two major abilities: task recognition (TR) for recognizing the task from demonstrations and utilizing pre-tr…
An Integrated Data Processing Framework for Pretraining Foundation Models
Yiding Sun, Feng Wang, Yutao Zhu +2
The ability of the foundation models heavily relies on large-scale, diverse, and high-quality pretraining data. In order to improve data quality, researchers and practitioners ofte…