Reinforcement Learning on Pre-Training Data

Siheng Li, Kejiao Li, Zenan Xu +33

The growing disparity between the exponential scaling of computational resources and the finite growth of high-quality text data now constrains conventional scaling approaches for…