Training Foundation Models with a Single Algorithm

Yuda Song, Alvin Zhang, Fahim Tajwar +7

Foundation models are typically trained in stages: pretraining, midtraining, supervised fine-tuning (SFT), and reinforcement learning (RL), with changes in both data and learning a…