Train Separately, Merge Together: Modular Post-Training with Mixture-of-Experts

Jacob Morrison, Sanjay Adhikesaven, Akshita Bhagia +3

Extending a fully post-trained language model with new domain capabilities is fundamentally limited by monolithic training paradigms: retraining from scratch is expensive and scale…