Balanced and Elastic End-to-end Training of Dynamic LLMs

Mohamed Wahib, Muhammed Abdullah Soyturk, Didem Unat

To reduce the computational and memory overhead of Large Language Models, various approaches have been proposed. These include a) Mixture of Experts (MoEs), where token routing aff…