cMoLLM at Scale: Horizontal Scaling Laws for Mixture-of-LLMs

Xin Yang, Yemin Wang, Mingda Liu +4

Scaling large language models (LLMs) has driven their success, yet dense Transformers couple capacity and computation: every parameter is activated for every token, making training…