Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Models

Siqi Wang, Zhengyu Chen, Bei Li +3

The scaling of large language models (LLMs) is a critical research area for the efficiency and effectiveness of model training and deployment. Our work investigates the transferabi…