Towards A Unified View of Sparse Feed-Forward Network in Pretraining Large Language Model

Zeyu Leo Liu, Tim Dettmers, Xi Victoria Lin +2

Large and sparse feed-forward layers (S-FFN) such as Mixture-of-Experts (MoE) have proven effective in scaling up Transformers model size for \textit{pretraining} large language mo…