More Expressive Feedforward Layers: Part I. Token-Adaptive Mixing of Activations

Mingze Wang, Jinbo Wang, Yikuan Xia +2

Feedforward network (FFN) layers account for a large fraction of parameters and nonlinear expressivity in Transformer-based large language models (LLMs). Despite the evolution from…