PowLU: An Activation Function for Stable Pre-Training of LLMs

Peijie Jiang, Yuqi Feng, Cunyin Peng +5

In contemporary large language models (LLMs), the swish-gated linear unit (SwiGLU) activation function is widely adopted to regulate the information flow and introduce non-linearit…