ReLU${}^2$ Wins: Discovering Efficient Activation Functions for Sparse LLMs

Zhengyan Zhang, Yixin Song, Guanghui Yu +7

Sparse computation offers a compelling solution for the inference of Large Language Models (LLMs) in low-resource scenarios by dynamically skipping the computation of inactive neur…