Achieving Sparse Activation in Small Language Models

Jifeng Song, Kai Huang, Xiangyu Yin +2

Sparse activation, which selectively activates only an input-dependent set of neurons in inference, is a useful technique to reduce the computing cost of Large Language Models (LLM…