Massive Activation Gating Channel in Large Language Models

Minjia Mao, Shi Chen, Bowen Yin +1

Massive activations, a phenomenon in which a small number of hidden channels exhibit exceptionally large magnitudes, are pervasive in large language models (LLMs). However, the mec…