Massive Activations in Large Language Models

Mingjie Sun, Xinlei Chen, J. Zico Kolter +1

We observe an empirical phenomenon in Large Language Models (LLMs) -- very few activations exhibit significantly larger values than others (e.g., 100,000 times larger). We call the…