House of Cards: Massive Weights in LLMs

Jaehoon Oh, Seungjun Shin, Dokwan Oh

Massive activations, which manifest in specific feature dimensions of hidden states, introduce a significant bias in large language models (LLMs), leading to an overemphasis on the…