Which the Eye Fears: Writing with Read-Blindness Explains Massive Activations in Transformers

Swagatam Mukhopadhyay, Vishal Vivek Saley, Vraj Parikh +1

Massive activation features (MAs) in Transformers are extreme-value residual-stream features that persist across layers despite the model's ability to suppress them. Why do they su…