Progressive Behavioral Drift through Compression Valleys in Large Language Models

Zhiyuan Xu, Stanislav Abaimov, Joseph Gardiner +1

We show that attention sinks and compression valleys create a vulnerable region in decoder-only Transformers, where small activation perturbations can be amplified through the auto…