BERT Busters: Outlier Dimensions that Disrupt Transformers

Olga Kovaleva, Saurabh Kulshreshtha, Anna Rogers +1

Multiple studies have shown that Transformers are remarkably robust to pruning. Contrary to this received wisdom, we demonstrate that pre-trained Transformer encoders are surprisin…