Self-Ablating Transformers: More Interpretability, Less Sparsity

Jeremias Ferrao, Luhan Mikaelson, Keenan Pepper +1

A growing intuition in machine learning suggests a link between sparsity and interpretability. We introduce a novel self-ablation mechanism to investigate this connection ante-hoc…