Interior interpretability with attention rollout: contraction and propagation profiles in Transformers

Umberto Biccari, Qian Huang, Enrique Zuazua

Feature-attribution methods assign scores relating input variables to a model's output, but do not by themselves characterize how explicitly defined interaction operators compose a…