CAST: Clustering Self-Attention using Surrogate Tokens for Efficient Transformers

Adjorn van Engelenhoven, Nicola Strisciuglio, Estefanía Talavera

The Transformer architecture has shown to be a powerful tool for a wide range of tasks. It is based on the self-attention mechanism, which is an inherently computationally expensiv…