Attentions Under the Microscope: A Comparative Study of Resource Utilization for Variants of Self-Attention

Zhengyu Tian, Anantha Padmanaban Krishna Kumar, Hemant Krishnakumar +1

As large language models (LLMs) and visual language models (VLMs) grow in scale and application, attention mechanisms have become a central computational bottleneck due to their hi…