Everything in its right place: attention is a radio head

Lluís Arola-Fernández, Aissam Khadraoui, Alex Arenas

Attention, the core of language models, is a digital calculation over tokens: an exponential selects them, a division normalizes and a positional code sets the order. We show that…