DTRNet: Dynamic Token Routing Network to Reduce Quadratic Costs in Transformers

Aman Sharma, Saeed Najafi, Parsa Farinneya +6

Transformers achieve state-of-the-art results across many tasks, but their uniform application of quadratic self-attention to every token at every layer makes them computationally…