WildCat: Near-Linear Attention in Theory and Practice

Tobias Schröder, Lester Mackey

We introduce WildCat, a high-accuracy, low-cost approach to compressing the attention mechanism in neural networks. While attention is a staple of modern network architectures, it…