LUCID: Attention with Preconditioned Representations

Sai Surya Duvvuri, Nirmal Patel, Nilesh Gupta +1

Softmax-based dot-product attention is a cornerstone of Transformer architectures, enabling remarkable capabilities such as in-context learning. However, as context lengths increas…