Fine-Tuning Pre-trained Transformers into Decaying Fast Weights

Huanru Henry Mao

Autoregressive Transformers are strong language models but incur O(T) complexity during per-token generation due to the self-attention mechanism. Recent work proposes kernel-based…