A Pipelined FPGA Architecture for Banded Sparse Matrix Dense Matrix Multiplication in Longformer

Phillip Pramberger, Athanasios Tziouvaras, Shreejith Shanker +1

Sparse attention mechanisms have become increasingly important for transformer models processing long input sequences due to their lower computational and memory complexity compare…