Transformers Learn Faster with Semantic Focus

Parikshit Ram, Kenneth L. Clarkson, Tim Klinger +2

Various forms of sparse attention have been explored to mitigate the quadratic computational and memory cost of the attention mechanism in transformers. We study sparse transformer…