Efficient Sparsely Activated Transformers

Salar Latifi, Saurav Muralidharan, Michael Garland

Transformer-based neural networks have achieved state-of-the-art task performance in a number of machine learning domains including natural language processing and computer vision.…