Spark Transformer: Reactivating Sparsity in FFN and Attention

Chong You, Kan Wu, Zhipeng Jia +16

The discovery of the lazy neuron phenomenon in trained Transformers, where the vast majority of neurons in their feed-forward networks (FFN) are inactive for each token, has spurre…