Uncovering Layer-Dependent Activation Sparsity Patterns in ReLU Transformers

Cody Wild, Jesper Anderson

Previous work has demonstrated that MLPs within ReLU Transformers exhibit high levels of sparsity, with many of their activations equal to zero for any given token. We build on tha…