A Theoretical Explanation of Activation Sparsity through Flat Minima and Adversarial Robustness

Ze Peng, Lei Qi, Yinghuan Shi +1

A recent empirical observation (Li et al., 2022b) of activation sparsity in MLP blocks offers an opportunity to drastically reduce computation costs for free. Although having attri…