Rethinking Activation Functions: Reducing Bias Can Improve Learning

Zirui Peng, Heyi Zheng, Ziming Liu

Many widely used activation functions, including ReLU, GELU, Swish, and Sigmoid, produce predominantly positive outputs, resulting in a nonzero post-activation mean that we term ac…