AReLU: Attention-based Rectified Linear Unit

Dengsheng Chen, Jun Li, Kai Xu

Element-wise activation functions play a critical role in deep neural networks via affecting the expressivity power and the learning dynamics. Learning-based activation functions h…