Multi-Bias Non-linear Activation in Deep Neural Networks

Hongyang Li, Wanli Ouyang, Xiaogang Wang

As a widely used non-linear activation, Rectified Linear Unit (ReLU) separates noise and signal in a feature map by learning a threshold or bias. However, we argue that the classif…