Learning specialized activation functions with the Piecewise Linear Unit

Yucong Zhou, Zezhou Zhu, Zhao Zhong

The choice of activation functions is crucial for modern deep neural networks. Popular hand-designed activation functions like Rectified Linear Unit(ReLU) and its variants show pro…