Deep Network Approximation: Beyond ReLU to Diverse Activation Functions

Shijun Zhang, Jianfeng Lu, Hongkai Zhao

This paper explores the expressive power of deep neural networks for a diverse range of activation functions. An activation function set $\mathcal{A}$ is defined to encompass the m…