On Approximation Capabilities of ReLU Activation and Softmax Output Layer in Neural Networks

Behnam Asadi, Hui Jiang

In this paper, we have extended the well-established universal approximator theory to neural networks that use the unbounded ReLU activation function and a nonlinear softmax output…