Activation function design for deep networks: linearity and effective initialisation

Michael Murray, Vinayak Abrol, Jared Tanner

The activation function deployed in a deep neural network has great influence on the performance of the network at initialisation, which in turn has implications for training. In t…