Be Careful What You Backpropagate: A Case For Linear Output Activations & Gradient Boosting

Anders Oland, Aayush Bansal, Roger B. Dannenberg +1

In this work, we show that saturating output activation functions, such as the softmax, impede learning on a number of standard classification tasks. Moreover, we present results s…