The Z-loss: a shift and scale invariant classification loss belonging to the Spherical Family

Alexandre de Brébisson, Pascal Vincent

Despite being the standard loss function to train multi-class neural networks, the log-softmax has two potential limitations. First, it involves computations that scale linearly wi…