Exact gradient updates in time independent of output size for the spherical loss family

Pascal Vincent, Alexandre de Brébisson, Xavier Bouthillier

An important class of problems involves training deep neural networks with sparse prediction targets of very high dimension D. These occur naturally in e.g. neural language models…