From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification

André F. T. Martins, Ramón Fernandez Astudillo

We propose sparsemax, a new activation function similar to the traditional softmax, but able to output sparse probabilities. After deriving its properties, we show how its Jacobian…