On Controllable Sparse Alternatives to Softmax

Abstract

Converting an n-dimensional vector to a probability distribution over n objects is a commonly used component in many machine learning tasks like multiclass classification, multilabel classification, attention mechanisms etc. For this, several probability mapping functions have been proposed and employed in literature such as softmax, sum-normalization, spherical softmax, and sparsemax, but there is very little understanding in terms how they relate with each other. Further, none of the above formulations offer an explicit control over the degree of sparsity. To address this, we develop a unified framework that encompasses all these formulations as special cases. This framework ensures simple closed-form solutions and existence of sub-gradients suitable for learning via backpropagation. Within this framework, we propose two novel sparse formulations, sparsegen-lin and sparsehourglass, that seek to provide a control over the degree of desired sparsity. We further develop novel convex loss functions that help induce the behavior of aforementioned formulations in the multilabel classification setting, showing improved performance. We also demonstrate empirically that the proposed formulations, when used to compute attention weights, achieve better or comparable performance on standard seq2seq tasks like neural machine translation and abstractive summarization.

arXiv commentTo appear in NIPS 2018, Total 16 pages including appendix

References in corpus (2)

• From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification
• On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learning

Cited by in corpus (9)

• Towards Interpretable Sparse Graph Representation Learning with Laplacian Pooling
• $O(n)$ Connections are Expressive Enough: Universal Approximability of Sparse Transformers
• Know Your Limits: Uncertainty Estimation with ReLU Classifiers Fails at Reliable OOD Detection
• Evidential Sparsification of Multimodal Latent Spaces in Conditional Variational Autoencoders
• Sparse Communication via Mixed Distributions
• Yet Another Representation of Binary Decision Trees: A Mathematical Demonstration
• Context-aware Non-linear and Neural Attentive Knowledge-based Models for Grade Prediction
• Decision Machines: Congruent Decision Trees
• Optimal Approximation -- Smoothness Tradeoffs for Soft-Max Functions