Learning with Fenchel-Young Losses

Abstract

Over the past decades, numerous loss functions have been been proposed for a variety of supervised learning tasks, including regression, classification, ranking, and more generally structured prediction. Understanding the core principles and theoretical properties underpinning these losses is key to choose the right loss for the right problem, as well as to create new losses which combine their strengths. In this paper, we introduce Fenchel-Young losses, a generic way to construct a convex loss function for a regularized prediction function. We provide an in-depth study of their properties in a very broad setting, covering all the aforementioned supervised learning tasks, and revealing new connections between sparsity, generalized entropies, and separation margins. We show that Fenchel-Young losses unify many well-known loss functions and allow to create useful new ones easily. Finally, we derive efficient predictive and training algorithms, making Fenchel-Young losses appealing both in theory and practice.

In Journal of Machine Learning Research, volume 21

References in corpus (13)

• Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory
• From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification
• Smooth and Sparse Optimal Transport
• Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances
• On the Equivalence between Herding and Conditional Gradient Algorithms
• Sparse projections onto the simplex
• Metric Learning for Temporal Sequence Alignment
• Top-k Multiclass SVM
• Regularized Optimal Transport and the Rot Mover's Distance
• Barrier Frank-Wolfe for Marginal Inference
• Structured Prediction with Projection Oracles
• Two-temperature logistic regression based on the Tsallis divergence
• A General Theory for Structured Prediction with Smooth Convex Surrogates

Cited by in corpus (7)

• Learning with Differentiable Perturbed Optimizers
• Fast Differentiable Sorting and Ranking
• The Limited Multi-Label Projection Layer
• A General Theory for Structured Prediction with Smooth Convex Surrogates
• Geometric Losses for Distributional Learning
• Learning Classifiers with Fenchel-Young Losses: Generalized Entropies, Margins, and Algorithms
• Semantic Role Labeling with Iterative Structure Refinement