On Tilted Losses in Machine Learning: Theory and Applications

Abstract

Exponential tilting is a technique commonly used in fields such as statistics, probability, information theory, and optimization to create parametric distribution shifts. Despite its prevalence in related fields, tilting has not seen widespread use in machine learning. In this work, we aim to bridge this gap by exploring the use of tilting in risk minimization. We study a simple extension to ERM -- tilted empirical risk minimization (TERM) -- which uses exponential tilting to flexibly tune the impact of individual losses. The resulting framework has several useful properties: We show that TERM can increase or decrease the influence of outliers, respectively, to enable fairness or robustness; has variance-reduction properties that can benefit generalization; and can be viewed as a smooth approximation to the tail probability of losses. Our work makes rigorous connections between TERM and related objectives, such as Value-at-Risk, Conditional Value-at-Risk, and distributionally robust optimization (DRO). We develop batch and stochastic first-order optimization methods for solving TERM, provide convergence guarantees for the solvers, and show that the framework can be efficiently solved relative to common alternatives. Finally, we demonstrate that TERM can be used for a multitude of applications in machine learning, such as enforcing fairness between subgroups, mitigating the effect of outliers, and handling class imbalance. Despite the straightforward modification TERM makes to traditional ERM objectives, we find that the framework can consistently outperform ERM and deliver competitive performance with state-of-the-art, problem-specific approaches.

arXiv commentarXiv admin note: substantial text overlap with arXiv:2007.01162

References in corpus (26)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Communication-Efficient Learning of Deep Networks from Decentralized Data
• Equality of Opportunity in Supervised Learning
• Learning to Reweight Examples for Robust Deep Learning
• Agnostic Federated Learning
• Empirical Bernstein Bounds and Sample Variance Penalization
• Fairness Without Demographics in Repeated Loss Minimization
• Sever: A Robust Meta-Algorithm for Stochastic Optimization
• Solving a Class of Non-Convex Min-Max Games Using Iterative First Order Methods
• On the Dual Formulation of Boosting Algorithms
• Robust Regression via Hard Thresholding
• The Price of Fair PCA: One Extra Dimension
• What is Local Optimality in Nonconvex-Nonconcave Minimax Optimization?
• Learning with Average Top-k Loss
• Biased Importance Sampling for Deep Neural Network Training
• Fairness for Robust Log Loss Classification
• Accelerating Deep Learning by Focusing on the Biggest Losers
• PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review
• A Characterization of Guesswork on Swiftly Tilting Curves
• SimNets: A Generalization of Convolutional Networks
• Multi-Criteria Dimensionality Reduction with Applications to Fairness
• Active Sampler: Light-weight Accelerator for Complex Data Analytics at Scale
• Distributionally Robust Learning
• Adaptive Normalized Risk-Averting Training For Deep Neural Networks
• Mismatched Guesswork
• Output Perturbation for Differentially Private Convex Optimization: Faster and More General

Cited by in corpus (2)

• Data Heterogeneity Modeling for Trustworthy Machine Learning
• A Survey of Learning Criteria Going Beyond the Usual Risk