Tilted Empirical Risk Minimization

Abstract

Empirical risk minimization (ERM) is typically designed to perform well on the average loss, which can result in estimators that are sensitive to outliers, generalize poorly, or treat subgroups unfairly. While many methods aim to address these problems individually, in this work, we explore them through a unified framework -- tilted empirical risk minimization (TERM). In particular, we show that it is possible to flexibly tune the impact of individual losses through a straightforward extension to ERM using a hyperparameter called the tilt. We provide several interpretations of the resulting framework: We show that TERM can increase or decrease the influence of outliers, respectively, to enable fairness or robustness; has variance-reduction properties that can benefit generalization; and can be viewed as a smooth approximation to a superquantile method. We develop batch and stochastic first-order optimization methods for solving TERM, and show that the problem can be efficiently solved relative to common alternatives. Finally, we demonstrate that TERM can be used for a multitude of applications, such as enforcing fairness between subgroups, mitigating the effect of outliers, and handling class imbalance. TERM is not only competitive with existing solutions tailored to these individual problems, but can also enable entirely new applications, such as simultaneously addressing outliers and promoting fairness.

Accepted by ICLR 2021

References in corpus (17)

• Equality of Opportunity in Supervised Learning
• Learning to Reweight Examples for Robust Deep Learning
• Agnostic Federated Learning
• Empirical Bernstein Bounds and Sample Variance Penalization
• Fairness Without Demographics in Repeated Loss Minimization
• Sever: A Robust Meta-Algorithm for Stochastic Optimization
• Solving a Class of Non-Convex Min-Max Games Using Iterative First Order Methods
• Learning From Noisy Singly-labeled Data
• Robust Regression via Hard Thresholding
• The Price of Fair PCA: One Extra Dimension
• Biased Importance Sampling for Deep Neural Network Training
• Fairness for Robust Log Loss Classification
• Accelerating Deep Learning by Focusing on the Biggest Losers
• PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review
• A Characterization of Guesswork on Swiftly Tilting Curves
• SimNets: A Generalization of Convolutional Networks
• Globally-convergent Iteratively Reweighted Least Squares for Robust Regression Problems

Cited by in corpus (1)

• Output Perturbation for Differentially Private Convex Optimization: Faster and More General