Balancing Average and Worst-case Accuracy in Multitask Learning
arXiv:2110.05838
Abstract
When training and evaluating machine learning models on a large number of tasks, it is important to not only look at average task accuracy -- which may be biased by easy or redundant tasks -- but also worst-case accuracy (i.e. the performance on the task with the lowest accuracy). In this work, we show how to use techniques from the distributionally robust optimization (DRO) literature to improve worst-case performance in multitask learning. We highlight several failure cases of DRO when applied off-the-shelf and present an improved method, Lookahead-DRO (L-DRO), which mitigates these issues. The core idea of L-DRO is to anticipate the interaction between tasks during training in order to choose a dynamic re-weighting of the various task losses, which will (i) lead to minimal worst-case loss and (ii) train on as many tasks as possible. After demonstrating the efficacy of L-DRO on a small controlled synthetic setting, we evaluate it on two realistic benchmarks: a multitask version of the CIFAR-100 image classification dataset and a large-scale multilingual language modeling experiment. Our empirical results show that L-DRO achieves a better trade-off between average and worst-case accuracy with little computational overhead compared to several strong baselines.
arXiv commentUnder review
References in corpus (11)
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- An Overview of Multi-Task Learning in Deep Neural Networks
- DARTS: Differentiable Architecture Search
- Multi-Task Learning with Deep Neural Networks: A Survey
- Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
- The Natural Language Decathlon: Multitask Learning as Question Answering
- Multi-Task Learning as Multi-Objective Optimization
- XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization
- Pareto Multi-Task Learning
- Adaptive Scheduling for Multi-Task Learning
- Efficient Continuous Pareto Exploration in Multi-Task Learning