Scalable Unidirectional Pareto Optimality for Multi-Task Learning with Constraints
arXiv:2110.15442
Abstract
Multi-objective optimization (MOO) problems require balancing competing objectives, often under constraints. The Pareto optimal solution set defines all possible optimal trade-offs over such objectives. In this work, we present a novel method for Pareto-front learning: inducing the full Pareto manifold at train-time so users can pick any desired optimal trade-off point at run-time. Our key insight is to exploit Fritz-John Conditions for a novel guided double gradient descent strategy. Evaluation on synthetic benchmark problems allows us to vary MOO problem difficulty in controlled fashion and measure accuracy vs. known analytic solutions. We further test scalability and generalization in learning optimal neural model parameterizations for Multi-Task Learning (MTL) on image classification. Results show consistent improvement in accuracy and efficiency over prior MTL methods as well as techniques from operations research.
References in corpus (13)
- Adam: A Method for Stochastic Optimization
- Fairness in Machine Learning: A Survey
- Multi-Task Learning with Deep Neural Networks: A Survey
- Multi-Task Learning as Multi-Objective Optimization
- The Expressive Power of Neural Networks: A View from the Width
- PU Learning for Matrix Completion
- Depth-Width Tradeoffs in Approximating Natural Functions with Neural Networks
- Minimax Pareto Fairness: A Multi Objective Perspective
- Pareto Multi-Task Learning
- Efficient Continuous Pareto Exploration in Multi-Task Learning
- Multi-Gradient Descent for Multi-Objective Recommender Systems
- The Fairness-Accuracy Pareto Front
- A Hybrid 2-stage Neural Optimization for Pareto Front Extraction