AdaTask: A Task-aware Adaptive Learning Rate Approach to Multi-task Learning

Abstract

Multi-task learning (MTL) models have demonstrated impressive results in computer vision, natural language processing, and recommender systems. Even though many approaches have been proposed, how well these approaches balance different tasks on each parameter still remains unclear. In this paper, we propose to measure the task dominance degree of a parameter by the total updates of each task on this parameter. Specifically, we compute the total updates by the exponentially decaying Average of the squared Updates (AU) on a parameter from the corresponding task.Based on this novel metric, we observe that many parameters in existing MTL methods, especially those in the higher shared layers, are still dominated by one or several tasks. The dominance of AU is mainly due to the dominance of accumulative gradients from one or several tasks. Motivated by this, we propose a Task-wise Adaptive learning rate approach, AdaTask in short, to separate the \emph{accumulative gradients} and hence the learning rate of each task for each parameter in adaptive learning rate approaches (e.g., AdaGrad, RMSProp, and Adam). Comprehensive experiments on computer vision and recommender system MTL datasets demonstrate that AdaTask significantly improves the performance of dominated tasks, resulting SOTA average task-wise performance. Analysis on both synthetic and real-world datasets shows AdaTask balance parameters in every shared layer well.

arXiv comment14 pages, 8 figures

References in corpus (19)

• Adam: A Method for Stochastic Optimization
• ADADELTA: An Adaptive Learning Rate Method
• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
• On the Convergence of Adam and Beyond
• Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
• GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks
• Multi-Task Learning as Multi-Objective Optimization
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning
• AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients
• Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout
• Entire Space Multi-Task Model: An Effective Approach for Estimating Post-Click Conversion Rate
• MetaBalance: Improving Multi-Task Recommendations via Adapting Gradient Magnitudes of Auxiliary Tasks
• Predicting Different Types of Conversions with Multi-Task Learning in Online Advertising
• Conflict-Averse Gradient Descent for Multi-task Learning
• A Sufficient Condition for Convergences of Adam and RMSProp
• Towards Practical Adam: Non-Convexity, Convergence Theory, and Mini-Batch Acceleration
• Efficient-Adam: Communication-Efficient Distributed Adam
• Adaptive Activation Network and Functional Regularization for Efficient and Flexible Deep Multi-Task Learning

Cited by in corpus (6)

• M3oE: Multi-Domain Multi-Task Mixture-of Experts Recommendation Framework
• Ads Recommendation in a Collapsed and Entangled World
• Unlocking the Hidden Treasures: Enhancing Recommendations with Unlabeled Data
• Continual Panoptic Perception: Towards Multi-modal Incremental Interpretation of Remote Sensing Images
• Unconstrained Monotonic Calibration of Predictions in Deep Ranking Systems
• Learning Task-preferred Inference Routes for Gradient De-conflict in Multi-output DNNs