GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks

Abstract

Deep multitask networks, in which one neural network produces multiple predictive outputs, can offer better speed and performance than their single-task counterparts but are challenging to train properly. We present a gradient normalization (GradNorm) algorithm that automatically balances training in deep multitask models by dynamically tuning gradient magnitudes. We show that for various network architectures, for both regression and classification tasks, and on both synthetic and real datasets, GradNorm improves accuracy and reduces overfitting across multiple tasks when compared to single-task networks, static baselines, and other adaptive multitask loss balancing techniques. GradNorm also matches or surpasses the performance of exhaustive grid search methods, despite only involving a single asymmetry hyperparameter $\alpha$. Thus, what was once a tedious search process that incurred exponentially more compute for each task added can now be accomplished within a few training runs, irrespective of the number of tasks. Ultimately, we will demonstrate that gradient manipulation affords us great control over the training dynamics of multitask networks and may be one of the keys to unlocking the potential of multitask learning.

arXiv commentICML 2018

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
• YOLO9000: Better, Faster, Stronger
• Clustered Multi-Task Learning: A Convex Formulation
• Universal representations:The missing link between faces, text, planktons, and cat breeds
• A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks
• Automated Curriculum Learning for Neural Networks
• Fully-adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classification

Cited by in corpus (57)

• FairMOT: On the Fairness of Detection and Re-Identification in Multiple Object Tracking
• How to Train Your Robot with Deep Reinforcement Learning; Lessons We've Learned
• Understanding and mitigating gradient pathologies in physics-informed neural networks
• Multi-Objective Loss Balancing for Physics-Informed Deep Learning
• Physics-informed neural network simulation of multiphase poroelasticity using stress-split sequential training
• Dynamic Feature Integration for Simultaneous Detection of Salient Object, Edge and Skeleton
• Estimating Depth from RGB and Sparse Sensing
• OmiEmbed: a unified multi-task deep learning framework for multi-omics data
• Review: Deep Learning in Electron Microscopy
• How to Fine-Tune BERT for Text Classification?
• Faster Training of Mask R-CNN by Focusing on Instance Boundaries
• Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
• SoftAdapt: Techniques for Adaptive Loss Weighting of Neural Networks with Multi-Part Loss Functions
• Gradient Harmonized Single-stage Detector
• Multi-Task Learning of Height and Semantics from Aerial Images
• Multi-Task Reinforcement Learning with Soft Modularization
• AdaTask: A Task-aware Adaptive Learning Rate Approach to Multi-task Learning
• Real-time Neural Network Inference on Extremely Weak Devices: Agile Offloading with Explainable AI
• Avoiding Catastrophe: Active Dendrites Enable Multi-Task Learning in Dynamic Environments
• Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety
• Efficient Multi-Task RGB-D Scene Analysis for Indoor Environments
• MetaBalance: Improving Multi-Task Recommendations via Adapting Gradient Magnitudes of Auxiliary Tasks
• Global-Reasoned Multi-Task Learning Model for Surgical Scene Understanding
• Detecting 32 Pedestrian Attributes for Autonomous Vehicles
• Adaptive Scheduling for Multi-Task Learning
• Regularizing Deep Multi-Task Networks using Orthogonal Gradients
• Adversarial Multi-task Learning Enhanced Physics-informed Neural Networks for Solving Partial Differential Equations
• Heterogeneous Multi-task Learning with Expert Diversity
• AuxNet: Auxiliary tasks enhanced Semantic Segmentation for Automated Driving
• Modeling Task Relationships in Multi-variate Soft Sensor with Balanced Mixture-of-Experts
• Physics-Guided Neural Networks for Intraventricular Vector Flow Mapping
• Dynamic Multi-Task Learning for Face Recognition with Facial Expression
• BHIN2vec: Balancing the Type of Relation in Heterogeneous Information Network
• Optimizing Airbnb Search Journey with Multi-task Learning
• Adversarial Attacks on Monocular Depth Estimation
• Auxiliary Task Reweighting for Minimum-data Learning
• MaskFace: multi-task face and landmark detector
• Multitask Learning Strengthens Adversarial Robustness
• FisheyeMultiNet: Real-time Multi-task Learning Architecture for Surround-view Automated Parking System
• Measuring and Harnessing Transference in Multi-Task Learning
• Variational PINNs with tree-based integration and boundary element data in the modeling of multi-phase architected materials
• JEPOO: Highly Accurate Joint Estimation of Pitch, Onset and Offset for Music Information Retrieval
• Small Towers Make Big Differences
• GDOD: Effective Gradient Descent using Orthogonal Decomposition for Multi-Task Learning
• Adaptive Precision Training (AdaPT): A dynamic fixed point quantized training approach for DNNs
• Scalable Multitask Learning Using Gradient-based Estimation of Task Affinity
• Knowledge Distillation for Multi-task Learning
• Boosting Supervision with Self-Supervision for Few-shot Learning
• Latent Domain Learning with Dynamic Residual Adapters
• Universal Embeddings for Spatio-Temporal Tagging of Self-Driving Logs
• PanRep: Graph neural networks for extracting universal node embeddings in heterogeneous graphs
• HydaLearn: Highly Dynamic Task Weighting for Multi-task Learning with Auxiliary Tasks
• Learning a Unified Embedding for Visual Search at Pinterest
• Interleaved Multitask Learning with Energy Modulated Learning Progress
• Ada-Segment: Automated Multi-loss Adaptation for Panoptic Segmentation
• Efficient 2.5D Hand Pose Estimation via Auxiliary Multi-Task Training for Embedded Devices
• Cross-modal Multi-task Learning for Graphic Recognition of Caricature Face