Small Towers Make Big Differences

Abstract

Multi-task learning aims at solving multiple machine learning tasks at the same time. A good solution to a multi-task learning problem should be generalizable in addition to being Pareto optimal. In this paper, we provide some insights on understanding the trade-off between Pareto efficiency and generalization as a result of parameterization in multi-task deep learning models. As a multi-objective optimization problem, enough parameterization is needed for handling task conflicts in a constrained solution space; however, from a multi-task generalization perspective, over-parameterization undermines the benefit of learning a shared representation which helps harder tasks or tasks with limited training examples. A delicate balance between multi-task generalization and multi-objective optimization is therefore needed for finding a better trade-off between efficiency and generalization. To this end, we propose a method of under-parameterized self-auxiliaries for multi-task models to achieve the best of both worlds. It is task-agnostic and works with other multi-task learning algorithms. Empirical results show that small towers of under-parameterized self-auxiliaries can make big differences in improving Pareto efficiency in various multi-task applications.

References in corpus (16)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Distilling the Knowledge in a Neural Network
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
• Unsupervised Domain Adaptation by Backpropagation
• An Overview of Multi-Task Learning in Deep Neural Networks
• Reconciling modern machine learning practice and the bias-variance trade-off
• Understanding deep learning requires rethinking generalization
• GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks
• Multi-Task Learning as Multi-Objective Optimization
• Ask the GRU: Multi-Task Learning for Deep Text Recommendations
• Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
• Large scale distributed neural network training through online distillation
• Gradient Surgery for Multi-Task Learning
• Pareto Multi-Task Learning
• Pseudo-task Augmentation: From Deep Multitask Learning to Intratask Sharing---and Back

Cited by in corpus (4)

• DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning
• Measuring and Harnessing Transference in Multi-Task Learning
• COMET: Learning Cardinality Constrained Mixture of Experts with Trees and Local Search
• Exceeding the Limits of Visual-Linguistic Multi-Task Learning