The Benefit of Multitask Representation Learning

Abstract

We discuss a general method to learn data representations from multiple tasks. We provide a justification for this method in both settings of multitask learning and learning-to-learn. The method is illustrated in detail in the special case of linear feature learning. Conditions on the theoretical advantage offered by multitask representation learning over independent task learning are established. In particular, focusing on the important example of half-space learning, we derive the regime in which multitask representation learning is beneficial over independent task learning, as a function of the sample size, the number of tasks and the intrinsic data dimensionality. Other potential applications of our results include multitask feature learning in reproducing kernel Hilbert spaces and multilayer, deep networks.

To appear in Journal of Machine Learning Research (JMLR). 31 pages

References in corpus (3)

• Theoretical Models of Learning to Learn
• A PAC-Bayesian bound for Lifelong Learning
• An Inequality with Applications to Structured Sparsity and Multitask Dictionary Learning

Cited by in corpus (58)

• Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges
• Physics-Guided Deep Neural Networks for Power Flow Analysis
• Domain Generalization by Marginal Transfer Learning
• Exploiting Shared Representations for Personalized Federated Learning
• On the Theory of Transfer Learning: The Importance of Task Diversity
• Sharing Knowledge in Multi-Task Deep Reinforcement Learning
• Few-Shot Learning via Learning the Representation, Provably
• Provable Meta-Learning of Linear Representations
• FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPs
• Machine learning for complete intersection Calabi-Yau manifolds: a methodological study
• CASTLE: Regularization via Auxiliary Causal Graph Discovery
• Variable Selection and Task Grouping for Multi-Task Learning
• Multi-Task Reinforcement Learning with Context-based Representations
• Meta-Adaptive Nonlinear Control: Theory and Algorithms
• MKD: a Multi-Task Knowledge Distillation Approach for Pretrained Language Models
• How to Train Your MAML to Excel in Few-Shot Classification
• Revisiting Meta-Learning as Supervised Learning
• Optimization and Generalization of Regularization-Based Continual Learning: a Loss Approximation Viewpoint
• Generalization Bounds For Meta-Learning: An Information-Theoretic Analysis
• Meta-Learning with Graph Neural Networks: Methods and Applications
• How Important is the Train-Validation Split in Meta-Learning?
• Minimax Estimation for Personalized Federated Learning: An Alternative between FedAvg and Local Training?
• On Inductive Biases for Heterogeneous Treatment Effect Estimation
• An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models
• Bandit Algorithms for Precision Medicine
• Lifelong Bayesian Optimization
• A Sample Complexity Separation between Non-Convex and Convex Meta-Learning
• Weighted Training for Cross-Task Learning
• Meta-learning with Stochastic Linear Bandits
• A Scaling Law for Synthetic-to-Real Transfer: How Much Is Your Pre-training Effective?
• On the Power of Multitask Representation Learning in Linear MDP
• Learning Robust State Abstractions for Hidden-Parameter Block MDPs
• Meta-Learning Dynamics Forecasting Using Task Inference
• How Fine-Tuning Allows for Effective Meta-Learning
• Learning Functions to Study the Benefit of Multitask Learning
• Impact of Representation Learning in Linear Bandits
• Provable Representation Learning for Imitation Learning via Bi-level Optimization
• Target-Embedding Autoencoders for Supervised Representation Learning
• On Localized Discrepancy for Domain Adaptation
• Adversarial Training Helps Transfer Learning via Better Representations
• Sequoia: A Software Framework to Unify Continual Learning Research
• Global Convergence and Generalization Bound of Gradient-Based Meta-Learning with Deep Neural Nets
• Conditional Meta-Learning of Linear Representations
• Multi-task Learning by Leveraging the Semantic Information
• Decision Making Problems with Funnel Structure: A Multi-Task Learning Approach with Application to Email Marketing Campaigns
• Representation Learning Beyond Linear Prediction Functions
• On the Statistical Benefits of Curriculum Learning
• Provable Adaptation across Multiway Domains via Representation Learning
• For Manifold Learning, Deep Neural Networks can be Locality Sensitive Hash Functions
• A Multi-task Learning Approach for Named Entity Recognition using Local Detection
• A Distribution-Dependent Analysis of Meta-Learning
• Estimation of Models with Limited Data by Leveraging Shared Structure
• Learning-to-learn non-convex piecewise-Lipschitz functions
• HydaLearn: Highly Dynamic Task Weighting for Multi-task Learning with Auxiliary Tasks
• Eden: A Unified Environment Framework for Booming Reinforcement Learning Algorithms
• Learning Mixtures of Low-Rank Models
• Linear Speedup in Personalized Collaborative Learning
• Margin-Based Transfer Bounds for Meta Learning with Deep Feature Embedding