Self-Supervised Generalisation with Meta Auxiliary Learning

Abstract

Learning with auxiliary tasks can improve the ability of a primary task to generalise. However, this comes at the cost of manually labelling auxiliary data. We propose a new method which automatically learns appropriate labels for an auxiliary task, such that any supervised learning task can be improved without requiring access to any further data. The approach is to train two neural networks: a label-generation network to predict the auxiliary labels, and a multi-task network to train the primary task alongside the auxiliary task. The loss for the label-generation network incorporates the loss of the multi-task network, and so this interaction between the two networks can be seen as a form of meta learning with a double gradient. We show that our proposed method, Meta AuXiliary Learning (MAXL), outperforms single-task learning on 7 image datasets, without requiring any additional data. We also show that MAXL outperforms several other baselines for generating auxiliary labels, and is even competitive when compared with human-defined auxiliary labels. The self-supervised nature of our method leads to a promising new direction towards automated generalisation. Source code can be found at https://github.com/lorenmt/maxl.

arXiv commentPublished at Conference on Neural Information Processing Systems 2019

References in corpus (10)

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Prototypical Networks for Few-shot Learning
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• An Overview of Multi-Task Learning in Deep Neural Networks
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• Auxiliary Tasks in Multi-task Learning
• Adapting Auxiliary Losses Using Gradient Similarity
• Learning to Generalize from Sparse and Underspecified Rewards

Cited by in corpus (16)

• Learning to Generalize from Sparse and Underspecified Rewards
• MetaBalance: Improving Multi-Task Recommendations via Adapting Gradient Magnitudes of Auxiliary Tasks
• Self-supervised Auxiliary Learning with Meta-paths for Heterogeneous Graphs
• Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Tasks
• To Balance or Not to Balance: A Simple-yet-Effective Approach for Learning with Long-Tailed Distributions
• Signal Transformer: Complex-valued Attention and Meta-Learning for Signal Recognition
• Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction
• Adapting Triplet Importance of Implicit Feedback for Personalized Recommendation
• Optimizing Data Usage via Differentiable Rewards
• Auxiliary Learning by Implicit Differentiation
• SALT: Subspace Alignment as an Auxiliary Learning Task for Domain Adaptation
• Improving Generalization in Meta-RL with Imaginary Tasks from Latent Dynamics Mixture
• Semi-supervised Relation Extraction via Incremental Meta Self-Training
• Cross-subject Action Unit Detection with Meta Learning and Transformer-based Relation Modeling
• Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learning
• Meta Auxiliary Learning for Facial Action Unit Detection