Teaching with Commentaries
arXiv:2011.03037
Abstract
Effective training of deep neural networks can be challenging, and there remain many open questions on how to best learn these models. Recently developed methods to improve neural network training examine teaching: providing learned information during the training process to improve downstream model performance. In this paper, we take steps towards extending the scope of teaching. We propose a flexible teaching framework using commentaries, learned meta-information helpful for training on a particular task. We present gradient-based methods to learn commentaries, leveraging recent work on implicit differentiation for scalability. We explore diverse applications of commentaries, from weighting training examples, to parameterising label-dependent data augmentation policies, to representing attention masks that highlight salient image regions. We find that commentaries can improve training speed and/or performance, and provide insights about the dataset and training process. We also observe that commentaries generalise: they can be reused when training new models to obtain performance benefits, suggesting a use-case where commentaries are stored with a dataset and leveraged in future for improved model training.
arXiv commentICLR 2021
References in corpus (12)
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- Understanding deep learning requires rethinking generalization
- Learning to Reweight Examples for Robust Deep Learning
- Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoning
- Generalized Inner Loop Meta-Learning
- Learning to Teach with Dynamic Loss Functions
- Stochastic Hyperparameter Optimization through Hypernetworks
- Learning Data Manipulation for Augmentation and Weighting
- Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions
- Learning to Balance: Bayesian Meta-Learning for Imbalanced and Out-of-distribution Tasks
- Meta Approach to Data Augmentation Optimization
- Auxiliary Learning by Implicit Differentiation