Tailoring: encoding inductive biases by optimizing unsupervised objectives at prediction time
arXiv:2009.10623
Abstract
From CNNs to attention mechanisms, encoding inductive biases into neural networks has been a fruitful source of improvement in machine learning. Adding auxiliary losses to the main objective function is a general way of encoding biases that can help networks learn better representations. However, since auxiliary losses are minimized only on training data, they suffer from the same generalization gap as regular task losses. Moreover, by adding a term to the loss function, the model optimizes a different objective than the one we care about. In this work we address both problems: first, we take inspiration from \textit{transductive learning} and note that after receiving an input but before making a prediction, we can fine-tune our networks on any unsupervised loss. We call this process {\em tailoring}, because we customize the model to each input to ensure our prediction satisfies the inductive bias. Second, we formulate {\em meta-tailoring}, a nested optimization similar to that in meta-learning, and train our models to perform well on the task objective after adapting them using an unsupervised loss. The advantages of tailoring and meta-tailoring are discussed theoretically and demonstrated empirically on a diverse set of examples.
arXiv commentNeurIPS 2020 workshops on Interpretable Inductive Biases and Meta-learning
References in corpus (18)
- PyTorch: An Imperative Style, High-Performance Deep Learning Library
- A Simple Framework for Contrastive Learning of Visual Representations
- Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
- DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
- Theoretical Models of Learning to Learn
- Momentum Contrast for Unsupervised Visual Representation Learning
- A Learned Representation For Artistic Style
- Certified Adversarial Robustness via Randomized Smoothing
- Learning to Propagate Labels: Transductive Propagation Network for Few-shot Learning
- Hamiltonian Neural Networks
- Few-Shot Learning with Graph Neural Networks
- Interactive Supercomputing on 40,000 Cores for Machine Learning and Data Analysis
- Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers
- Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables
- Generalized Inner Loop Meta-Learning
- MACER: Attack-free and Scalable Robust Training via Maximizing Certified Radius
- Neural Relational Inference with Fast Modular Meta-learning
- Dynamic Scale Inference by Entropy Minimization