SpotTune: Transfer Learning through Adaptive Fine-tuning

Abstract

Transfer learning, which allows a source task to affect the inductive bias of the target task, is widely used in computer vision. The typical way of conducting transfer learning with deep neural networks is to fine-tune a model pre-trained on the source task using data from the target task. In this paper, we propose an adaptive fine-tuning approach, called SpotTune, which finds the optimal fine-tuning strategy per instance for the target data. In SpotTune, given an image from the target task, a policy network is used to make routing decisions on whether to pass the image through the fine-tuned layers or the pre-trained layers. We conduct extensive experiments to demonstrate the effectiveness of the proposed approach. Our method outperforms the traditional fine-tuning approach on 12 out of 14 standard datasets.We also compare SpotTune with other state-of-the-art fine-tuning strategies, showing superior performance. On the Visual Decathlon datasets, our method achieves the highest score across the board without bells and whistles.

References in corpus (19)

• Deep Residual Learning for Image Recognition
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• Categorical Reparameterization with Gumbel-Softmax
• Convolutional Neural Networks for Medical Image Analysis: Full Training or Fine Tuning?
• Learning Transferable Features with Deep Adaptation Networks
• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
• CNN Features off-the-shelf: an Astounding Baseline for Recognition
• Residual Networks Behave Like Ensembles of Relatively Shallow Networks
• Learning multiple visual domains with residual adapters
• Rich feature hierarchies for accurate object detection and semantic segmentation
• Domain Adaptation for Visual Applications: A Comprehensive Survey
• Conditional Computation in Neural Networks for faster models
• Universal representations:The missing link between faces, text, planktons, and cat breeds
• Return of Frustratingly Easy Domain Adaptation
• Explicit Inductive Bias for Transfer Learning with Convolutional Networks
• Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Feature
• Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights
• Spatially Adaptive Computation Time for Residual Networks

Cited by in corpus (2)

• Transfer learning for galaxy morphology from one survey to another
• Flexible Multi-task Networks by Learning Parameter Allocation