DELTA: DEep Learning Transfer using Feature Map with Attention for Convolutional Networks

Abstract

Transfer learning through fine-tuning a pre-trained neural network with an extremely large dataset, such as ImageNet, can significantly accelerate training while the accuracy is frequently bottlenecked by the limited dataset size of the new target task. To solve the problem, some regularization methods, constraining the outer layer weights of the target network using the starting point as references (SPAR), have been studied. In this paper, we propose a novel regularized transfer learning framework DELTA, namely DEep Learning Transfer using Feature Map with Attention. Instead of constraining the weights of neural network, DELTA aims to preserve the outer layer outputs of the target network. Specifically, in addition to minimizing the empirical loss, DELTA intends to align the outer layer outputs of two networks, through constraining a subset of feature maps that are precisely selected by attention that has been learned in an supervised learning manner. We evaluate DELTA with the state-of-the-art algorithms, including L2 and L2-SP. The experiment results show that our proposed method outperforms these baselines with higher accuracy for new tasks.

arXiv commentAccepted at ICLR 2019

References in corpus (9)

• Distilling the Knowledge in a Neural Network
• Overcoming catastrophic forgetting in neural networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition
• How transferable are features in deep neural networks?
• FitNets: Hints for Thin Deep Nets
• Recurrent Models of Visual Attention
• What makes ImageNet good for transfer learning?
• Parameter Transfer Unit for Deep Neural Networks

Cited by in corpus (17)

• Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning
• Bi-tuning of Pre-trained Representations
• Rethinking Supervised Pre-training for Better Downstream Transferring
• Adaptive Consistency Regularization for Semi-Supervised Transfer Learning
• XMixup: Efficient Transfer Learning with Auxiliary Samples by Cross-domain Mixup
• Non-binary deep transfer learning for image classification
• Skin disease diagnosis with deep learning: a review
• Sample-based Regularization: A Transfer Learning Strategy Toward Better Generalization
• SMILE: Self-Distilled MIxup for Efficient Transfer LEarning
• Representation Transfer by Optimal Transport
• Transfer Learning with Pre-trained Conditional Generative Models
• One to Transfer All: A Universal Transfer Framework for Vision Foundation Model with Few Data
• Renofeation: A Simple Transfer Learning Method for Improved Adversarial Robustness
• StochCA: A Novel Approach for Exploiting Pretrained Models with Cross-Attention
• Self-Tuning for Data-Efficient Deep Learning
• Towards Accurate Knowledge Transfer via Target-awareness Representation Disentanglement
• Cross-Task Representation Learning for Anatomical Landmark Detection