Joint Learning of Neural Transfer and Architecture Adaptation for Image Recognition

Abstract

Current state-of-the-art visual recognition systems usually rely on the following pipeline: (a) pretraining a neural network on a large-scale dataset (e.g., ImageNet) and (b) finetuning the network weights on a smaller, task-specific dataset. Such a pipeline assumes the sole weight adaptation is able to transfer the network capability from one domain to another domain, based on a strong assumption that a fixed architecture is appropriate for all domains. However, each domain with a distinct recognition target may need different levels/paths of feature hierarchy, where some neurons may become redundant, and some others are re-activated to form new network structures. In this work, we prove that dynamically adapting network architectures tailored for each domain task along with weight finetuning benefits in both efficiency and effectiveness, compared to the existing image recognition pipeline that only tunes the weights regardless of the architecture. Our method can be easily generalized to an unsupervised paradigm by replacing supernet training with self-supervised learning in the source domain tasks and performing linear evaluation in the downstream tasks. This further improves the search efficiency of our method. Moreover, we also provide principled and empirical analysis to explain why our approach works by investigating the ineffectiveness of existing neural architecture search. We find that preserving the joint distribution of the network architecture and weights is of importance. This analysis not only benefits image recognition but also provides insights for crafting neural networks. Experiments on five representative image recognition tasks such as person re-identification, age estimation, gender recognition, image classification, and unsupervised domain adaptation demonstrate the effectiveness of our method.

arXiv commentTo appear in IEEE Transactions ON Neural Networks and Learning Systems. We prove that dynamically adapting network architectures tailored for each domain task along with weight finetuning benefits in both efficiency and effectiveness, compared to the existing image recognition pipeline that only tunes the weights regardless of the architecture

References in corpus (16)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• A Simple Framework for Contrastive Learning of Visual Representations
• Bootstrap your own latent: A new approach to self-supervised Learning
• In Defense of the Triplet Loss for Person Re-Identification
• Improved Baselines with Momentum Contrastive Learning
• DARTS: Differentiable Architecture Search
• Designing Neural Network Architectures using Reinforcement Learning
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• Single Path One-Shot Neural Architecture Search with Uniform Sampling
• Progressive Neural Architecture Search
• NAS evaluation is frustratingly hard
• One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers
• FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search
• Moment Matching for Multi-Source Domain Adaptation
• Evolutionary-Neural Hybrid Agents for Architecture Search