Transfer Learning with Pre-trained Conditional Generative Models

Abstract

Transfer learning is crucial in training deep neural networks on new target tasks. Current transfer learning methods always assume at least one of (i) source and target task label spaces overlap, (ii) source datasets are available, and (iii) target network architectures are consistent with source ones. However, holding these assumptions is difficult in practical settings because the target task rarely has the same labels as the source task, the source dataset access is restricted due to storage costs and privacy, and the target architecture is often specialized to each task. To transfer source knowledge without these assumptions, we propose a transfer learning method that uses deep generative models and is composed of the following two stages: pseudo pre-training (PP) and pseudo semi-supervised learning (P-SSL). PP trains a target architecture with an artificial dataset synthesized by using conditional source generative models. P-SSL applies SSL algorithms to labeled target data and unlabeled pseudo samples, which are generated by cascading the source classifier and generative models to condition them with target samples. Our experimental results indicate that our method can outperform the baselines of scratch training and knowledge distillation.

Accepted by Machine Learning

References in corpus (22)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• A Simple Framework for Contrastive Learning of Visual Representations
• EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Neural Architecture Search with Reinforcement Learning
• Large Scale GAN Training for High Fidelity Natural Image Synthesis
• Spectral Normalization for Generative Adversarial Networks
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• Unsupervised Data Augmentation for Consistency Training
• Training Generative Adversarial Networks with Limited Data
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning
• Model Adaptation: Unsupervised Domain Adaptation without Source Data
• From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification
• Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation
• Explicit Inductive Bias for Transfer Learning with Convolutional Networks
• DELTA: DEep Learning Transfer using Feature Map with Attention for Convolutional Networks
• On Leveraging Pretrained GANs for Generation with Limited Data
• Consistency Regularization for Generative Adversarial Networks
• Generative Models for Effective ML on Private, Decentralized Datasets
• Rebooting ACGAN: Auxiliary Classifier GANs with Stable Training
• PEARL: Data Synthesis via Private Embeddings and Adversarial Reconstruction Learning