Learning multiple visual domains with residual adapters

Abstract

There is a growing interest in learning data representations that work well for many different types of problems and data. In this paper, we look in particular at the task of learning a single visual representation that can be successfully utilized in the analysis of very different types of images, from dog breeds to stop signs and digits. Inspired by recent work on learning networks that predict the parameters of another, we develop a tunable deep network architecture that, by means of adapter residual modules, can be steered on the fly to diverse visual domains. Our method achieves a high degree of parameter sharing while maintaining or even improving the accuracy of domain-specific representations. We also introduce the Visual Decathlon Challenge, a benchmark that evaluates the ability of representations to capture simultaneously ten very different visual domains and measures their ability to recognize well uniformly.

Cited by in corpus (78)

• LoRA: Low-Rank Adaptation of Large Language Models
• Deep Visual Domain Adaptation: A Survey
• A Survey of Deep Learning-based Object Detection
• Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
• Multi-Task Learning for Dense Prediction Tasks: A Survey
• On the Cross-lingual Transferability of Monolingual Representations
• MS-Net: Multi-Site Network for Improving Prostate Segmentation with Heterogeneous MRI Data
• Deep Learning for Generic Object Detection: A Survey
• Contrastive Cross-site Learning with Redesigned Net for COVID-19 CT Classification
• A Comprehensive Study of Class Incremental Learning Algorithms for Visual Tasks
• BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning
• Improving robustness against common corruptions by covariate shift adaptation
• Meta-learning with differentiable closed-form solvers
• K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters
• Learning to Generalize: Meta-Learning for Domain Generalization
• Pretrained Transformers as Universal Computation Engines
• Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs
• Meta-Learning with Warped Gradient Descent
• CrossTransformers: spatially-aware few-shot transfer
• Contrastive Learning with Adversarial Examples
• Few-Shot Adaptation of Generative Adversarial Networks
• Deep Cocktail Network: Multi-source Unsupervised Domain Adaptation with Category Shift
• A Constructive Prediction of the Generalization Error Across Scales
• SpotTune: Transfer Learning through Adaptive Fine-tuning
• Incorporating BERT into Parallel Sequence Decoding with Adapters
• Towards continual learning in medical imaging
• FixyNN: Efficient Hardware for Mobile Computer Vision via Transfer Learning
• Budget-Aware Adapters for Multi-Domain Learning
• Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights
• SAFENet: Self-Supervised Monocular Depth Estimation with Semantic-Aware Feature Extraction
• Multi-Task Zipping via Layer-wise Neuron Sharing
• Self-Supervised Visual Learning by Variable Playback Speeds Prediction of a Video
• Episodic Training for Domain Generalization
• Pseudo-task Augmentation: From Deep Multitask Learning to Intratask Sharing---and Back
• End-to-End Multi-Task Learning with Attention
• FabGPT: An Efficient Large Multimodal Model for Complex Wafer Defect Knowledge Queries
• Robust Policies via Mid-Level Visual Representations: An Experimental Study in Manipulation and Navigation
• Multitask Learning for Scalable and Dense Multilayer Bayesian Map Inference
• Simple, Scalable Adaptation for Neural Machine Translation
• How Will It Drape Like? Capturing Fabric Mechanics from Depth Images
• Feature Partitioning for Efficient Multi-Task Architectures
• Emergent Communication Pretraining for Few-Shot Machine Translation
• Adapting by Pruning: A Case Study on BERT
• Model-agnostic Multi-Domain Learning with Domain-Specific Adapters for Action Recognition
• Cross-domain Few-shot Learning with Task-specific Adapters
• Parameter-Efficient Transfer from Sequential Behaviors for User Modeling and Recommendation
• 3D U${}^2$-Net: A 3D Universal U-Net for Multi-Domain Medical Image Segmentation
• Learning to Impute: A General Framework for Semi-supervised Learning
• Boosting Binary Masks for Multi-Domain Learning through Affine Transformations
• Regularizing by the Variance of the Activations' Sample-Variances
• Evolutionary Architecture Search For Deep Multitask Networks
• Disjoint Label Space Transfer Learning with Common Factorised Space
• Evaluating Online Continual Learning with CALM
• On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment
• Context-Aware Multipath Networks
• A Lifelong Learning Approach to Brain MR Segmentation Across Scanners and Protocols
• Latent Domain Learning with Dynamic Residual Adapters
• Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning
• Large-Scale Multilingual Speech Recognition with a Streaming End-to-End Model
• Learning Language Specific Sub-network for Multilingual Machine Translation
• Modular Universal Reparameterization: Deep Multi-task Learning Across Diverse Domains
• Training Neural Networks with Fixed Sparse Masks
• Learning Finer-class Networks for Universal Representations
• Optimizing Reusable Knowledge for Continual Learning via Metalearning
• Consistent Classification with Generalized Metrics
• DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models
• The Traveling Observer Model: Multi-task Learning Through Spatial Variable Embeddings
• PolyViT: Co-training Vision Transformers on Images, Videos and Audio
• Domain-Agnostic Few-Shot Classification by Learning Disparate Modulators
• Multi-Task, Multi-Domain Deep Segmentation with Shared Representations and Contrastive Regularization for Sparse Pediatric Datasets
• Introducing Pose Consistency and Warp-Alignment for Self-Supervised 6D Object Pose Estimation in Color Images
• Multi-path Neural Networks for On-device Multi-domain Visual Classification
• A Procedural World Generation Framework for Systematic Evaluation of Continual Learning
• AlphaNet: Improving Long-Tail Classification By Combining Classifiers
• What and Where: Learn to Plug Adapters via NAS for Multi-Domain Learning
• Unlocking the Full Potential of Small Data with Diverse Supervision
• Counter-Interference Adapter for Multilingual Machine Translation
• Disentangling Transfer and Interference in Multi-Domain Learning