What and Where: Learn to Plug Adapters via NAS for Multi-Domain Learning
arXiv:2007.12415
Abstract
As an important and challenging problem, multi-domain learning (MDL) typically seeks for a set of effective lightweight domain-specific adapter modules plugged into a common domain-agnostic network. Usually, existing ways of adapter plugging and structure design are handcrafted and fixed for all domains before model learning, resulting in the learning inflexibility and computational intensiveness. With this motivation, we propose to learn a data-driven adapter plugging strategy with Neural Architecture Search (NAS), which automatically determines where to plug for those adapter modules. Furthermore, we propose a NAS-adapter module for adapter structure design in a NAS-driven learning scheme, which automatically discovers effective adapter module structures for different domains. Experimental results demonstrate the effectiveness of our MDL model against existing approaches under the conditions of comparable performance.
References in corpus (13)
- Overcoming catastrophic forgetting in neural networks
- Neural Architecture Search with Reinforcement Learning
- Neural Architecture Search: A Survey
- DARTS: Differentiable Architecture Search
- Learning multiple visual domains with residual adapters
- Large-Scale Evolution of Image Classifiers
- Designing Neural Network Architectures using Reinforcement Learning
- PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
- Universal representations:The missing link between faces, text, planktons, and cat breeds
- A comprehensive, application-oriented study of catastrophic forgetting in DNNs
- Budget-Aware Adapters for Multi-Domain Learning
- RC-DARTS: Resource Constrained Differentiable Architecture Search
- Feature Partitioning for Efficient Multi-Task Architectures