Multi-path Neural Networks for On-device Multi-domain Visual Classification

Abstract

Learning multiple domains/tasks with a single model is important for improving data efficiency and lowering inference cost for numerous vision tasks, especially on resource-constrained mobile devices. However, hand-crafting a multi-domain/task model can be both tedious and challenging. This paper proposes a novel approach to automatically learn a multi-path network for multi-domain visual classification on mobile devices. The proposed multi-path network is learned from neural architecture search by applying one reinforcement learning controller for each domain to select the best path in the super-network created from a MobileNetV3-like search space. An adaptive balanced domain prioritization algorithm is proposed to balance optimizing the joint model on multiple domains simultaneously. The determined multi-path model selectively shares parameters across domains in shared nodes while keeping domain-specific parameters within non-shared nodes in individual domain paths. This approach effectively reduces the total number of parameters and FLOPS, encouraging positive knowledge transfer while mitigating negative interference across domains. Extensive evaluations on the Visual Decathlon dataset demonstrate that the proposed multi-path model achieves state-of-the-art performance in terms of accuracy, model size, and FLOPS against other approaches using MobileNetV3-like architectures. Furthermore, the proposed method improves average accuracy over learning single-domain models individually, and reduces the total number of parameters and FLOPS by 78% and 32% respectively, compared to the approach that simply bundles single-domain models for multi-domain learning.

WACV 2021

References in corpus (12)

• Multi-Task Learning for Dense Prediction Tasks: A Survey
• Efficient Neural Architecture Search via Parameter Sharing
• Learning multiple visual domains with residual adapters
• Multi-Task Learning as Multi-Objective Optimization
• SNAS: Stochastic Neural Architecture Search
• AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning
• Universal representations:The missing link between faces, text, planktons, and cat breeds
• Learning to Branch for Multi-Task Learning
• Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learning
• Budget-Aware Adapters for Multi-Domain Learning
• Feature Partitioning for Efficient Multi-Task Architectures
• A Simple General Approach to Balance Task Difficulty in Multi-Task Learning