Deep Multimodal Neural Architecture Search
arXiv:2004.12070
Abstract
Designing effective neural networks is fundamentally important in deep multimodal learning. Most existing works focus on a single task and design neural architectures manually, which are highly task-specific and hard to generalize to different tasks. In this paper, we devise a generalized deep multimodal neural architecture search (MMnas) framework for various multimodal learning tasks. Given multimodal input, we first define a set of primitive operations, and then construct a deep encoder-decoder based unified backbone, where each encoder or decoder block corresponds to an operation searched from a predefined operation pool. On top of the unified backbone, we attach task-specific heads to tackle different multimodal learning tasks. By using a gradient-based NAS algorithm, the optimal architectures for different tasks are learned efficiently. Extensive ablation studies, comprehensive analysis, and comparative experimental results show that the obtained MMnasNet significantly outperforms existing state-of-the-art approaches across three multimodal learning tasks (over five datasets), including visual question answering, image-text matching, and visual grounding.
arXiv commentAccept to ACM MM2020, code available at https://github.com/MILVLG/mmnas/
References in corpus (15)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Neural Architecture Search with Reinforcement Learning
- Microsoft COCO Captions: Data Collection and Evaluation Server
- VisualBERT: A Simple and Performant Baseline for Vision and Language
- VQA: Visual Question Answering
- Efficient Neural Architecture Search via Parameter Sharing
- VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
- Beyond Bilinear: Generalized Multimodal Factorized High-order Pooling for Visual Question Answering
- PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search
- UNITER: UNiversal Image-TExt Representation Learning
- Hadamard Product for Low-rank Bilinear Pooling
- Multimodal Unified Attention Networks for Vision-and-Language Interactions
- Dynamic Graph Attention for Referring Expression Comprehension