Learning Deep Multimodal Feature Representation with Asymmetric Multi-layer Fusion

Abstract

We propose a compact and effective framework to fuse multimodal features at multiple layers in a single network. The framework consists of two innovative fusion schemes. Firstly, unlike existing multimodal methods that necessitate individual encoders for different modalities, we verify that multimodal features can be learnt within a shared single network by merely maintaining modality-specific batch normalization layers in the encoder, which also enables implicit fusion via joint feature representation learning. Secondly, we propose a bidirectional multi-layer fusion scheme, where multimodal features can be exploited progressively. To take advantage of such scheme, we introduce two asymmetric fusion operations including channel shuffle and pixel shift, which learn different fused features with respect to different fusion directions. These two operations are parameter-free and strengthen the multimodal feature interactions across channels as well as enhance the spatial feature discrimination within channels. We conduct extensive experiments on semantic segmentation and image translation tasks, based on three publicly available datasets covering diverse modalities. Results indicate that our proposed framework is general, compact and is superior to state-of-the-art fusion frameworks.

arXiv commentACMMM 2020 (2020.3)

References in corpus (10)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
• The Cityscapes Dataset for Semantic Urban Scene Understanding
• Revisiting Batch Normalization For Practical Domain Adaptation
• Searching for Efficient Multi-Scale Architectures for Dense Image Prediction
• K for the Price of 1: Parameter-efficient Multi-task and Transfer Learning
• Learning Common and Specific Features for RGB-D Semantic Segmentation with Deconvolutional Networks
• All You Need is a Few Shifts: Designing Efficient Convolutional Neural Networks for Image Classification
• Learning Efficient Video Representation with Video Shuffle Networks