Deep Multimodal Fusion by Channel Exchanging

Abstract

Deep multimodal fusion by using multiple sources of data for classification or regression has exhibited a clear advantage over the unimodal counterpart on various applications. Yet, current methods including aggregation-based and alignment-based fusion are still inadequate in balancing the trade-off between inter-modal fusion and intra-modal processing, incurring a bottleneck of performance improvement. To this end, this paper proposes Channel-Exchanging-Network (CEN), a parameter-free multimodal fusion framework that dynamically exchanges channels between sub-networks of different modalities. Specifically, the channel exchanging process is self-guided by individual channel importance that is measured by the magnitude of Batch-Normalization (BN) scaling factor during training. The validity of such exchanging process is also guaranteed by sharing convolutional filters yet keeping separate BN layers across modalities, which, as an add-on benefit, allows our multimodal architecture to be almost as compact as a unimodal network. Extensive experiments on semantic segmentation via RGB-D data and image translation through multi-domain input verify the effectiveness of our CEN compared to current state-of-the-art methods. Detailed ablation studies have also been carried out, which provably affirm the advantage of each component we propose. Our code is available at https://github.com/yikaiw/CEN.

NeurIPS 2020. Code and models: https://github.com/yikaiw/CEN

Cited by in corpus (15)

• Changer: Feature Interaction is What You Need for Change Detection
• What Makes Multi-modal Learning Better than Single (Provably)
• Dynamic Neural Networks: A Survey
• Common Practices and Taxonomy in Deep Multi-view Fusion for Remote Sensing Applications
• SRC-Net: Bi-Temporal Spatial Relationship Concerned Network for Change Detection
• TransFusionOdom: Interpretable Transformer-based LiDAR-Inertial Fusion Odometry Estimation
• Exploring Advances in Transformers and CNN for Skin Lesion Diagnosis on Small Datasets
• Joint Self-Supervised and Supervised Contrastive Learning for Multimodal MRI Data: Towards Predicting Abnormal Neurodevelopment
• Cross Attention-guided Dense Network for Images Fusion
• G2DA: Geometry-Guided Dual-Alignment Learning for RGB-Infrared Person Re-Identification
• Visual Sensation and Perception Computational Models for Deep Learning: State of the art, Challenges and Prospects
• BWCP: Probabilistic Learning-to-Prune Channels for ConvNets via Batch Whitening
• Elastic Tactile Simulation Towards Tactile-Visual Perception
• AudioVisual Video Summarization
• Multimodal Knowledge Expansion