Gated Multimodal Units for Information Fusion
arXiv:1702.01992
Abstract
This paper presents a novel model for multimodal learning based on gated neural networks. The Gated Multimodal Unit (GMU) model is intended to be used as an internal unit in a neural network architecture whose purpose is to find an intermediate representation based on a combination of data from different modalities. The GMU learns to decide how modalities influence the activation of the unit using multiplicative gates. It was evaluated on a multilabel scenario for genre classification of movies using the plot and the poster. The GMU improved the macro f-score performance of single-modality approaches and outperformed other fusion strategies, including mixture of experts models. Along with this work, the MM-IMDb dataset is released which, to the best of our knowledge, is the largest publicly available multimodal dataset for genre prediction on movies.
Cited by in corpus (16)
- Multimodal Intelligence: Representation Learning, Information Fusion, and Applications
- The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes
- What Makes Training Multi-Modal Classification Networks Hard?
- MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
- A Survey on Multimodal Disinformation Detection
- SMIL: Multimodal Learning with Severely Missing Modality
- Movie Trailer Genre Classification Using Multimodal Pretrained Features
- FaVoA: Face-Voice Association Favours Ambiguous Speaker Detection
- Graph Inference Representation: Learning Graph Positional Embeddings with Anchor Path Encoding
- Deep Multi-Modal Sets
- Multispectral Fusion for Object Detection with Cyclic Fuse-and-Refine Blocks
- Fine-grained Action Segmentation using the Semi-Supervised Action GAN
- DRDF: Determining the Importance of Different Multimodal Information with Dual-Router Dynamic Framework
- A Comparative Study of Speaker Role Identification in Air Traffic Communication Using Deep Learning Approaches
- Self-Supervised Neural Architecture Search for Multimodal Deep Neural Networks
- WakaVT: A Sequential Variational Transformer for Waka Generation