Efficient Large-Scale Multi-Modal Classification
arXiv:1802.02892
Abstract
While the incipient internet was largely text-based, the modern digital world is becoming increasingly multi-modal. Here, we examine multi-modal classification where one modality is discrete, e.g. text, and the other is continuous, e.g. visual representations transferred from a convolutional neural network. In particular, we focus on scenarios where we have to be able to classify large quantities of data quickly. We investigate various methods for performing multi-modal fusion and analyze their trade-offs in terms of classification accuracy and computational efficiency. Our findings indicate that the inclusion of continuous information improves performance over text-only on a range of multi-modal classification tasks, even with simple fusion methods. In addition, we experiment with discretizing the continuous features in order to speed up and simplify the fusion process even further. Our results show that fusion with discretized features outperforms text-only classification, at a fraction of the computational cost of full multi-modal fusion, with the additional benefit of improved interpretability.
arXiv commentPublished at AAAI-18, 7 pages
Cited by in corpus (15)
- Deep Multimodal Subspace Clustering Networks
- Semantic-Aware Scene Recognition
- Decoding EEG Brain Activity for Multi-Modal Natural Language Processing
- Trusted Multi-View Classification
- What Makes Training Multi-Modal Classification Networks Hard?
- Detection of Illicit Drug Trafficking Events on Instagram: A Deep Multimodal Multilabel Learning Approach
- One Embedding To Do Them All
- Discriminative Ridge Machine: A Classifier for High-Dimensional Data or Imbalanced Data
- On-Device Document Classification using multimodal features
- Trustworthy Long-Tailed Classification
- Audio-Visual Transformer Based Crowd Counting
- Can audio-visual integration strengthen robustness under multimodal attacks?
- Cross-modal Speaker Verification and Recognition: A Multilingual Perspective
- Dual-modality seq2seq network for audio-visual event localization
- Training Multimodal Systems for Classification with Multiple Objectives