MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
arXiv:2107.07502
Abstract
Learning multimodal representations involves integrating information from multiple heterogeneous sources of data. It is a challenging yet crucial area with numerous real-world applications in multimedia, affective computing, robotics, finance, human-computer interaction, and healthcare. Unfortunately, multimodal research has seen limited resources to study (1) generalization across domains and modalities, (2) complexity during training and inference, and (3) robustness to noisy and missing modalities. In order to accelerate progress towards understudied modalities and tasks while ensuring real-world robustness, we release MultiBench, a systematic and unified large-scale benchmark spanning 15 datasets, 10 modalities, 20 prediction tasks, and 6 research areas. MultiBench provides an automated end-to-end machine learning pipeline that simplifies and standardizes data loading, experimental setup, and model evaluation. To enable holistic evaluation, MultiBench offers a comprehensive methodology to assess (1) generalization, (2) time and space complexity, and (3) modality robustness. MultiBench introduces impactful challenges for future research, including scalability to large-scale multimodal datasets and robustness to realistic imperfections. To accompany this benchmark, we also provide a standardized implementation of 20 core approaches in multimodal learning. Simply applying methods proposed in different research areas can improve the state-of-the-art performance on 9/15 datasets. Therefore, MultiBench presents a milestone in unifying disjoint efforts in multimodal research and paves the way towards a better understanding of the capabilities and limitations of multimodal models, all the while ensuring ease of use, accessibility, and reproducibility. MultiBench, our standardized code, and leaderboards are publicly available, will be regularly updated, and welcomes inputs from the community.
arXiv commentNeurIPS 2021 Datasets and Benchmarks Track. Code: https://github.com/pliang279/MultiBench and Website: https://cmu-multicomp-lab.github.io/multibench/
References in corpus (27)
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
- Semantics derived automatically from language corpora contain human-like biases
- The Kinetics Human Action Video Dataset
- VisualBERT: A Simple and Performant Baseline for Vision and Language
- Zero-Shot Text-to-Image Generation
- Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge
- Contrastive Multiview Coding
- Differentially Private Federated Learning: A Client Level Perspective
- Federated Optimization in Heterogeneous Networks
- MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videos
- WILDS: A Benchmark of in-the-Wild Distribution Shifts
- Uncertainty-aware multi-view co-training for semi-supervised medical image segmentation and domain adaptation
- Learning Factorized Multimodal Representations
- Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift
- Supervised Multimodal Bitransformers for Classifying Images and Text
- UR-FUNNY: A Multimodal Language Dataset for Understanding Humor
- Think Locally, Act Globally: Federated Learning with Local and Global Representations
- Gated-Attention Architectures for Task-Oriented Language Grounding
- Gated Multimodal Units for Information Fusion
- Using Syntax to Ground Referring Expressions in Natural Images
- Probabilistic Neural-symbolic Models for Interpretable Visual Question Answering
- Beyond Pinball Loss: Quantile Methods for Calibrated Uncertainty Quantification
- SMIL: Multimodal Learning with Severely Missing Modality
- Multimodal Fusion Refiner Networks
- Multimodal deep learning for short-term stock volatility prediction
- MUFASA: Multimodal Fusion Architecture Search for Electronic Health Records