Using Multiple Instance Learning to Build Multimodal Representations

Abstract

Image-text multimodal representation learning aligns data across modalities and enables important medical applications, e.g., image classification, visual grounding, and cross-modal retrieval. In this work, we establish a connection between multimodal representation learning and multiple instance learning. Based on this connection, we propose a generic framework for constructing permutation-invariant score functions with many existing multimodal representation learning approaches as special cases. Furthermore, we use the framework to derive a novel contrastive learning approach and demonstrate that our method achieves state-of-the-art results in several downstream tasks.

References in corpus (5)

• Learning Transferable Visual Models From Natural Language Supervision
• Multiple Instance Learning: A Survey of Problem Characteristics and Applications
• Classifying and Segmenting Microscopy Images Using Convolutional Multiple Instance Learning
• Multi-Instance Multi-Label Learning
• Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing

Cited by in corpus (1)

• PathM3: A Multimodal Multi-Task Multiple Instance Learning Framework for Whole Slide Image Classification and Captioning