Semi-supervised Multimodal Representation Learning through a Global Workspace

Abstract

Recent deep learning models can efficiently combine inputs from different modalities (e.g., images and text) and learn to align their latent representations, or to translate signals from one domain to another (as in image captioning, or text-to-image generation). However, current approaches mainly rely on brute-force supervised training over large multimodal datasets. In contrast, humans (and other animals) can learn useful multimodal representations from only sparse experience with matched cross-modal data. Here we evaluate the capabilities of a neural network architecture inspired by the cognitive notion of a "Global Workspace": a shared representation for two (or more) input modalities. Each modality is processed by a specialized system (pretrained on unimodal data, and subsequently frozen). The corresponding latent representations are then encoded to and decoded from a single shared workspace. Importantly, this architecture is amenable to self-supervised training via cycle-consistency: encoding-decoding sequences should approximate the identity function. For various pairings of vision-language modalities and across two datasets of varying complexity, we show that such an architecture can be trained to align and translate between two modalities with very little need for matched data (from 4 to 7 times less than a fully supervised approach). The global workspace representation can be used advantageously for downstream classification tasks and for robust transfer learning. Ablation studies reveal that both the shared workspace and the self-supervised cycle-consistency training are critical to the system's performance.

arXiv commentUnder review

References in corpus (13)

• Auto-Encoding Variational Bayes
• Neural Machine Translation by Jointly Learning to Align and Translate
• The Symbol Grounding Problem
• Hierarchical Text-Conditional Image Generation with CLIP Latents
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Emergent Abilities of Large Language Models
• Dual Learning for Machine Translation
• CoCa: Contrastive Captioners are Image-Text Foundation Models
• Phrase-Based & Neural Unsupervised Machine Translation
• Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks
• Mastering Diverse Domains through World Models
• C-Pack: Packed Resources For General Chinese Embeddings
• Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities