Visually Grounded Continual Learning of Compositional Phrases

Abstract

Humans acquire language continually with much more limited access to data samples at a time, as compared to contemporary NLP systems. To study this human-like language acquisition ability, we present VisCOLL, a visually grounded language learning task, which simulates the continual acquisition of compositional phrases from streaming visual scenes. In the task, models are trained on a paired image-caption stream which has shifting object distribution; while being constantly evaluated by a visually-grounded masked language prediction task on held-out test sets. VisCOLL compounds the challenges of continual learning (i.e., learning from continuously shifting data distribution) and compositional generalization (i.e., generalizing to novel compositions). To facilitate research on VisCOLL, we construct two datasets, COCO-shift and Flickr-shift, and benchmark them using different continual learning methods. Results reveal that SoTA continual learning approaches provide little to no improvements on VisCOLL, since storing examples of all possible compositions is infeasible. We conduct further ablations and analysis to guide future work.

EMNLP 2020; Fixed typos

References in corpus (15)

• Adam: A Method for Stochastic Optimization
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Decoupled Weight Decay Regularization
• Overcoming catastrophic forgetting in neural networks
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Continual Learning Through Synaptic Intelligence
• Efficient Lifelong Learning with A-GEM
• On Tiny Episodic Memories in Continual Learning
• Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines
• UNITER: UNiversal Image-TExt Representation Learning
• Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks
• CLOSURE: Assessing Systematic Generalization of CLEVR Models
• Interactive Language Learning by Question Answering
• ContCap: A scalable framework for continual image captioning

Cited by in corpus (3)

• Analogical Reasoning for Visually Grounded Language Acquisition
• Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion
• Concepts, Properties and an Approach for Compositional Generalization