Visually Grounded Continual Language Learning with Selective Specialization

Abstract

A desirable trait of an artificial agent acting in the visual world is to continually learn a sequence of language-informed tasks while striking a balance between sufficiently specializing in each task and building a generalized knowledge for transfer. Selective specialization, i.e., a careful selection of model components to specialize in each task, is a strategy to provide control over this trade-off. However, the design of selection strategies requires insights on the role of each model component in learning rather specialized or generalizable representations, which poses a gap in current research. Thus, our aim with this work is to provide an extensive analysis of selection strategies for visually grounded continual language learning. Due to the lack of suitable benchmarks for this purpose, we introduce two novel diagnostic datasets that provide enough control and flexibility for a thorough model analysis. We assess various heuristics for module specialization strategies as well as quantifiable measures for two different types of model architectures. Finally, we design conceptually simple approaches based on our analysis that outperform common continual learning baselines. Our results demonstrate the need for further efforts towards better aligning continual learning algorithms with the learning behaviors of individual model parts.

arXiv commentAccepted to EMNLP 2023 Findings

References in corpus (21)

• Overcoming catastrophic forgetting in neural networks
• Representation Learning with Contrastive Predictive Coding
• Progressive Neural Networks
• Efficient Lifelong Learning with A-GEM
• Three scenarios for continual learning
• Progress & Compress: A scalable framework for continual learning
• Compacting, Picking and Growing for Unforgetting Continual Learning
• Universal representations:The missing link between faces, text, planktons, and cat breeds
• CLIPort: What and Where Pathways for Robotic Manipulation
• A Comprehensive Survey of Continual Learning: Theory, Method and Application
• Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning
• Efficient Continual Learning with Modular Networks and Task-Driven Priors
• CLiMB: A Continual Learning Benchmark for Vision-and-Language Tasks
• SparCL: Sparse Continual Learning on the Edge
• Continual Prune-and-Select: Class-incremental learning with specialized subnetworks
• NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks
• Memory Efficient Continual Learning with Transformers
• Lifelong Learning of Compositional Structures
• CoLLIE: Continual Learning of Language Grounding from Language-Image Embeddings
• Modular Lifelong Reinforcement Learning via Neural Composition
• BabyAI 1.1