Transfer Learning in Visual and Relational Reasoning

Abstract

Transfer learning has become the de facto standard in computer vision and natural language processing, especially where labeled data is scarce. Accuracy can be significantly improved by using pre-trained models and subsequent fine-tuning. In visual reasoning tasks, such as image question answering, transfer learning is more complex. In addition to transferring the capability to recognize visual features, we also expect to transfer the system's ability to reason. Moreover, for video data, temporal reasoning adds another dimension. In this work, we formalize these unique aspects of transfer learning and propose a theoretical framework for visual reasoning, exemplified by the well-established CLEVR and COG datasets. Furthermore, we introduce a new, end-to-end differentiable recurrent model (SAMNet), which shows state-of-the-art accuracy and better performance in transfer learning on both datasets. The improved performance of SAMNet stems from its capability to decouple the abstract multi-step reasoning from the length of the sequence and its selective attention enabling to store only the question-relevant objects in the external memory.

18 pages; more baseline comparisons; additional clarifications

References in corpus (21)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distributed Representations of Words and Phrases and their Compositionality
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• Dynamic Memory Networks for Visual and Textual Question Answering
• TernausNet: U-Net with VGG11 Encoder Pre-Trained on ImageNet for Image Segmentation
• A simple neural network module for relational reasoning
• Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• What makes ImageNet good for transfer learning?
• Simple Baseline for Visual Question Answering
• ABC-CNN: An Attention Based Convolutional Neural Network for Visual Question Answering
• Transparency by Design: Closing the Gap Between Performance and Interpretability in Visual Reasoning
• Stacked Attention Networks for Image Question Answering
• A Focused Dynamic Attention Model for Visual Question Answering
• Trends in Integration of Vision and Language Research: A Survey of Tasks, Datasets, and Methods
• End-to-end Concept Word Detection for Video Captioning, Retrieval, and Question Answering
• Learning by Abstraction: The Neural State Machine
• Leveraging Medical Visual Question Answering with Supporting Facts
• On transfer learning using a MAC model variant
• Learning to Remember, Forget and Ignore using Attention Control in Memory
• Dynamic Graph Generation Network: Generating Relational Knowledge from Diagrams