Leveraging Medical Visual Question Answering with Supporting Facts
arXiv:1905.12008
Abstract
In this working notes paper, we describe IBM Research AI (Almaden) team's participation in the ImageCLEF 2019 VQA-Med competition. The challenge consists of four question-answering tasks based on radiology images. The diversity of imaging modalities, organs and disease types combined with a small imbalanced training set made this a highly complex problem. To overcome these difficulties, we implemented a modular pipeline architecture that utilized transfer learning and multi-task learning. Our findings led to the development of a novel model called Supporting Facts Network (SFN). The main idea behind SFN is to cross-utilize information from upstream tasks to improve the accuracy on harder downstream ones. This approach significantly improved the scores achieved in the validation set (18 point improvement in F-1 score). Finally, we submitted four runs to the competition and were ranked seventh.
arXiv commentWorking notes from the ImageCLEF 2019 VQA-Med competition
References in corpus (8)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- VQA: Visual Question Answering
- A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Input
- Hadamard Product for Low-rank Bilinear Pooling
- Show, Ask, Attend, and Answer: A Strong Baseline For Visual Question Answering
- The Visual QA Devil in the Details: The Impact of Early Fusion and Batch Norm on CLEVR
- On transfer learning using a MAC model variant