MDNet: A Semantically and Visually Interpretable Medical Image Diagnosis Network
arXiv:1707.02485
Abstract
The inability to interpret the model prediction in semantically and visually meaningful ways is a well-known shortcoming of most existing computer-aided diagnosis methods. In this paper, we propose MDNet to establish a direct multimodal mapping between medical images and diagnostic reports that can read images, generate diagnostic reports, retrieve images by symptom descriptions, and visualize attention, to provide justifications of the network diagnosis process. MDNet includes an image model and a language model. The image model is proposed to enhance multi-scale feature ensembles and utilization efficiency. The language model, integrated with our improved attention mechanism, aims to read and explore discriminative image feature descriptions from reports to learn a direct mapping from sentence words to image pixels. The overall network is trained end-to-end by using our developed optimization strategy. Based on a pathology bladder cancer images and its diagnostic reports (BCIDR) dataset, we conduct sufficient experiments to demonstrate that MDNet outperforms comparative baselines. The proposed image model obtains state-of-the-art performance on two CIFAR datasets as well.
CVPR2017 Oral
Cited by in corpus (13)
- Density-aware Single Image De-raining using a Multi-stream Dense Network
- Tell Me Where to Look: Guided Attention Inference Network
- Lessons Learned from Designing an AI-Enabled Diagnosis Tool for Pathologists
- Photographic Text-to-Image Synthesis with a Hierarchically-nested Adversarial Network
- DeepMiner: Discovering Interpretable Representations for Mammogram Classification and Explanation
- Deep Texture Manifold for Ground Terrain Recognition
- ProtoryNet - Interpretable Text Classification Via Prototype Trajectories
- Unifying Neural Learning and Symbolic Reasoning for Spinal Medical Report Generation
- TandemNet: Distilling Knowledge from Medical Images Using Diagnostic Reports as Optional Semantic References
- Leveraging Disease Progression Learning for Medical Image Recognition
- RTEX: A novel methodology for Ranking, Tagging, and Explanatory diagnostic captioning of radiography exams
- CLARA: Clinical Report Auto-completion
- Automated Knee X-ray Report Generation