Symbolic Semantic Segmentation and Interpretation of COVID-19 Lung Infections in Chest CT volumes based on Emergent Languages

Abstract

The coronavirus disease (COVID-19) has resulted in a pandemic crippling the a breadth of services critical to daily life. Segmentation of lung infections in computerized tomography (CT) slices could be be used to improve diagnosis and understanding of COVID-19 in patients. Deep learning systems lack interpretability because of their black box nature. Inspired by human communication of complex ideas through language, we propose a symbolic framework based on emergent languages for the segmentation of COVID-19 infections in CT scans of lungs. We model the cooperation between two artificial agents - a Sender and a Receiver. These agents synergistically cooperate using emergent symbolic language to solve the task of semantic segmentation. Our game theoretic approach is to model the cooperation between agents unlike Generative Adversarial Networks (GANs). The Sender retrieves information from one of the higher layers of the deep network and generates a symbolic sentence sampled from a categorical distribution of vocabularies. The Receiver ingests the stream of symbols and cogenerates the segmentation mask. A private emergent language is developed that forms the communication channel used to describe the task of segmentation of COVID infections. We augment existing state of the art semantic segmentation architectures with our symbolic generator to form symbolic segmentation models. Our symbolic segmentation framework achieves state of the art performance for segmentation of lung infections caused by COVID-19. Our results show direct interpretation of symbolic sentences to discriminate between normal and infected regions, infection morphology and image characteristics. We show state of the art results for segmentation of COVID-19 lung infections in CT.

References in corpus (13)

• Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models
• DenseNet: Implementing Efficient ConvNet Descriptor Pyramids
• Association of genomic subtypes of lower-grade gliomas with shape features automatically extracted by a deep learning algorithm
• Semantic Segmentation of Pathological Lung Tissue with Dilated Fully Convolutional Networks
• Severity Assessment of Coronavirus Disease 2019 (COVID-19) Using Quantitative Features from Chest CT Images
• Artificial Intelligence Forecasting of Covid-19 in China
• Harmony-Search and Otsu based System for Coronavirus Disease (COVID-19) Detection using Lung CT Scan Images
• A Survey of Semantic Segmentation
• Emergent Translation in Multi-Agent Communication
• Emergence of Compositional Language with Deep Generational Transmission
• Emergence of Linguistic Communication from Referential Games with Symbolic and Pixel Input
• CT-Realistic Lung Nodule Simulation from 3D Conditional Generative Adversarial Networks for Robust Lung Segmentation
• Towards Emergent Language Symbolic Semantic Segmentation and Model Interpretability