Decomposing Normal and Abnormal Features of Medical Images into Discrete Latent Codes for Content-Based Image Retrieval

Abstract

In medical imaging, the characteristics purely derived from a disease should reflect the extent to which abnormal findings deviate from the normal features. Indeed, physicians often need corresponding images without abnormal findings of interest or, conversely, images that contain similar abnormal findings regardless of normal anatomical context. This is called comparative diagnostic reading of medical images, which is essential for a correct diagnosis. To support comparative diagnostic reading, content-based image retrieval (CBIR), which can selectively utilize normal and abnormal features in medical images as two separable semantic components, will be useful. Therefore, we propose a neural network architecture to decompose the semantic components of medical images into two latent codes: normal anatomy code and abnormal anatomy code. The normal anatomy code represents normal anatomies that should have existed if the sample is healthy, whereas the abnormal anatomy code attributes to abnormal changes that reflect deviation from the normal baseline. These latent codes are discretized through vector quantization to enable binary hashing, which can reduce the computational burden at the time of similarity search. By calculating the similarity based on either normal or abnormal anatomy codes or the combination of the two codes, our algorithm can retrieve images according to the selected semantic component from a dataset consisting of brain magnetic resonance images of gliomas. Our CBIR system qualitatively and quantitatively achieves remarkable results.

References in corpus (20)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Auto-Encoding Variational Bayes
• Neural Discrete Representation Learning
• Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations
• Empirical Evaluation of Rectified Activations in Convolutional Network
• Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations
• Deep Convolutional Inverse Graphics Network
• Deep Variational Information Bottleneck
• Towards a Definition of Disentangled Representations
• Self-Supervised Video Hashing with Hierarchical Binary Auto-encoder
• Understanding disentangling in $\beta$-VAE
• Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
• Disentangled Representation Learning in Cardiac Image Analysis
• Fast Decoding in Sequence Models using Discrete Latent Variables
• Discovering Hidden Factors of Variation in Deep Networks
• Structured Disentangled Representations
• Disentangling Human Error from the Ground Truth in Segmentation of Medical Images
• Pseudo-healthy synthesis with pathology disentanglement and adversarial learning
• An Explicit Local and Global Representation Disentanglement Framework with Applications in Deep Clustering and Unsupervised Object Detection
• Learning Cross-domain Generalizable Features by Representation Disentanglement

Cited by in corpus (1)

• Learning Disentangled Representations in the Imaging Domain