Combining 3D Image and Tabular Data via the Dynamic Affine Feature Map Transform

Abstract

Prior work on diagnosing Alzheimer's disease from magnetic resonance images of the brain established that convolutional neural networks (CNNs) can leverage the high-dimensional image information for classifying patients. However, little research focused on how these models can utilize the usually low-dimensional tabular information, such as patient demographics or laboratory measurements. We introduce the Dynamic Affine Feature Map Transform (DAFT), a general-purpose module for CNNs that dynamically rescales and shifts the feature maps of a convolutional layer, conditional on a patient's tabular clinical information. We show that DAFT is highly effective in combining 3D image and tabular information for diagnosis and time-to-dementia prediction, where it outperforms competing CNNs with a mean balanced accuracy of 0.622 and mean c-index of 0.748, respectively. Our extensive ablation study provides valuable insights into the architectural properties of DAFT. Our implementation is available at https://github.com/ai-med/DAFT.

arXiv commentAccepted at 2021 International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI)

References in corpus (5)

• Decoupled Weight Decay Regularization
• Convolutional Neural Networks for Classification of Alzheimer's Disease: Overview and Reproducible Evaluation
• End-To-End Alzheimer's Disease Diagnosis and Biomarker Identification
• A Wide and Deep Neural Network for Survival Analysis from Anatomical Shape and Tabular Clinical Data
• Semi-Structured Deep Piecewise Exponential Models

Cited by in corpus (4)

• Deep Neural Networks and Tabular Data: A Survey
• A multi-stream convolutional neural network for classification of progressive MCI in Alzheimer's disease using structural MRI images
• DrFuse: Learning Disentangled Representation for Clinical Multi-Modal Fusion with Missing Modality and Modal Inconsistency
• TabAttention: Learning Attention Conditionally on Tabular Data