Transformers in Healthcare: A Survey

Abstract

With Artificial Intelligence (AI) increasingly permeating various aspects of society, including healthcare, the adoption of the Transformers neural network architecture is rapidly changing many applications. Transformer is a type of deep learning architecture initially developed to solve general-purpose Natural Language Processing (NLP) tasks and has subsequently been adapted in many fields, including healthcare. In this survey paper, we provide an overview of how this architecture has been adopted to analyze various forms of data, including medical imaging, structured and unstructured Electronic Health Records (EHR), social media, physiological signals, and biomolecular sequences. Those models could help in clinical diagnosis, report generation, data reconstruction, and drug/protein synthesis. We identified relevant studies using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. We also discuss the benefits and limitations of using transformers in healthcare and examine issues such as computational cost, model interpretability, fairness, alignment with human values, ethical implications, and environmental impact.

References in corpus (66)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• Survey of Hallucination in Natural Language Generation
• Training language models to follow instructions with human feedback
• TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• The Future of Digital Health with Federated Learning
• Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
• NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding
• Cross-lingual Language Model Pretraining
• BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining
• Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting
• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Skin Lesion Analysis Toward Melanoma Detection 2018: A Challenge Hosted by the International Skin Imaging Collaboration (ISIC)
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Molecular Transformer - A Model for Uncertainty-Calibrated Chemical Reaction Prediction
• Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentation
• REFUGE Challenge: A Unified Framework for Evaluating Automated Methods for Glaucoma Assessment from Fundus Photographs
• TransMorph: Transformer for unsupervised medical image registration
• A large annotated medical image dataset for the development and evaluation of segmentation algorithms
• AraBERT: Transformer-based Model for Arabic Language Understanding
• PadChest: A large chest x-ray image dataset with multi-label annotated reports
• MolTrans: Molecular Interaction Transformer for Drug Target Interaction Prediction
• ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission
• ResViT: Residual vision transformers for multi-modal medical image synthesis
• Efficiently Modeling Long Sequences with Structured State Spaces
• ChemBERTa: Large-Scale Self-Supervised Pretraining for Molecular Property Prediction
• Enhancing Clinical Concept Extraction with Contextual Embeddings
• nnFormer: Interleaved Transformer for Volumetric Segmentation
• MURA: Large Dataset for Abnormality Detection in Musculoskeletal Radiographs
• CTformer: Convolution-free Token2Token Dilated Vision Transformer for Low-dose CT Denoising
• BrainTorrent: A Peer-to-Peer Environment for Decentralized Federated Learning
• MISSFormer: An Effective Medical Image Segmentation Transformer
• Towards Explainable Anticancer Compound Sensitivity Prediction via Multimodal Attention-based Convolutional Encoders
• FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
• BiTr-Unet: a CNN-Transformer Combined Network for MRI Brain Tumor Segmentation
• Compressing Large-Scale Transformer-Based Models: A Case Study on BERT
• Bidirectional Representation Learning from Transformers using Multimodal Electronic Health Record Data to Predict Depression
• Federated Tensor Factorization for Computational Phenotyping
• One Model to Synthesize Them All: Multi-contrast Multi-scale Transformer for Missing Data Imputation
• DEPTWEET: A Typology for Social Media Texts to Detect Depression Severities
• AttentionXML: Label Tree-based Attention-Aware Deep Model for High-Performance Extreme Multi-Label Text Classification
• TransAttUnet: Multi-level Attention-guided U-Net with Transformer for Medical Image Segmentation
• XLSor: A Robust and Accurate Lung Segmentor on Chest X-Rays Using Criss-Cross Attention and Customized Radiorealistic Abnormalities Generation
• EEGDnet: Fusing Non-Local and Local Self-Similarity for 1-D EEG Signal Denoising with 2-D Transformer
• ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers
• Multi-view analysis of unregistered medical images using cross-view transformers
• Attention-Based Deep Learning Framework for Human Activity Recognition with User Adaptation
• BERT-based Ranking for Biomedical Entity Normalization
• The Russian Drug Reaction Corpus and Neural Models for Drug Reactions and Effectiveness Detection in User Reviews
• Pyramid Medical Transformer for Medical Image Segmentation
• SpecTr: Spectral Transformer for Hyperspectral Pathology Image Segmentation
• Sensitive Data Detection and Classification in Spanish Clinical Text: Experiments with BERT
• PTNet: A High-Resolution Infant MRI Synthesizer Based on Transformer
• DeepGene Transformer: Transformer for the gene expression-based classification of cancer subtypes
• Eformer: Edge Enhancement based Transformer for Medical Image Denoising
• DAISI: Database for AI Surgical Instruction
• TransClaw U-Net: Claw U-Net with Transformers for Medical Image Segmentation
• Holistically Explainable Vision Transformers
• ReconFormer: Accelerated MRI Reconstruction Using Recurrent Transformer
• Transformer Networks for Data Augmentation of Human Physical Activity Recognition
• Low-Dose CT Denoising via Sinogram Inner-Structure Transformer
• Detecting Adverse Drug Reactions from Twitter through Domain-Specific Preprocessing and BERT Ensembling
• Spach Transformer: Spatial and Channel-wise Transformer Based on Local and Global Self-attentions for PET Image Denoising
• NVIDIA NeMo Neural Machine Translation Systems for English-German and English-Russian News and Biomedical Tasks at WMT21
• A Deep Learning Approach Using Masked Image Modeling for Reconstruction of Undersampled K-spaces

Cited by in corpus (3)

• Advanced Deep Learning and Large Language Models: Comprehensive Insights for Cancer Detection
• CogniAlign: Word-Level Multimodal Speech Alignment with Gated Cross-Attention for Alzheimer's Detection
• Interpretable Auto Window Setting for Deep-Learning-Based CT Analysis