Large AI Models in Health Informatics: Applications, Challenges, and the Future

Abstract

Large AI models, or foundation models, are models recently emerging with massive scales both parameter-wise and data-wise, the magnitudes of which can reach beyond billions. Once pretrained, large AI models demonstrate impressive performance in various downstream tasks. A prime example is ChatGPT, whose capability has compelled people's imagination about the far-reaching influence that large AI models can have and their potential to transform different domains of our lives. In health informatics, the advent of large AI models has brought new paradigms for the design of methodologies. The scale of multi-modal data in the biomedical and health domain has been ever-expanding especially since the community embraced the era of deep learning, which provides the ground to develop, validate, and advance large AI models for breakthroughs in health-related areas. This article presents a comprehensive review of large AI models, from background to their applications. We identify seven key sectors in which large AI models are applicable and might have substantial influence, including 1) bioinformatics; 2) medical diagnosis; 3) medical imaging; 4) medical informatics; 5) medical education; 6) public health; and 7) medical robotics. We examine their challenges, followed by a critical discussion about potential future directions and pitfalls of large AI models in transforming the field of health informatics.

arXiv commentThis article has been accepted for publication in IEEE Journal of Biomedical and Health Informatics

References in corpus (75)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• BioBERT: a pre-trained biomedical language representation model for biomedical text mining
• A Simple Framework for Contrastive Learning of Visual Representations
• Training language models to follow instructions with human feedback
• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• LLaMA: Open and Efficient Foundation Language Models
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• LoRA: Low-Rank Adaptation of Large Language Models
• Hierarchical Text-Conditional Image Generation with CLIP Latents
• On the Opportunities and Risks of Foundation Models
• Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
• PaLM: Scaling Language Modeling with Pathways
• Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision
• Scaling Instruction-Finetuned Language Models
• BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining
• EfficientNetV2: Smaller Models and Faster Training
• Zero-Shot Text-to-Image Generation
• Large Language Models are Zero-Shot Reasoners
• Emergent Abilities of Large Language Models
• LAION-5B: An open large-scale dataset for training next generation image-text models
• Transformer in Transformer
• GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
• BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
• BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
• Tracking Social Media Discourse About the COVID-19 Pandemic: Development of a Public Coronavirus Twitter Data Set
• ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases
• CoAtNet: Marrying Convolution and Attention for All Data Sizes
• Publicly Available Clinical BERT Embeddings
• LaMDA: Language Models for Dialog Applications
• Training Compute-Optimal Large Language Models
• Multitask Prompted Training Enables Zero-Shot Task Generalization
• Capabilities of GPT-4 on Medical Challenge Problems
• PaLM-E: An Embodied Multimodal Language Model
• SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
• Florence: A New Foundation Model for Computer Vision
• Med3D: Transfer Learning for 3D Medical Image Analysis
• Scaling Autoregressive Models for Content-Rich Text-to-Image Generation
• Towards Expert-Level Medical Question Answering with Large Language Models
• Improving language models by retrieving from trillions of tokens
• Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
• Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation
• PaLI: A Jointly-Scaled Multilingual Language-Image Model
• Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models
• ClimaX: A foundation model for weather and climate
• Quantifying Memorization Across Neural Language Models
• Language Is Not All You Need: Aligning Perception with Language Models
• DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature
• Generalist Vision Foundation Models for Medical Imaging: A Case Study of Segment Anything Model on Zero-Shot Medical Segmentation
• Self-supervised Pretraining of Visual Features in the Wild
• Improving alignment of dialogue agents via targeted human judgements
• Scaling Vision Transformers to 22 Billion Parameters
• CLIPort: What and Where Pathways for Robotic Manipulation
• Aligning AI With Shared Human Values
• HuaTuo: Tuning LLaMA Model with Chinese Medical Knowledge
• On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective
• Understanding artificial intelligence ethics and safety
• Can large language models reason about medical questions?
• ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Models
• In ChatGPT We Trust? Measuring and Characterizing the Reliability of ChatGPT
• ChatCAD+: Towards a Universal and Reliable Interactive CAD using LLMs
• VIMA: General Robot Manipulation with Multimodal Prompts
• STU-Net: Scalable and Transferable Medical Image Segmentation Models Empowered by Large-Scale Supervised Pre-training
• Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulation
• Meta-Transformer: A Unified Framework for Multimodal Learning
• Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domains
• XrayGPT: Chest Radiographs Summarization using Medical Vision-Language Models
• Egocentric Image Captioning for Privacy-Preserved Passive Dietary Intake Monitoring
• Discovering the Hidden Vocabulary of DALLE-2
• Egocentric Human Trajectory Forecasting with a Wearable Camera and Multi-Modal Fusion
• Tuning computer vision models with task rewards
• Are Diffusion Models Vulnerable to Membership Inference Attacks?
• Data Augmentation Alone Can Improve Adversarial Training
• Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics

Cited by in corpus (11)

• Generalist Vision Foundation Models for Medical Imaging: A Case Study of Segment Anything Model on Zero-Shot Medical Segmentation
• A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine
• VisionFM: a Multi-Modal Multi-Task Vision Foundation Model for Generalist Ophthalmic Artificial Intelligence
• ChatCAD+: Towards a Universal and Reliable Interactive CAD using LLMs
• Leveraging Large Language Models for Patient Engagement: The Power of Conversational AI in Digital Health
• Patch is Enough: Naturalistic Adversarial Patch against Vision-Language Pre-training Models
• Effectiveness Assessment of Recent Large Vision-Language Models
• BC-MRI-SEG: A Breast Cancer MRI Tumor Segmentation Benchmark
• A Hybrid Artificial Intelligence System for Automated EEG Background Analysis and Report Generation
• High-Throughput Phenotyping of Clinical Text Using Large Language Models
• Quantum-Inspired Weight-Constrained Neural Network: Reducing Variable Numbers by 100x Compared to Standard Neural Networks