Large-vocabulary forensic pathological analyses via prototypical cross-modal contrastive learning

Abstract

Forensic pathology is critical in determining the cause and manner of death through post-mortem examinations, both macroscopic and microscopic. The field, however, grapples with issues such as outcome variability, laborious processes, and a scarcity of trained professionals. This paper presents SongCi, an innovative visual-language model (VLM) designed specifically for forensic pathology. SongCi utilizes advanced prototypical cross-modal self-supervised contrastive learning to enhance the accuracy, efficiency, and generalizability of forensic analyses. It was pre-trained and evaluated on a comprehensive multi-center dataset, which includes over 16 million high-resolution image patches, 2,228 vision-language pairs of post-mortem whole slide images (WSIs), and corresponding gross key findings, along with 471 distinct diagnostic outcomes. Our findings indicate that SongCi surpasses existing multi-modal AI models in many forensic pathology tasks, performs comparably to experienced forensic pathologists and significantly better than less experienced ones, and provides detailed multi-modal explainability, offering critical assistance in forensic investigations. To the best of our knowledge, SongCi is the first VLM specifically developed for forensic pathological analysis and the first large-vocabulary computational pathology (CPath) model that directly processes gigapixel WSIs in forensic science.

28 pages, 6 figures, under review

References in corpus (12)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Representation Learning with Contrastive Predictive Coding
• Learning Transferable Visual Models From Natural Language Supervision
• Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
• Flamingo: a Visual Language Model for Few-Shot Learning
• DINOv2: Learning Robust Visual Features without Supervision
• LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day
• GIT: A Generative Image-to-text Transformer for Vision and Language
• Perceiver: General Perception with Iterative Attention
• Quilt-1M: One Million Image-Text Pairs for Histopathology
• Connecting Multi-modal Contrastive Representations
• Large-vocabulary forensic pathological analyses via prototypical cross-modal contrastive learning

Cited by in corpus (1)

• Large-vocabulary forensic pathological analyses via prototypical cross-modal contrastive learning