Microscope 2.0: An Augmented Reality Microscope with Real-time Artificial Intelligence Integration

Abstract

The brightfield microscope is instrumental in the visual examination of both biological and physical samples at sub-millimeter scales. One key clinical application has been in cancer histopathology, where the microscopic assessment of the tissue samples is used for the diagnosis and staging of cancer and thus guides clinical therapy. However, the interpretation of these samples is inherently subjective, resulting in significant diagnostic variability. Moreover, in many regions of the world, access to pathologists is severely limited due to lack of trained personnel. In this regard, Artificial Intelligence (AI) based tools promise to improve the access and quality of healthcare. However, despite significant advances in AI research, integration of these tools into real-world cancer diagnosis workflows remains challenging because of the costs of image digitization and difficulties in deploying AI solutions. Here we propose a cost-effective solution to the integration of AI: the Augmented Reality Microscope (ARM). The ARM overlays AI-based information onto the current view of the sample through the optical pathway in real-time, enabling seamless integration of AI into the regular microscopy workflow. We demonstrate the utility of ARM in the detection of lymph node metastases in breast cancer and the identification of prostate cancer with a latency that supports real-time workflows. We anticipate that ARM will remove barriers towards the use of AI in microscopic analysis and thus improve the accuracy and efficiency of cancer diagnosis. This approach is applicable to other microscopy tasks and AI algorithms in the life sciences and beyond.

References in corpus (6)

• TensorFlow: A system for large-scale machine learning
• Detecting Cancer Metastases on Gigapixel Pathology Images
• Development and Validation of a Deep Learning Algorithm for Improving Gleason Scoring of Prostate Cancer
• Similar Image Search for Histopathology: SMILY
• Whole-Slide Image Focus Quality: Automatic Assessment and Impact on AI Cancer Detection
• Deep Convolutional Neural Networks for Microscopy-Based Point of Care Diagnostics

Cited by in corpus (19)

• Deep Learning-based Computational Pathology Predicts Origins for Cancers of Unknown Primary
• Development and Validation of a Deep Learning Algorithm for Improving Gleason Scoring of Prostate Cancer
• Artificial Intelligence for Digital and Computational Pathology
• CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection
• Hard Sample Aware Noise Robust Learning for Histopathology Image Classification
• Deep Learning for Computational Cytology: A Survey
• Whole-Slide Image Focus Quality: Automatic Assessment and Impact on AI Cancer Detection
• A Deep Learning Based Automatic Defect Analysis Framework for In-situ TEM Ion Irradiations
• Applications and Techniques for Fast Machine Learning in Science
• Multi defect detection and analysis of electron microscopy images with deep learning
• A Survey on Applications of Cache-Aided NOMA
• Dual-stream Multiple Instance Learning Network for Whole Slide Image Classification with Self-supervised Contrastive Learning
• Representation Learning for Electronic Health Records
• Multimodal Multitask Representation Learning for Pathology Biobank Metadata Prediction
• OpenSRH: optimizing brain tumor surgery using intraoperative stimulated Raman histology
• Domain-Specific Pre-training Improves Confidence in Whole Slide Image Classification
• Autofocusing technologies for whole slide imaging and automated microscopy
• Blind deblurring for microscopic pathology images using deep learning networks
• Human-centric Metric for Accelerating Pathology Reports Annotation