Three-dimensional virtual refocusing of fluorescence microscopy images using deep learning

Abstract

Three-dimensional (3D) fluorescence microscopy in general requires axial scanning to capture images of a sample at different planes. Here we demonstrate that a deep convolutional neural network can be trained to virtually refocus a 2D fluorescence image onto user-defined 3D surfaces within the sample volume. With this data-driven computational microscopy framework, we imaged the neuron activity of a Caenorhabditis elegans worm in 3D using a time-sequence of fluorescence images acquired at a single focal plane, digitally increasing the depth-of-field of the microscope by 20-fold without any axial scanning, additional hardware, or a trade-off of imaging resolution or speed. Furthermore, we demonstrate that this learning-based approach can correct for sample drift, tilt, and other image aberrations, all digitally performed after the acquisition of a single fluorescence image. This unique framework also cross-connects different imaging modalities to each other, enabling 3D refocusing of a single wide-field fluorescence image to match confocal microscopy images acquired at different sample planes. This deep learning-based 3D image refocusing method might be transformative for imaging and tracking of 3D biological samples, especially over extended periods of time, mitigating photo-toxicity, sample drift, aberration and defocusing related challenges associated with standard 3D fluorescence microscopy techniques.

arXiv comment47 pages, 5 figures (main text)

References in corpus (9)

• Simultaneous whole-animal 3D-imaging of neuronal activity using light field microscopy
• Deep Learning Microscopy
• Deep-STORM: super-resolution single-molecule microscopy by deep learning
• Whole-brain calcium imaging with cellular resolution in freely behaving C. elegans
• Brain-wide 3D imaging of neuronal activity in Caenorhabditis elegans with sculpted light
• Extended depth-of-field in holographic image reconstruction using deep learning based auto-focusing and phase-recovery
• Deep learning enhanced mobile-phone microscopy
• Cross-modality deep learning brings bright-field microscopy contrast to holography
• Automatically tracking neurons in a moving and deforming brain

Cited by in corpus (20)

• Terahertz Pulse Shaping Using Diffractive Surfaces
• Computational Imaging Without a Computer: Seeing Through Random Diffusers at the Speed of Light
• Single-Shot 3D Widefield Fluorescence Imaging with a Computational Miniature Mesoscope
• Neural network-based on-chip spectroscopy using a scalable plasmonic encoder
• Single-shot autofocusing of microscopy images using deep learning
• Deep learning in biomedical optics
• Scale-, shift- and rotation-invariant diffractive optical networks
• Spatially-Variant CNN-based Point Spread Function Estimation for Blind Deconvolution and Depth Estimation in Optical Microscopy
• Artificial confocal microscopy for deep label-free imaging
• Neural network-based image reconstruction in swept-source optical coherence tomography using undersampled spectral data
• SiWa: See into Walls via Deep UWB Radar
• Recurrent neural network-based volumetric fluorescence microscopy
• Virtual staining of defocused autofluorescence images of unlabeled tissue using deep neural networks
• Deep learning-based virtual refocusing of images using an engineered point-spread function
• Neural Network-Based Processing and Reconstruction of Compromised Biophotonic Image Data
• Universal point spread function engineering for 3D optical information processing
• Learning to Model and Calibrate Optics via a Differentiable Wave Optics Simulator
• NeuRegenerate: A Framework for Visualizing Neurodegeneration
• Advancing biological super-resolution microscopy through deep learning: a brief review
• Diffractive Interconnects: All-Optical Permutation Operation Using Diffractive Networks